Someone once told me that the need to profile I/O will soon be over as NVMe becomes more affordable and faster DIMM-mounted storage solutions merge in the market. We are starting to see the arrival of that revolution now.
To some extent, my lunch companion was right; it is increasingly possible to bury bad I/O with fast storage. However, the need for I/O profiling becomes even more important as the hardware increases in speed, not for optimisation and tuning as he assumed, but for steering and tiering as storage solutions become more complex.
The range of storage options on the market makes it more important to deploy applications in the appropriate way as storage architectures become more complex. The difference in performance between DIMM storage, NVMe, PCI mounted flash, spinning disk and tape continues to widen and humans’ insatiable thirst to generate more and more data grows.
Advances in genomic biology and medical physics unleash data sets with a high magnitude every year. As we peak in the area of self-driving cars, the need to log the behaviour of the machines that are making ever more complex decisions will bring challenges in the IoT that go far beyond what we have to cope with in HPC.
Profile I/O from the start
The worst mistake to make would be to spend a lot of money on fast storage and assume a complex storage architecture will just handle it. You will be wasting a lot of money.
The best approach to take is to profile I/O before you implement your shiny new storage, to know exactly what your applications are doing and what they need to run efficiently. Once you build the right storage architecture, you will start to feel the benefits of speed.
We recently carried out some work with the Wellcome Sanger Institute to profile applications. The project saved 10% of project costs by showing that cheaper storage would carry out the work just as effectively as faster, more expensive storage because of the I/O activity involved.
We can tell you a lot more about how I/O profiling can hugely assist in a business’s migration plan, whether that’s from one on-premises solution to another or to the cloud. Get in touch to ask us all your burning questions.