White paper: Maximising HPC performance on AWS public cloud

Ellexus and Alces Flight have jointly released a white paper titled ‘Maximising HPC performance on AWS public cloud’.

The growth of cloud computing alongside on-premise clusters has been rapid. This migration is throwing up unforeseen challenges and uncovering bottlenecks that are impacting on performance. Never has it been more important to understand how applications are accessing storage and to prepare them for a new environment.

Thankfully, tools are emerging to profile application performance on environments such as that offered by Amazon Web Services – namely Mistral from Ellexus.

Read below for the introduction and download the full white paper.

Maximising HPC performance on AWS public cloud

With the move to cloud computing comes a flexibility in the compute environment that has never been experienced before. This flexibility is great for scaling on demand and optimising on compute spend, but it can introduce challenges when tuning and optimising applications.

It used to be possible to tune a high performance computing (HPC) application and let it run for years, but major changes in compute, memory and storage capabilities that can be reconfigured at the click of a button mean that a new approach is needed for applications to be able to run well in the cloud. When the infrastructure changes, bottlenecks shift and previously masked problems can become obvious. Whether doing financial modelling, cancer research or designing the next super car, it’s never been a better time to take advantage of what cloud computing can offer and ready your applications for the change. One way of doing this is by profiling file I/O and eliminating wasteful I/O patterns.

In this white paper, we look at how easy it is to spin up an HPC cluster on Amazon Web Services (AWS) using Alces Flight and profile the performance of a typical HPC application in that environment using Mistral from Ellexus. The application we have chosen is a publicly available genome pipeline from a well-respected cancer research organisation. We were looking at I/O patterns and how to characterise the requirements of the pipeline.