Whitepaper: I/O profiling to improve the storage performance at Diamond Light Source on a Univa Grid Engine cluster

No two high-performance computing (HPC) teams and architectures will ever be the same, but the team at Diamond Light Source handles a wider variety of workloads than many. Performance of both in-house and third-party applications is therefore vital, which is where I/O profiling becomes important.

The Diamond Light Source team used Ellexus Mistral to identify straightforward improvements that could be made in order to improve performance and cut down runtime. We have put together a whitepaper to detail the results.

“By using Mistral, our team has already made a marked improvement to various applications that we maintain in-house. We have been able to reduce the impact of noisy neighbours, reduce runtime and identify applications with bad I/O. We intend to keep using Mistral to profile more applications and improve the overall architecture of our systems for in-house and third-party tools.” – Frederik Ferner, Senior Computer Systems Administrator at Diamond Light Source  

Diamond Light Source

Diamond Light Source is the UK’s national synchrotron or particle accelerator. Unlike particle accelerators such as those used in the labs at CERN to look at the result of colliding particles, the Diamond Light Source uses the radiation from speeding electrons to beam bright light into a range of experiments arranged around the edges of the accelerator. It works like a giant microscope, harnessing the power of electrons to produce bright light that scientists can use to study anything from fossils to jet engines to viruses and vaccines.

The machine accelerates electrons to near light speeds so that they give off light 10 billion times brighter than the sun. These bright beams are then directed into laboratories known as ‘beamlines’. Here, scientists use the light to study a vast range of subject matter, from new medicines and treatments for disease to innovative engineering and cutting-edge technology.

Each beamline is used in an experiment that generates vast amounts of data, which must be stored in real time and processed quickly. Experiments cover a range of fields from industrial engineering to microbiology and cystalography, which means the HPC team needs to be able to handle a wide range of data rates and compute applications.

Diamond Light Source has been using the tools from Ellexus to profile the wide range of applications run by the team and to check them for data access efficiency. In particular, they have been using Mistral. Not all applications are written or maintained in house at Diamond Light Source. The team wanted to see if there were any obvious changes that could be made to the maintained tools to improve performance, or changes to the way that third-party applications are used.