I/O profiling and system telemetry for Singularity containers

The use of containers in high performance computing is growing, particularly Singularity. Developers need to work in reproducible environments that they design themselves and move from platform to platform.

However, the faster the super computer, the more problems can arise if the contents are not performing as they should. We have been carrying out some work to trace containers to make sure they contain what they should. Our recent work has been to look at Singularity containers.


While Docker is still the most widely-used container, some HPC organisations are choosing to use Singularity containers instead. Created by the Lawrence Berkeley National Lab, Singularity is free, cross-platform and open-source and is aimed directly at the HPC industry.

“Singularity is a container system focused on computational performance, artificial intelligence, reproducibility and mobility. When coupled with the I/O profiling tools from Ellexus, Singularity and Ellexus provide the critical insights into the lifecycle and optimisation of a given workload to be used anywhere from on-prem out to the cloud and edge.”

– Gregory Kurtzer, CEO of Sylabs Inc. which is Founder of Singularity and CentO

Here is a detailed article detailing how Docker and Singularity differ, published on HPC – Admin Magazine.

Generating and checking the container

Breeze can generate a list of application dependencies which speeds up the containerization process. However, you should also check application dependencies once the container has been created. This will ensure that you have everything that you need, only what you need and that you have not included the wrong version of any libraries or config files.

Before deploying the container you should check the I/O patterns with Breeze. If you are deploying third party applications, you won’t have the opportunity to remove any bad I/O patterns in the code. However, you will be able to make better decisions about the resources needed to run it. This is especially important for cloud deployments where over-provisioning costs you money.

Continuous monitoring

Mistral is a lightweight production monitoring tool from Ellexus that can record I/O bandwidth, metadata operations and storage performance. A plugin API makes it easy to integrate Mistral with extendable dashboards like Elastic Search, Grafana and Splunk. Ellexus has open sourced the API plugins so you can easily use Mistral as a vehicle to take other measurements within the container.

Use Mistral to check system performance, verify appropriate cloud storage sizing and to make sure that none of your containers are operating outside of expected parameters.

How to trace a Singularity container

Breeze can trace containers to give you all the detail you need about how it is performing. It will show you whether anything is missing for it to be able to run.

The simplest way to trace a singularity container currently is to run the following:

<breezedir>/trace-program.sh -f <outputdir> SINGULARITYENV_LD_PRELOAD="$LD_PRELOAD" singularity run --bind  <breezedir>,<outputdir> <container>

If the Breeze and output directories are already in your home directory, then the bind command can be omitted. Mistral works in the same way.

Get in touch to talk to us about how we can help you plan your container usage.