Have we reached a point of inflection for Singularity?

Singularity, a tool for creating and running containers, is slowly but surely making a real impact on the high-performance computing (HPC) sector. The desire among users for a Singularity User Group was demonstrated by the high level of attendance; 92 attendees convened for the inaugural user group in San Diego.

Ellexus was hoping to win the ‘travelled furthest to be here’ award, but container enthusiasts travelled all the way from Australia and Eastern Europe to be there. We expected a good turn out from Sylabs and from the hosts, San Diego Supercomputing Center, but there was also good representation from many of the US National Labs, biotech companies, finance, energy, machine learning pioneers and partner companies.

Audience poll

The Singularity User Group started with a poll of attendance showing a good split between users, administrators and developers. This demonstrates how container-use is infiltrating many areas of computing and engineering.

The audience poll also showed an interesting mix of uses for Singularity.

Containers for AI workloads

User group sponsors Lenovo were promoting their LiCo project that provides an easy way to manage containers for AI workloads, but already there is a substantial eco-system of support from many other companies and projects large and small.

Profiling and 3rd party acceleration

One of the announcements for the next version of Singularity is a plugin framework that will allow verified third party tools to be installed alongside Singularity. This opens the door to a whole host of admin and developer tools for enabling profiling tools and accelerators to be attached to a container via a clean API.

Technical know-how

Other talks during the Singularity User Group included a mix of technical tutorials for those wanting to know more about deploying, running or developing the tool. User presentations gave information about using Singularity in the cloud, tuning genome pipelines in containers and how to test for 15TB daily stream from a telescope that isn’t built yet.

Ellexus CEO Rosemary Francis gave a talk on system telemetry and I/O profiling for cloud migration and containers, an area that is going to increase in importance as the use of both cloud and containers grows.

Many organisations waste a huge amount of time doing bad I/O, which becomes extremely apparent in a cloud environment where you pay per time. Similarly, organisations waste money by paying for more or faster storage than they need. The answer? Profiling your applications so you know what they are doing, how they are performing and all their dependencies. 

We recently published a whitepaper on work Ellexus carried out with the Wellcome Sanger Institute to tune their genome pipelines. Through I/O profiling, the institute reduced their cloud spend by 10% and cut run time from 32 hours to 18 hours.

A look ahead

As we have said in the past, the use of cloud and containers is only going to grow. Hybrid cloud environments are emerging that will make tools such as Singularity vital.

Overall, the first Singularity User Group was a great event. Good job guys – we’re already looking forward to next year!