Containers

Singularity Extreme Mobility of Computing

Singularity is a technology to run software containers (think Docker containers) in an High-Performance Computing environment. The most striking difference in comparison to Docker is that running these singularity containers do not need root rights.

Possible uses for Singularity on Ibex:

Run an application that was built for a different distribution of Linux than the host OS.
Reproduce an environment to run a workflow created by someone else.
Run a series of applications (a 'pipeline') that includes applications built on different platforms.
Run an application from Singularity Hub or Docker Hub on Ibex without actually installing anything.

Index

Creating Singularity containers
Simplify the preparation of Singularity containers
Binding external directories
Interactive Singularity containers
Singularity containers in serial batch jobs
MPI code within a single node (using container MPI)
MPI code over multiple nodes (using host and container MPI)

Useful Links

Additional Learning Resources

Singularity Basics videos by Sylabs

Please Note:
Singularity gives you the ability to install and run applications on your own Linux environment with your own customized software stack. With this ability comes the added responsibility of managing your own Linux environment. While the KSL-CS staff can provide guidance on how to create and use singularity containers, we do not have the resources to manage containers for individual users. If you decide to use Singularity, it is your responsibility to build and manage your own containers.

Creating Singularity containers

To use Singularity on Ibex, you either need to build your own Singularity container or use one created by someone else.

The build command accepts a target as input and produces a container as output.

The target defines the method that build uses to create the container. It can be one of the following:

URI beginning with library:// to build from the Container Library.
URI beginning with docker:// to build from Docker Hub.
URI beginning with shub:// to build from Singularity Hub.
PATH to a existing container on your local machine.
PATH to a directory to build from a sandbox.
PATH to a Singularity definition file.

build can produce containers in two different formats that can be specified as follows.

Compressed read-only Singularity Image File (SIF) format suitable for production (default).
Writable (ch)root directory called a sandbox for interactive development ( --sandbox option).

Because build can accept an existing container as a target and create a container in either supported format you can convert existing containers from one format to another.

If you choose to build the image from a Singularity definition file, you should have root access and these are your possible options:

Use fakeroot option available on Ibex compute nodes.
Use remote option to build on Singularity Container Services.
If you have a Linux system to which you have root (admin) access, you can install Singularity and build your Singularity container there.
If you don't have a Linux system you could easily install one in a virtual machine using software like VirtualBox, Vagrant, VMware, or Parallels.
You can allocate a cloud instance, to which you will have root access. Install Singularity and build your Singularity container there.

Example to build an image from DockerHub:

[user@dbn303-16-l]$ module load singularity
Loading module for Singularity
Singularity 3.5 modules now loaded
[user@dbn503-33-r$ singularity build lolcow.sif docker://godlovedc/lolcow
INFO:    Starting build...
Getting image source signatures
Copying blob 9fb6c798fa41 done
Copying blob 3b61febd4aef done
Copying blob 9d99b9777eb0 done
Copying blob d010c8cf75d7 done
Copying blob 7fac07fb303e done
Copying blob 8e860504ff1e done
Copying config 73d5b1025f done
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: lolcow.sif

Example to build an image from recipe file using fakeroot:

Notice you can't use this feature on login nodes.

 
[user@dbn503-33-r:]$ srun -N1 t 01:00:00 --pty /bin/bash
[user@cn605-16-l$ module load singularity
Loading module for Singularity
Singularity 3.5 modules now loaded
[user@cn605-16-l:containers]$ cat lolcow.def
BootStrap: docker
From: godlovedc/lolco
%post
chown root.root /tmp/
apt update -y && apt upgrade -y && apt install vim -y
[user@cn605-16-l]$ singularity build --fakeroot lolcow.sif-2 lolcow.def
INFO:    Starting build...
Getting image source signatures
Writing manifest to image destination
Storing signatures
INFO:    Running post scriptlet
INFO:    Creating SIF file...
INFO:    Build complete: lolcow.sif-2

Simplify the preparation of singularity containers

A Singularity Definition File is like a set of blueprints explaining how to build a custom container. It is the same like Dockerfile but in different syntax. It includes specifics about the base OS to build or the base container to start from, software to install, environment variables to set at runtime, files to add from the host system, and container metadata. Unfortunately writing Singularity recipe or Dockerfile is not the straight forward task, sometimes it require advanced linux knowledge to write them. Hence we are using HPCCM an open source tool to make it easier to generate container specification files.

Example to use HPCCM on ibex:

First load hpccm module (available on ilogin)

[user@cn509-02-r:~]$ module load hpccm
Loading module for hpccm
hpccm 20.2.0 module now loaded
[user@cn509-02-r:~]$

Then write HPCCM high level Python recipe, that will be translated to Dockerfile or Singularity recipe; for example this is an recipe for openmpi image:

[user@cn509-02-r:~]$cat mpi_3.0.0_ubuntu18.py
"""
MPI Bandwidth
Contents:
  Ubuntu 18.04
  GNU compilers (upstream)
  Mellanox OFED
  OpenMPI version 3.0.0
"""

Stage0 += comment(__doc__, reformat=False)

# CentOS base image
Stage0 += baseimage(image='ubuntu:18.04')

# GNU compilers
Stage0 += gnu()

# Mellanox OFED
Stage0 += mlnx_ofed()

# OpenMPI
Stage0 += openmpi(configure_opts=['--with-slurm'], infiniband=False, cuda=False, version='3.0.0')

# MPI Hello World
Stage0 += copy(src='hello_world.c', dest='/var/tmp/hello_world.c')
Stage0 += shell(commands=['mpicc -o /usr/local/bin/hello_world /var/tmp/hello_world.c'])

Then you are ready to convert this python recipe to Singularity recipe or Dockerfile:

[user@cn509-02-r:~]$ hpccm --format singularity --recipe ./mpi_3.0.0_ubuntu18.py --singularity-version=3.4 > mpi_3.0.0_ubuntu18.def

Now you are ready to build customized singularity image

[user@cn509-02-r:]$ srun -N1 t 01:00:00 --pty /bin/bash
[user@cn605-16-l$ module load singularity
Loading module for Singularity
Singularity 3.4 modules now loaded
[user@cn605-16-l:]$ singularity build --fakeroot mpi_3.0.0_ubuntu18.sif mpi_3.0.0_ubuntu18.def
INFO:    Starting build...
Getting image source signatures
Copying blob ab5ef0e58194 skipped: already exists
Copying config 0a7908e1b9 done
Writing manifest to image destination
NFO:    Running post scriptlet
INFO:    Creating SIF file...
INFO:    Build complete: mpi_3.0.0_ubuntu18.sif

Binding external directories

Binding a directory to your Singularity container allows you to access files in a host system directory from within your container. By default, Singularity will bind your $HOME directory (along with a few other directories such as /tmp and /dev). You can also bind other directories into your Singularity container yourself.

The Singularity action commands (run, exec and shell will accept the --bind/-B command-line option to specify bind paths, and will also honor the $SINGULARITY_BIND or $SINGULARITY_BINDPATH environment variable. The argument for this option is a comma-delimited string of bind path specifications in the format src[:dest[:opts]], where src and dest are paths outside and inside of the container respectively. If dest is not given, it is set equal tosrc. Mount options (opts) may be specified as ro (read-only) or rw (read/write, which is the default). The --bind/-B option can be specified multiple times, or a comma-delimited string of bind path specifications can be used.

$ singularity shell --bind /home/$USER,/ibex/scratch/$USER my-container.sif

Or using the environment variable:

$ export SINGULARITY_BINDPATH="/home/$USER,/ibex/scratch/$USER"
$ singularity shell my-container.sif

Interactive Singularity containers

To run a Singularity container image on Ibex interactively,you need to allocate an interactive session and load the Singularity module. In this sample session (user input in bold), an Ubuntu 18.04.2 Singularity container is downloaded and run from Docker Hub. If you want to run a local Singularity container instead of downloading one, just replace the DockerHub URL with the path to your container image file.

[user@dbn503-35-r:~]$ srun -N1 -t 1:0:0 --pty bash
srun: job 5696878 queued and waiting for resources
srun: job 5696878 has been allocated resources
[user@cn603-12-r:~]$
[user@cn603-12-r:/ibex/scratch/user/]$ module load singularity
Loading module for Singularity
Singularity 3.1.1 modules now loaded
[user@cn603-12-r:/ibex/scratch/user/]$ singularity shell docker://ubuntu
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
Singularity ubuntu_latest.sif:/ibex/scratch/user/>
Singularity ubuntu_latest.sif:/ibex/scratch/user/> cat /etc/os-release
cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Singularity ubuntu_latest.sif:/ibex/scratch/user/>  exit
[user@cn603-12-r:~]$ exit

Note that you need to exit your Singularity container as well as your allocated interactive Slurm session when you are done.

Singularity containers in serial batch jobs

Create the singularity image (e.g. serial.sif) from dockerhub , using the command:

singularity build serial.sif docker://chuanwen/cowsay

In the same directory as the singularity image, create a Slurm submission file (e.g. submit_serial.sh) with the contents:

#!/bin/sh
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --partition=batch
#SBATCH --job-name=serial
#SBATCH --output=log_slurm.txt

module load singularity/3.5.0
singularity exec serial.sif cowsay 'Hello!'

Submit a job, using the command

sbatch submit_serial.sh

the job finishes, the output (in the log_slurm.txt) will look like

 ________
< Hello! >
 --------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

MPI code within a single node (using container MPI)

On your scratch create a C file (e.g. hello_world.c) with the contents:

#include 
#include 
#include 
  
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
  
// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  
//Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);

// Get the id of the core
int core_id;
core_id = sched_getcpu();

// Print off a hello world message
printf("host=%s, size=%d, rank=%d, core=%d\n", processor_name, world_size, world_rank, core_id);

// Finalize the MPI environment.
MPI_Finalize();
 
}

Create a singularity definition file (e.g. mpi_container.def) with the contents

BootStrap: docker
From: centos:7

%post
  yum -y update
  yum -y install openmpi3-devel
  ln -s /usr/lib64/openmpi3/bin/mpicc /usr/bin/mpicc
  ln -s /usr/lib64/openmpi3/bin/mpirun /usr/bin/mpirun

%files
  hello_world.c /var/tmp/hello_world.c

%post
  mpicc -o /usr/local/bin/hello_world /var/tmp/hello_world.c

Create the singularity image (e.g. mpi_container.sif) from the singularity definition file (mpi_container.def), using the fakeroot feature:

[user@dbn503-33-r:]$ srun -N1 t 01:00:00 --pty /bin/bash
[user@cn605-16-l$ module load singularity/3.5
Loading module for Singularity
Singularity 3.5 modules now loaded
[user@cn605-16-l]$ singularity build --fakeroot mpi_container.sif mpi_container.def

In the same directory as the singularity image, create a Slurm submission file (e.g. submit_mpi_container.sh) with the contents:

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=20
#SBATCH --partition=batch
#SBATCH --job-name=mpi_container
#SBATCH --output=log_slurm.txt
module load singularity/3.5
singularity exec mpi_container.sif /usr/bin/mpirun -np 20 /usr/local/bin/hello_world

Submit a job, using the command.

sbatch submit_mpi_container.sh

Once the job finishes, the output (in the log_slurm.txt) will look like:

host=cn605-16-r, size=20, rank=1,  core=1
host=cn605-16-r, size=20, rank=2,  core=2
host=cn605-16-r, size=20, rank=3,  core=3
host=cn605-16-r, size=20, rank=4,  core=4
host=cn605-16-r, size=20, rank=5,  core=5
host=cn605-16-r, size=20, rank=6,  core=6
host=cn605-16-r, size=20, rank=7,  core=7
host=cn605-16-r, size=20, rank=8,  core=8
host=cn605-16-r, size=20, rank=9,  core=9
host=cn605-16-r, size=20, rank=10, core=10
host=cn605-16-r, size=20, rank=11, core=11
host=cn605-16-r, size=20, rank=12, core=12
host=cn605-16-r, size=20, rank=13, core=13
host=cn605-16-r, size=20, rank=14, core=14
host=cn605-16-r, size=20, rank=15, core=15
host=cn605-16-r, size=20, rank=16, core=16
host=cn605-16-r, size=20, rank=17, core=17
host=cn605-16-r, size=20, rank=18, core=18
host=cn605-16-r, size=20, rank=19, core=19
host=cn605-16-r, size=20, rank=20, core=20

MPI code over multiple nodes (using host and container MPI)

To run singularity containers with MPI over multiple nodes, the host MPI and the container MPI should be compatible, hence we will create customized container image using hpccm

We will use the same C file (hello_world.c) above and the following hpccm recipe:

[user@cn509-02-r:~]$cat mpi_host_container.py
"""
MPI Bandwidth
Contents:
  CentOS 7
  GNU compilers (upstream)
  Mellanox OFED
  OpenMPI version 3.1.2
"""

Stage0 += comment(__doc__, reformat=False)

# CentOS base image
Stage0 += baseimage(image='centos:7')

# GNU compilers
Stage0 += gnu()

# Mellanox OFED
Stage0 += mlnx_ofed()

# OpenMPI
Stage0 += openmpi(configure_opts=['--with-slurm'], infiniband=False, cuda=False, version='3.1.2')

# MPI Hello World
Stage0 += copy(src='hello_world.c', dest='/var/tmp/hello_world.c')
Stage0 += shell(commands=['mpicc -o /usr/local/bin/hello_world /var/tmp/hello_world.c'])

Create the singularity definition file (mpi_host_container.def) from the HPCCM recipe file (mpi_host_container.py)

[user@cn509-02-r:~]$ module load hpccm
Loading module for hpccm
hpccm 20.2.0 module now loaded
[user@cn509-02-r:~]$ hpccm --format singularity --recipe ./mpi_host_container.py --singularity-version=3.5 > mpi_host_container.def

Then you can use the above examples to create the singularity image (mpi_host_container.sif) from the singularity definition file (mpi_host_container.def).

Afterthat, in the same directory as the singularity image (e.g. /ibex/scratch), create a Slurm submission file (e.g. submit_mpi_host_container.sh) with the contents

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks=40
#SBATCH --ntasks-per-node=20
#SBATCH --partition=batch
#SBATCH --job-name=mpi_host_container
#SBATCH --output=log_slurm.txt
module load singularity/3.5
module load openmpi/3.1.2/gnu-8.2.0
mpirun -np 40 singularity exec mpi_host_container.sif /usr/local/bin/hello_world