Using Slurm

What is Slurm?

Slurm (previously Simple Linux Utility for Resource Management), is a modern, open source job scheduler that is highly scaleable and customizable; currently, Slurm is implemented on the majority of the TOP500 supercomputers. Job schedulers enable large numbers of users to fairly and efficiently share large computational resources.

Please see About SCU for more information about our HPC infrastructure.

Slurm partitions 

Slum partitions are cluster specific. All users will have access to the general partitions: scu-cpu and scu-gpu

Of course, the above will be updated as needed; regardless, to see an up-to-date description of all available partitions, using the command sinfo on the login node. For a description of all the nodes' # CPU cores, memory (in Mb), runtime limits, and partition, use this command:

Interactive Session

To request specific numbers of GPUs, you should add your request to your srun/sbatch:  

Below is an example of requesting 1 GPU - can request up to 4 GPUs on a single node

A simple job submission example

To submit computation jobs to the cluster, you need to be logged in to the submit node for the cluster you're using. The following examples assume that you are logged in to the Slurm cluster submit node, curie.

Hello, slurm!

Let's start with a simple submission script. Copy the following into a new file:

hello_slurm.sh

This file is a pretty simple Slurm submission script; let's break it down:

^{#! /bin/bash -l:}

The first line is just the standard shebang line, with the -l indicating that we want a bash login shell. This should be included in every submission script, unless you are an advanced user and have a specific reason not to include this (e.g. you want to use a different shell than bash).

Next, there are several #SBATCH lines. These lines describe the resource allocation we wish to request:

--partition=scu-cpu:

Cluster resources (such as specific nodes, CPUs, memory, GPUs, etc) can be assigned to groups, called partitions. Additionally, the same resources (e.g. a specific node) may belong to multiple cluster partitions. Finally, partitions may be assigned different job priority weights, so that jobs in one partition move through the job queue more quickly than jobs in another partition.

Every job submission script must request a specific partition--otherwise, the default is used. To see what partitions are available on your cluster, click here, or execute the command: sinfo

--nodes=1:

the number of nodes requested

--ntasks=1:

the number of concurrently running tasks. Tasks can be thought of as processes. For this simple serial job, we only need 1 concurrently-running task/process. Also, by default, each task is allocated a single CPU core. For additional information on parallel/multicore environments, click here.

--cpus-per-task=1:

the number of allocated CPUs.

--job-name=test_job:

The job's name--this will appear in the job queue, and is publicly-viewable.

--time=00:00:10:

How long the job will run, formatted as: hours:minutes:seconds. If a specific job duration is not specified, then the default of 10 minutes is used.

--mem=1G

The requested amount of memory for the job; the following units can be used: kibibytes (K),mebibytes (M), gibibytes (G), and tebibytes (T). If a specific amount of memory is not requested, then the partition-dependent default is used

Note: memory is always described in base 2, hence the use of kibibyte (1,024 bytes) instead of kilobyte (1,000 bytes). The curious reader can find additional information here (very optional reading).

Additional job allocation descriptors (#SBATCH) options are described in laters sections, and can also be found here.

After all of the #SBATCH statements, you place the code you actually want to run. In this example, we simply print a couple of statements that describe the job, and redirect this output to the file hello_slurm_output.txt. Some of this information is provided by OS-provided commands (e.g. date and hostname), whereas the rest of the information is provided by Slurm environment variables.

Note: The first line of this code (after the #SBATCH statements) is source ~/.bashrc; this is because Slurm does NOT source your .bashrc, and so you MUST include this line in any submission script that you want to use the environment created by your .bashrc.

Specifically, 3 Slurm environment variables are used in this example: $SLURM_JOB_ID, $SLURM_CLUSTER_NAME, $TMPDIR; these variables store the job's Slurm-assigned ID #, the name of the cluster used, and the local temporary directory your job has access to, respectively. We'll introduce additional Slurm environment variables later; for additional information, click here.

It's very important for your job performance, as well as cluster stability, that intense I/O (e.g. creating many temporary files) is performed in this temporary directory; this is most relevant when using in-house code. Click here for more information on best practices.

Submit the example job

To submit the job described in hello_slurm.sh to the submission queue, execute the following command:

Submit hello_slurm.sh to the scheduler

Where $JOB_ID_NUMBER is the ID number assigned to your job. This job will create the file hello_slurm_output.txt that should contain output similar to this:

hello_slurm_output.txt

Monitoring job status

To check the status of our job, use the following command:

user_ID is your CWID. The output is pretty self-explanatory; specifically, the command displays your jobs':

ID number
Job names
Slurm partition 
Run state (e.g. running, pending, etc.)
Run time
Number of nodes used
Specific nodes used

For additional information, use the command:

This command will display the squeue manual pages (i.e. manpages). Press q to leave the squeue manpage.

If you set up your Slurm environment as described above, you can also use a different command with more advanced output:

Compared to the default squeue command, this has improved formatting and displays the following additional information:

Time limit
GPUs allocated

Interactive sessions

What is an interactive session?

Sometimes, instead of submitting a submission script to the scheduler and waiting for the output, we would like to use allocated resources in an interactive manner (much like you might work locally on your desktop machine). This is especially useful if you wish to use graphical user interfaces (GUIs), debugging and testing code, or if you just have a very simple work flow (ad hoc calculations). In these cases, we can launch an interactive session.

Note: as with any resource allocation, the larger the resource request (e.g. large numbers of CPU cores, large amounts of RAM, etc.), the longer you may have to wait for the resources to be allocated (this is entirely dependent on how much in demand the resources are at that moment).

Starting an interactive session

Execute the following command:

Starting an interactive session

Depending on the resource request, your interactive session might start right away, or it may take a very long time to start (for the above command, the interactive session should almost always start right away).

Here's a break down of the command:

Note: many of these options are the same as what we used in Step 3.1

-n1:

The number of concurrently running tasks. Tasks can be thought of as processes. For this simple serial job, we only need 1 concurrently-running task/process. Also, by default, each task is allocated a single CPU core. For additional information on parallel/multicore environments, click here.

--pty:

Runs task zero in pseudo terminal mode (this is what grants you a terminal)

--partition=scu-cpu:

Cluster resources (such as specific nodes, CPUs, memory, GPUs, etc.) can be assigned to groups, called partitions. Additionally, the same resources (e.g. a specific node) may belong to multiple cluster partitions. Finally, partitions may be assigned different job priority weights, so that jobs in one partition move through the job queue more quickly than jobs in another partition.

Every job submission script must request a specific partition--otherwise, the default is used. To see what partitions are available on your cluster, click here, or execute the command: sinfo

--mem=8G:

The requested amount of memory for the job; the following units can be used: kibibytes (K),mebibytes (M), gibibytes (G), and tebibytes (T). If a specific amount of memory is not requested, then the default is used (8,000M).

Note: memory is always described in base 2, hence the use of kibibyte (1,024 bytes) instead of kilobyte (1,000 bytes). The curious reader can find additional information here (very optional reading).

bash -i:

This tells Slurm to use bash as your shell; the -i tells Slurm to give you an interactive terminal (with prompts). Other shells can also be used (e.g. csh, tcsh), but be sure to use the -i, otherwise an error will occur.

Once you execute this command, eventually the prompt will change (it will reference a specific compute node, as opposed to the submission node); this indicates that you have been allocated the resources and they are accessible via the terminal you are working in. In this example, we have only requested 1 task (which is allocated 1 CPU core by default) and 8 GiB of memory.

Ending an interactive session

To end the interactive session from within the session, just type the following command:

Ending an interactive session

You can also use the following to end an interactive session from outside of the session:

Ending an interactive session from outside of the session

Where $interactive_session_job_number is your session's job-ID #.

In short, be a good HPC citizen!

Interactive session with Graphical User Interface (GUI)

If you need to use a GUI on SCU computational resources, you will need to use an interactive session and X11 forwarding. You'll have to log on to an SCU login node and the cluster submission node with X11 forwarding enabled, and then request an interactive session with X11 forwarding as well. The following commands will connect to the SCU login node pascal, then the Slurm submit node curie, and then request an interactive GUI session:

Request an interactive session with X11 forwarding

To test the session, try the following command:

This opens a window with a graphical clock.

To end the session, close any open windows, and execute the following command in your terminal:

Further reading

• Slurm command reference

• Slurm Rosetta Stone

Related articles