Using Slurm

What is Slurm?

Slurm (previously Simple Linux Utility for Resource Management), is a modern, open source job scheduler that is highly scaleable and customizable; currently, Slurm is implemented on the majority of the TOP500 supercomputers. Job schedulers enable large numbers of users to fairly and efficiently share large computational resources.

Cluster prerequisites

Before being able to take advantage of our computational resources, you must first set up your environment. This is pretty straightforward, but there are a few steps:

SSH access setup

You need to have your SSH keys set up to access cluster resources. If you haven't done this already, please set up your ssh keys.

Environment setup

While still logged in to an SCU login node, run the following:

cat - >> ~/.bashrc <<'EOF'

if [ -n "$SLURM_JOB_ID" ]
then
        source /etc/slurm/slurm_source/slurm_source.sh
fi

alias squeue_long='squeue -o "%.18i %.9P %.8j %.8u %.8T %.10M %.11l %.6b  %.6D %R"'

EOF
source ~/.bashrc

This command simply references a Slurm environment script (if resources have been requested and allocated), and also provides an alias for a more informative squeue command

SCU clusters and job partitions

Available SCU HPC resources

The SCU uses Slurm to manage the following resources:

General purpose cluster:

• The panda cluster (33 nodes): CPU-only cluster intended for general use

CyroEM cluster: 

• The cryoEM cluster (18 nodes): 15 CPU-only nodes, 3 GPU (P100) nodes. Available only for analysis of cryoEM data

PI-specific clusters:

• The Edison cluster (9 GPU nodes):  5 k40m and 4 k80 nodes reserved for the H. Weinstein lab
• node178: GPU (p100) node reserved for the Accardi and Huang labs
• node179: GPU (p100) node reserved for the Boudker lab
• node180: GPU (p100) node reserved for the Blanchard lab
• cantley-node0[1-2] (2 nodes): GPU (V100) nodes reserved for the Cantley lab 

All jobs, except those submitted the Edison cluster, should be submitted via our Slurm submission node: curie.pbtech. Jobs submitted to the Edison cluster should be submitted from its submission node, edison-mgmt.pbtech.

Please see About SCU for more information about our HPC infrastructure.

Slurm partitions

Slurm groups nodes into sets referred to as 'partitions'. The above resources belong to one or more Slurm partitions, with each partition possessing its own unique job submission rules. Some nodes belong to multiple partitions because this affords the SCU the configuration flexibility needed to ensure fair allocation of managed resources.

Panda cluster partitions:

• panda: 33 CPU-only nodes, 7-day runtime limit, only 50 jobs allowed to run concurrently
• panda_array: 33 CPU-only nodes, 7-day runtime limit, up to 100,000 jobs allowed to run concurrently 

CryoEM cluster:

• cryo-cpu: 15 CPU-only nodes, 2-day runtime limit
• cryo-gpu 3 GPU nodes, 2-day runtime limit

Edison cluster:

• edison: 9 GPU nodes, 2-day runtime limit
• edison_k40m: 5 GPU (k40m) nodes, 2-day runtime limit
• edison_k80: 4 GPU (k80) nodes, 2-day runtime limit

A simple job submission example

To submit computation jobs to the cluster, you need to be logged in to the submit node for the cluster you're using. The following examples assume that you are logged in to the Slurm cluster submit node, curie.

Hello, slurm!

Let's start with a simple submission script. Copy the following into a new file:

#! /bin/bash -l

#SBATCH --partition=panda   # cluster-specific 
#SBATCH --nodes=1  
#SBATCH --ntasks=1 
#SBATCH --job-name=hello_slurm
#SBATCH --time=00:02:00   # HH/MM/SS
#SBATCH --mem=1G   # memory requested, units available: K,M,G,T

source ~/.bashrc

echo "Starting at:" `date` >> hello_slurm_output.txt
sleep 30
echo "This is job #:" $SLURM_JOB_ID >> hello_slurm_output.txt
echo "Running on node:" `hostname` >> hello_slurm_output.txt
echo "Running on cluster:" $SLURM_CLUSTER_NAME >> hello_slurm_output.txt
echo "This job was assigned the temporary (local) directory:" $TMPDIR >> hello_slurm_output.txt

exit

This file is a pretty simple Slurm submission script; let's break it down:

#! /bin/bash -l:

The first line is just the standard shebang line, with the -l indicating that we want a bash login shell. This should be included in every submission script, unless you are an advanced user and have a specific reason not to include this (e.g. you want to use a different shell than bash).

Next, there are several #SBATCH lines. These lines describe the resource allocation we wish to request:

--partition=panda:

Cluster resources (such as specific nodes, CPUs, memory, GPUs, etc) can be assigned to groups, called partitions. Additionally, the same resources (e.g. a specific node) may belong to multiple cluster partitions. Finally, partitions may be assigned different job priority weights, so that jobs in one partition move through the job queue more quickly than jobs in another partition.

Every job submission script must request a specific partition--otherwise, the default is used. To see what partitions are available on your cluster, click here, or execute the command: sinfo

--nodes=1:

the number of nodes requested

--ntasks=1:

the number of concurrently running tasks. Tasks can be thought of as processes; this is explained in more detail in Advanced Job Submissions. For this simple serial job, we only need 1 concurrently-running task/process. Also, by default, each task is allocated a single CPU core. For additional information on parallel/multicore environments, click here.

--job-name=test_job:

The job's name--this will appear in the job queue, and is publicly-viewable.

--time=00:00:10:

How long the job will run, formatted as: hours:minutes:seconds. If a specific job duration is not specified, then the default of 10 minutes is used.

--mem=1G

The requested amount of memory for the job; the following units can be used: kibibytes (K),mebibytes (M), gibibytes (G), and tebibytes (T). If a specific amount of memory is not requested, then the partition-dependent default is used

Note: memory is always described in base 2, hence the use of kibibyte (1,024 bytes) instead of kilobyte (1,000 bytes). The curious reader can find additional information here (very optional reading).

Additional job allocation descriptors (#SBATCH) options are described in laters sections, and can also be found here.

After all of the #SBATCH statements, you place the code you actually want to run. In this example, we simply print a couple of statements that describe the job, and redirect this output to the file hello_slurm_output.txt. Some of this information is provided by OS-provided commands (e.g. date and hostname), whereas the rest of the information is provided by Slurm environment variables.

Note: The first line of this code (after the #SBATCH statements) is source ~/.bashrc; this is because Slurm does NOT source your .bashrc, and so you MUST include this line in any submission script that you want to use the environment created by your .bashrc.

Specifically, 3 Slurm environment variables are used in this example: $SLURM_JOB_ID, $SLURM_CLUSTER_NAME, $TMPDIR; these variables store the job's Slurm-assigned ID #, the name of the cluster used, and the local temporary directory your job has access to, respectively. We'll introduce additional Slurm environment variables later; for additional information, click here.

It's very important for your job performance, as well as cluster stability, that intense I/O (e.g. creating many temporary files) is performed in this temporary directory; this is most relevant when using in-house code. Click here for more information on best practices.

Submit the example job

To submit the job described in hello_slurm.sh to the submission queue, execute the following command:

sbatch hello_slurm.sh
 
> Submitted batch job $JOB_ID_NUMBER

Where $JOB_ID_NUMBER is the ID number assigned to your job. This job will create the file hello_slurm_output.txt that should contain output similar to this:

Starting at Fri Feb 9 13:12:13 EST 2018
This is job 6602
Running on node: node115.panda.pbtech
Running on cluster: panda
This job was assigned the temporary (local) directory: /scratchLocal/CWID_6602

Monitoring job status

To check the status of our job, use the following command:

squeue -u user_ID

user_ID is your CWID. The output is pretty self-explanatory; specifically, the command displays your jobs':

ID number
Job names
Slurm partition 
Run state (e.g. running, pending, etc.)
Run time
Number of nodes used
Specific nodes used

For additional information, use the command:

man squeue

This command will display the squeue manual pages (i.e. manpages). Press q to leave the squeue manpage.

If you set up your Slurm environment as described above, you can also use a different command with more advanced output:

squeue_long -u user_ID

Compared to the default squeue command, this has improved formatting and displays the following additional information:

Time limit
GPUs allocated

Interactive sessions

What is an interactive session?

Sometimes, instead of submitting a submission script to the scheduler and waiting for the output, we would like to use allocated resources in an interactive manner (much like you might work locally on your desktop machine). This is especially useful if you wish to use graphical user interfaces (GUIs), debugging and testing code, or if you just have a very simple work flow (ad hoc calculations). In these cases, we can launch an interactive session.

Note: as with any resource allocation, the larger the resource request (e.g. large numbers of CPU cores, large amounts of RAM, etc.), the longer you may have to wait for the resources to be allocated (this is entirely dependent on how much in demand the resources are at that moment).

Starting an interactive session

Execute the following command:

srun -n1 --pty --partition=panda --mem=8G bash -i
 
> your_CWID@nodeXXX ~ $

Depending on the resource request, your interactive session might start right away, or it may take a very long time to start (for the above command, the interactive session should almost always start right away).

Here's a break down of the command:

Note: many of these options are the same as what we used in Step 3.1

-n1:

The number of concurrently running tasks. Tasks can be thought of as processes; this is explained in more detail in Advanced Job Submissions. For this simple serial job, we only need 1 concurrently-running task/process. Also, by default, each task is allocated a single CPU core. For additional information on parallel/multicore environments, click here.

--pty:

Runs task zero in pseudo terminal mode (this is what grants you a terminal)

--partition=panda:

Cluster resources (such as specific nodes, CPUs, memory, GPUs, etc.) can be assigned to groups, called partitions. Additionally, the same resources (e.g. a specific node) may belong to multiple cluster partitions. Finally, partitions may be assigned different job priority weights, so that jobs in one partition move through the job queue more quickly than jobs in another partition.

Every job submission script must request a specific partition--otherwise, the default is used. To see what partitions are available on your cluster, click here, or execute the command: sinfo

--mem=8G:

The requested amount of memory for the job; the following units can be used: kibibytes (K),mebibytes (M), gibibytes (G), and tebibytes (T). If a specific amount of memory is not requested, then the default is used (8,000M).

Note: memory is always described in base 2, hence the use of kibibyte (1,024 bytes) instead of kilobyte (1,000 bytes). The curious reader can find additional information here (very optional reading).

bash -i:

This tells Slurm to use bash as your shell; the -i tells Slurm to give you an interactive terminal (with prompts). Other shells can also be used (e.g. csh, tcsh), but be sure to use the -i, otherwise an error will occur.

Once you execute this command, eventually the prompt will change (it will reference a specific compute node, as opposed to the submission node); this indicates that you have been allocated the resources and they are accessible via the terminal you are working in. In this example, we have only requested 1 task (which is allocated 1 CPU core by default) and 8 GiB of memory.

Ending an interactive session

To end the interactive session from within the session, just type the following command:

exit

You can also use the following to end an interactive session from outside of the session:

scancel $interactive_session_job_number

Where $interactive_session_job_number is your session's job-ID #.

In short, be a good HPC citizen!

Interactive session with Graphical User Interface (GUI)

If you need to use a GUI on SCU computational resources, you will need to use an interactive session and X11 forwarding. You'll have to log on to an SCU login node and the cluster submission node with X11 forwarding enabled, and then request an interactive session with X11 forwarding as well. The following commands will connect to the SCU login node pascal, then the Slurm submit node curie, and then request an interactive GUI session:

ssh -X pascal
 
ssh -X curie
 
srun --x11 -n1 --pty --partition=panda --mem=8G bash -i

To test the session, try the following command:

xclock

This opens a window with a graphical clock.

To end the session, close any open windows, and execute the following command in your terminal:

exit

Further reading

• Slurm command reference
• Slurm Rosetta Stone

Related articles

• Page:
  Using Slurm
• Page:
  Cryo-EM Cluster Resources
• Page:
  Running Docker images on Singularity
• Page:
  Slurm - In depth
• Page:
  Computation