| Table of Contents | ||
|---|---|---|
|
...
NoticeNotice
This guide assumes you have read and setup your environments as per the Using Slurm and Spack documentation.
...
- Single-threaded process requires 1 thread on any node: --ntasks=1 --cpus=per-task=1
- Multi-threaded process requires 16 threads on any node: --ntasks=1 --cpus-per-task=16
- Multi-threaded process requires 16 threads on a node with exclusivity (use only if undoubtedly needed): --ntasks=1 --cpus-per-task=16 --exclusive=1 --cpus-per-task=16 --exclusive
- 16 concurrent but isolated tasks, each task with 1 thread (no multithreading): --ntasks=16 --nodes=1
- 16 concurrent but isolated tasks, each task with 2 threads (multithreading only within each single task): --ntasks=16 --nodes=1 --cpus-per-task=2
The scenarios' options have the following implications
- cpus-per-task: Amount of threads for each task. Supports multithreading and intratask communication.
- ntasks: Amount of isolated threads. Can not communicate with any other task, including tasks in the same job submission.
- nodes: Amount of nodes in which to split tasks between. Should only be > 1 if utilizing MPI or Job Steps.
| Warning |
|---|
Unless using MPI or , slurm "job steps", or isolated concurrently running tasks: ntasks should remain = 1 and --nodes should either not be used or be = 1. The "--cpus-per-task" option should be used for thread allocations. For resource allocations with MPI, see: MPI with Slurm. |
Memory request scenarios
- If you have run the program before on SCU resources you can view the job's memory statistics. From there, you can view the job's Maximum used memory and request a small amount over it for your next iteration of the job. See: Requesting appropriate amounts of resources
- If you have never run the program before; refer to the program's documentation for recommended memory requirements. If none exist, start with 8GB and increase if necessary.
...
Slurm allocates resources based on the listed hardware requirements in your srun command or sbatch script. It is imperative that users neither exaggerate nor underestimate their resource requirements. Slurm provides tools and resources to help users understand their jobs' resource requirements. See Consequences of requesting inappropriate resources.
Memory
RAM is the SCU's most consumed resource. Therefor, the SCU recommends that your jobs request memory no more than 4GB or 20% greater than your previous jobs' maximum used memory (whichever is higher). This command can be used to query your completed jobs' memory allocation vs their maximum memory use in the past 7 days (modify date field as desired):
...
| Anchor | ||||
|---|---|---|---|---|
|
Consequences of requesting inappropriate resources
Slurm allocates and isolates your requested resources. Therefor, if resources are either overestimated, or underestimated, it could detriment the timeliness of your workflows and others'.
- Unused resources are resources that could have been used to run other jobs, including yours. If your job requests 100GB of RAM but only uses 12GB, the unused 88GB will have been made unavailable to all other jobs in the queue; slowing down the cluster's churn rate, and causing your jobs and others' to lag behind.
- Unused resources are counted towards your FairShare allocation, lowering your priority for your future jobs. See FairShare.
- Underestimated resources may cause slurm to kill your jobs. So it is important to neither underestimate nor greatly overestimate the resources needed. Slurm will only kill jobs that exceed their memory requests. Threads are individually allocated and isolated so your jobs cannot exceed their allocated thread count.
...
The above command displays several useful entries. Including the assigned node, resource configurations that slurm understood from the sbatch scripts, path to the sbatch script, stdout, etc.
...
For CPU and Memory statistics, see Requesting appropriate amounts of resources
...
| Anchor | ||||
|---|---|---|---|---|
|
FairShare
...
A large factor in Slurm's FairShare calculation is a lab's amount of priority shares. Slurm uses priority shares to identify labs' expected share of compute on the cluster. A lab's amount of priority shares is equal to their total amount of leased Athena storage x 100. (10T of scratch + 10T of store = 2000 priority shares). This amount of priority shares would be divided by the total amount of priority shares, the result being the lab's expected share of the cluster. Slurm would then prioritize that lab's jobs to use their compute shares until their share of the cluster has been usedutlized.
eg. A lab has procured 50T of storage on athena, which translates to 5000 priority shares. As of when this document was written, slurm would grant this lab priority up to its allocated ~3% of the cluster's use. The lab's jobs would be prioritized at first, but would be deprioritized as jobs run run, until it has consumed its allocated 3 percent effective usage.
...
Fairshare is the resulting score after Slurm calculates the aforementioned factors. Fairshare scores are the deciding factor when it comes to job prioritization and are represented from 0 to 1. 1 0 being the highest lowest priority and 0 1 being the lowesthighest. Below are fairshare scores and their implications.
...
In the output, we see the two factors that are considered in fairshare calculation, cluster usage and priority shares.
Going column Reviewing the output by column, we can the analyze the values of a lab's usage and shares and usage as well as how slurm takes that them into account when calculating fairshare.
- Account: Name of lab in Slurm's database.
- User: When using sshare -a, slurm displays the fairshare calculation for all users in all labs. This can be used for more thorough accounting.
- RawShares: The amount of shares a lab has been allotted. This is reached by calculating total athena storage * 100 = RawShares
- NormShares: This is the percentage of a lab's RawShares to the total shares for the entire cluster. This number is used by slurm to allocate n percentage of cluster usage to said lab.
- RawUsage: The amount of compute usage a lab has run or has requested in their jobs. sshare -l will display a column which lists exact CPU and RAM usages.
- EffectvUsage: The percentage of actual cluster use by the lab. This is compared with the lab's NormShares to determine the lab's FairShare score in the next column. If EffectvUsage is greater than the lab's NormShares, the fairshare will always be < 0.5. The reverse is also true. See FairShare Score
- FairShare: The lab's resulting FairShare score. See FairShare Score.
Halflife
Slurm FairShare calculations have a halflife of 7 days. So if a lab exhausts its allocated usage and its resulting FairShare is = 0, it's FairShare should reset to 0.5 if 7 days pass with no usage.
...