AI Cluster

Overview

In collaboration with Dr. Mert Sabuncu from Radiology, the ITS team has established the framework for a new high-performance computing (HPC) cluster dedicated to AI/ML type workflows, like training neural networks for imaging, LLMs and so on.

This cluster features high-memory nodes, Nvidia GPU servers (A100, A40 and L40), InfiniBand interconnect, and specialized storage designed for AI workloads.

Login to the AI cluster

AI cluster is accessible via terminal SSH sessions. You need to be connecting from the WCM network, or have VPN installed and enabled. Replace <cwid> with your credentials.

ssh <cwid>@ai-login01.med.cornell.edu
# or
ssh <cwid>@ai-login02.med.cornell.edu

Once logged on:

Last login: Fri Jan  3 11:35:53 2025 from 157.000.00.00
<cwid>@ai-login01:~$
<cwid>@ai-login01:~$ pwd
/home/<cwid>
<cwid>@ai-login01:~$ 

Storage

AI cluster has the following storage systems configured:

Common File Management

# List all files and directories in the scratch directory
ls /midtier/labname/scratch/

# Navigate to a specific subdirectory
cd /midtier/labname/scratch/cwid

# Copy a file from the current directory to another directory
cp data.txt /midtier/labname/scratch/cwid/

# Move the copied file to a different directory
mv /midtier/labname/scratch/cwid/data.txt /midtier/labname/scratch/backup/

# Create a new directory
mkdir /midtier/labname/scratch/cwid/new_project/

Code example

To illustrate how to run computational jobs, consider the following toy problem, that is implemented in C. It estimates value of π using a random sampling method:

1. Generate random points (x, y) in a unit square (0 ≤ x, y ≤ 1).

2. Count how many points fall inside the quarter circle (x² + y² ≤ 1).

3. The ratio of points inside the circle to total points approximates π: π≈4 × total points / points inside circle

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <omp.h>

long long monte_carlo_pi(long long num_samples, int num_threads) {
    long long inside_circle = 0;
    #pragma omp parallel num_threads(num_threads)
    {
        unsigned int seed = 1234 + omp_get_thread_num();  // Unique seed for each thread
        long long local_count = 0;

        #pragma omp for
        for (long long i = 0; i < num_samples; i++) {
            double x = (double)rand_r(&seed) / RAND_MAX;
            double y = (double)rand_r(&seed) / RAND_MAX;
            if (x * x + y * y <= 1.0) {
                local_count++;
            }
        }

        #pragma omp atomic
        inside_circle += local_count;
    }
    return inside_circle;
}

int main(int argc, char *argv[]) {
    if (argc != 3) {
        printf("Usage: %s <num_samples> <num_threads>\n", argv[0]);
        return 1;
    }

    long long num_samples = atoll(argv[1]);  // Number of random points
    int num_threads = atoi(argv[2]);         // Number of OpenMP threads

    double start_time = omp_get_wtime();
    long long inside_circle = monte_carlo_pi(num_samples, num_threads);
    double end_time = omp_get_wtime();

    double pi_approx = 4.0 * (double)inside_circle / num_samples;

    printf("Approximated π: %.15f\n", pi_approx);
    printf("Error: %.15f\n", fabs(pi_approx - 3.141592653589793));
    printf("Execution Time: %.6f seconds\n", end_time - start_time);

    return 0;
}

Save this code into a file code.c and compile using a command

gcc -fopenmp code.c

it will generate an executable a.out that we will use to illustrate how to submit jobs

Running jobs

Computational jobs on the AI cluster are managed with a SLURM job manager. We provide an in-depth tutorial on how to use SLURM <placeholder>, but some basic examples that are immediately applicable on the AI cluster will be discussed in this section.

Important notice:

Running your application code directly without submitting it through the scheduler is prohibited. Login nodes are shared resources and they are reserved for light tasks like file management and job submission. Running heavy computations on login nodes can degrade performance for all users. Instead, please submit your compute jobs to the appropriate SLURM queue, which is designed to handle such workloads efficiently.

There are two mechanisms to run SLURM jobs: “batch” and “interactive”. Interactive jobs are an inefficient way to utilize the cluster. By their nature, these jobs require the system to wait for user input, leaving the allocated resources idle during those periods. Since HPC clusters are designed to maximize resource utilization and efficiency, having nodes sit idle while still consuming CPU, memory, or GPU resources is counterproductive.

For example:

• If you're running an interactive session and step away or take time to analyze output, the allocated resources remain reserved but unused.

• This idle time adds up across multiple users, leading to significant underutilization of the cluster.

Because of these considerations we highly recommend that users execute as much of their computations in the batch mode so that WCM’s research community can make the most of the cluster's capabilities.

SLURM Batch job example

Here is an example of the batch script that can be ran on the cluster. In this script we are requesting a single node, with 4 CPU cores, used in an SMP mode.

#!/bin/bash

#SBATCH --job-name=testjup   # give your job a name
#SBATCH --nodes=1            # asking for 1 compute node
#SBATCH --ntasks=1           # 1 task
#SBATCH --cpus-per-task=4    # 4 CPU cores per task, so 4 cores in total         
#SBATCH --time=00:30:00      # set this time according to your need, 30 minutes here
#SBATCH --mem=8GB            # request appropriate amount of RAM
##SBATCH --gres=gpu:1        # if you need to use a GPU, note that this line is commented out
#SBATCH -p <partition_name>  # specify your partition

SLURM interactive job example

Jupyter job

Monitoring job status