GPUs and Accelerators at the CHPC

GPU Hardware Overview

The CHPC provides GPU devices on the granite (grn), notchpeak (np), kingspeak (kp), lonepeak (lp), and redwood (rw) clusters. Please note that the redwood cluster is part of the Protected Environment (PE).

In Table  1, we provide a summary of the different types of GPU devices (NVIDIA) , i.e.:

• Architecture: the GPU generation or architecture to which a device belongs.
• Device: the model or name of the GPU.
• Compute Capability: defines the hardware features and supported instructions of the device.
• Global Memory: the amount of global memory available on the GPU.
• Area: recommended application area for the GPU.
  • AI : indicates whether the GPU efficiently supports AI workloads.
  • FP64: indicates whether the GPU supports native FP64 floating-point operations.
• Gres Type: The string to be specified with the SLURM --gres option to request this GPU type.
• Cluster:  The cluster(s) where this GPU type is available

 

A detailed description of the features of each GPU type can be found here.

---

MIG (Multi-Instance GPU support)

NVIDIA's MIG technology allows a single physical GPU to be partitioned into multiple isolated instances. Each MIG has its own memory, cache, and compute cores. A single GPU can support up to 7 MIGs.

MIG configurations available on notchpeak and redwood clusters include:

 

Getting access to CHPC's GPUs

To access CHPC GPU resources, you must have an active CHPC account. Access levels to GPUs vary by cluster and node type:

1. General Nodes (CHPC-Owned)

• Free Access: Available to all users on Notchpeak, Kingspeak, Lonepeak, and Redwood.

• Allocation Required: Accessing GPUs on Granite requires a formal allocation. If exhausted, "freecycle" mode is available but is subject to preemption (cancellation).

2. Owner Nodes (Group-Owned)

• Guest Access: All users can use idle owner nodes, but jobs will be preempted if the owner needs the resource.

• Owner Access: Granted to members of the purchasing group. Collaborators gain access via the PI of that owner node.

3. Specialized Access

• One-U Responsible AI (RAI): University researchers with AI projects can request owner access to the RAI GPUs by emailing helpdesk@chpc.utah.edu.

• Shared-Short Partitions:notchpeak-shared-short and redwood-shared-short provide non-preemptible GPU access for small jobs (max 8 cores, 8-hour limit).

Running Compute Jobs with GPUs

Access to GPU-enabled nodes is managed through SLURM job scheduling. You can access these nodes in three ways:

1. By submitting a SLURM batch script using sbatch
2. Interactively using salloc
3. By using the CHPC's Open OnDemand web portal

The former two methods are described in detail in our SLURM documentation page.

Requesting GPUs requires specific SLURM options be set, and some standard options must be configured differently:

---

Slurm Flags to Use When Requesting GPUs:

• --gres=gpu:<gres_type>:<num_devices>
  • --gres=gpu Mandatory: to request actual GPU hardware.

  • :<gres_type> (optional): Specifies the GPU model required (e.g., a100 or rtx6000). If omitted, Slurm will assign any available GPU type in the partition.

  • :<num_devices>(optional): Specifies how many GPUs required. If omitted, the system defaults to one GPU.

    Tip: If you don't care about the specific model and just want the first available GPU, use:

    #SBATCH --gres=gpu:1

• --account & --partitionMandatory: to be set to GPU partitions.

• --qosMandatory: when using the granite cluster.

• --mem Optional: use only if your job has specific memory requirements.

  • Each job is allocated 2 GB of memory per requested core by default. 

Tip: Visit our GitHub to access example Slurm templates that request GPU resources for a variety of clusters and job types.

Cluster-Wide GPU Overview

To view all partitions and their associated features (including their gres_type values) across all CHPC clusters (including both the general and protected environments), use this helpful SLURM alias shortcut :

si2 -M all -a

Note: Open OnDemandsimplifies finding and selecting GPUs. Once a user selects the GPU partition needed, the list of available GPUs associated with that selection is automatically populated.

Best Practices

1.How to select the right GPU

CHPC clusters offer a diverse range of GPU architectures. To select the most efficient resource for your research, consider these four primary factors:

1. Numerical Precision (FP64)

• The Question: Does your code require double-precision operations?

• The Recommendation: For scientific simulations requiring high mathematical accuracy, choose GPUs with native FP64 support (labeled "FP64" in Table 1). AI and deep learning tasks typically rely on FP32 or FP16 and do not require this.

2. Global Memory Requirements

• The Question: How much VRAM does your application need?

• The Recommendation: Memory demand is driven by dataset size and kernel complexity. In Deep Learning, your choice will be influenced by model parameters, batch sizes, and numerical precision. Ensure your selected GPU has enough global memory to prevent "Out of Memory" (OOM) errors.

3. AI Acceleration (Tensor Cores)

• The Question: Can your application leverage Tensor Cores?

• The Recommendation: If you are performing deep learning training or inference, prioritize GPUs with Tensor Core acceleration to significantly reduce computation time. Have a look at the hardware page to identify GPUs that support Tensor Core acceleration.

4. Compute Capability & Architecture

• The Question: Does your software require a specific GPU generation?

• The Recommendation: Some modern libraries require newer architectures. Check the Compute Capability (CC) listed in Table 1 to ensure compatibility with your software requirements.

Once you have identified your requirements, use the following tools to find and target the right hardware:

• Table 1: View the gres_type shown on Table 1.

• si or si2 commands: Run these in your terminal to view available gres_type values and see exactly which nodes currently host those GPUs.

• --gres string: Use the specific identifier (e.g., gpu:a100:1) in your SLURM script to request your chosen hardware.

Pro Tip: If your job doesn't have strict hardware requirements, using a generic --gres=gpu:1 request will often result in a shorter queue time by allowing SLURM to pick the first available device.

2.Which GPUs are currently idle?

• Command: Use the freegpus command to find idle GPUs across all clusters in real-time.

• Output: Displays the hardware type and the count of idle units, formatted as:

SUMMARY: <partition>

type (x count)

For advanced usage and flags, run freegpus --help

Note: A GPU may appear idle but remain unavailable if the node's host memory (RAM) is already fully utilized by other processes.

3. Verifying GPU Allocation and Detection

A non-empty nvidia-smi -L listing and CUDA_VISIBLE_DEVICES output confirm that GPUs have been successfully allocated. 

Verifying with nvidia-smi

NVIDIA provides the NVIDIA System Management Interface program (nvidia-smi), a powerful command-line tool built on the NVIDIA's NVML library. It offers a wide range of options for monitoring and managing GPU resources. Use the following commands to inspect your allocation:

 

Checking Environment Variables

Slurm uses the CUDA_VISIBLE_DEVICES environment variable to restrict your job’s access to specifically assigned GPUs.

• How to check for a GPU: Use the following command in your terminal or within your Slurm script to confirm whether a GPU has been allocated to your job or not:

echo $CUDA_VISIBLE_DEVICES

• How to interpret the command output:

  • List of IDs (e.g., 0,1): Confirms these specific local GPU IDs are allocated to your job.

  • Empty Output: Indicates that no GPUs have been allocated to the current session.

Note: In addition to CUDA_VISIBLE_DEVICES, there are other CUDA-related environment variables that may be relevant. For a complete list, please  look here.

GPU Programming Environment and Performance

NVIDIA CUDA Toolkit

The NVIDIA CUDA Toolkit includes both the CUDA drivers and the CUDA Development Environment. Key components of the development environment include:

Compiling with nvcc

The standard NVIDIA C/C++ compiler is called nvcc.

As shown earlier (see Table  1), we listed all CHPC GPU devices along with their architecture and generation. When compiling CUDA code, it’s important to target the appropriate GPU architectures. Use the following nvcc compiler flags to support all the current GPU architectures at CHPC:

-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 \
-gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 \
-gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 \
-gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90.

For more info on the CUDA compilation and linking flags, please have a look at  CUDA C++ Programming Guide. 

Note: The Maxwell, Pascal, and Volta architectures are now feature-complete. While CUDA 12.x still supports building applications for these architectures, offline compilation and library support will be removed in the next major CUDA Toolkit release.

NVIDIA HPC Software Development Kit (SDK)

We offer the NVIDIA HPC SDK, a comprehensive toolkit for developing high-performance computing (HPC) applications that run on both CPUs and GPUs, supporting standard programming models like Fortran, OpenACC, OpenMP, and MPI.

The NVIDIA HPC SDK can be accessed by running:

module load nvhpc

Compiling using nvc, nvc++, nvfortran

The legacy PGI compilers (pgcc, pgc++, pgf90) have been succeeded by nvc, nvc++, and nvfortran. These modern compilers support C11, C++17, and Fortran 2003/2008, specifically optimized for NVIDIA GPUs and multicore CPUs via OpenACC and OpenMP.

Compilation Example

To compile CUDA code (.cu) while targeting specific GPU architectures (Compute Capabilities), use the following syntax:

 

Essential Flags

• -cuda: Enables CUDA compilation.

• -fast: Triggers aggressive performance optimizations.

• -Minfo=accel: Provides detailed feedback on accelerator optimizations.

• -acc: Required when combining CUDA with OpenACC.

Libraries

Nvidia HPC SDK includes a suite of Math libraries for offloading computations to the GPUs and Communication librariesfor high-speed data exchange between multiple GPUs.

These libraries are organized within the SDK root directory ($NVHPC_ROOT/Linux_x86_64/21.5):

• Math Libraries: math_libs/ (e.g., cuBLAS, cuFFT, cuSOLVER)

• Communication Libraries: comm_libs/ (e.g., NCCL)

To compile and link with, for example, cuBLAS, the full command would look like:

nvc++ my_code.cpp -o my_app \ 
-I$NVHPC_ROOT/Linux_x86_64/21.5/math_libs/include \ 
-L$NVHPC_ROOT/Linux_x86_64/21.5/math_libs/lib64 -Wl,-rpath=$NVROOT/math_libs/lib64 \ 
-lcublas

Debugging

The NVIDIA HPC SDK and CUDA distributions include a terminal-based debugger called cuda-gdb, which operates similarly to the GNU gdb debugger. For more information, see the cuda-gdb documentation.

To enable debugging information:

• For host (CPU) debugging:
  nvc -g -o your_program your_code.c
• For device (GPU) debugging:
  nvc -cuda -g -G -o your_program your_code.cu

To detect out-of-bounds and misaligned memory access errors, use the cuda-memcheck tool. Detailed usage instructions can be found in the cuda-memcheck documentation.

We also license the DDT debugger, which supports CUDA and OpenACC debugging. Due to its user-friendly graphical interface, we recommend DDT for GPU debugging. For guidance on using DDT, please look at our debugging page.

Profiling GPU Performance 

Profiling is essential for identifying bottlenecks, such as low GPU utilization or inefficient shared memory usage. CHPC supports two primary NVIDIA profiling tools:

• Nsight Systems (nsight-sys): For system-wide visualization and timeline analysis.

• Nsight Compute (ncu): For deep-dive kernel-level analysis.

Enabling Profiling:

By default, CHPC uses hardware performance counters for monitoring, which restricts profiling tools and may trigger the following error:

==ERROR== Profiling failed because a driver resource was unavailable.

To resolve this and enable profiling, you must explicitly request GPU profiling in your Slurm job through one of the following methods:  

1. Slurm Job (Batch or Interactive): Add the ,nsight:1 flag to your --gres request (e.g. --gres=gpu,nsight:1)
2. Open OnDemand: When launching an interactive app via OnDemand, locate and check the "Enable GPU profiling" box that appears in the form.

For further guidance, refer to the NVIDIA Nsight Compute Profiling Guide FAQ.