GPU Nodes

The IBEX cluster contains different architectures of GPUs like Turing, Pascal and Volta#. These different GPUs are described (for your source code compilation and job submission) below.

The IBEX cluster has 63 GPU compute nodes (396 GPU cards)  and it’s summarized in Table 1. These various GPUs are accessed by the SLURM scheduling using the constraints "--gres=gpu:<$$$>:<#>”, where: <$$$> is the GPU name and <#> is for number of GPUs.

For example, “--gres=gpu:gtx1080ti:4” is for 4 GTX GPUs.

 

                                        Table 1. List of GPU architectures in IBEX Cluster

Sl. No

GPU Architecture

Available GPU cards per node

Available number of nodes

GPU Memory (per card)

Usable Node Memory^

Constraint for (SLURM) scheduling

1.

Turing:

rtx2080ti#

8

4

12GB

366GB

"--gres=gpu:rtx2080ti:8"

2.

Pascal:

gtx1080ti

4 or 8

12
(4*8 and 8*4)

12GB

246GB
or
366GB

"--gres=gpu:gtx1080ti:4" and "--gres=gpu:gtx1080ti:8"

3.

Pascal:

p100

4

6

16GB

246GB

"--gres=gpu:p100:4"

4.

Pascal:

p6000

2

2

22GB

246GB

"--gres=gpu:p6000:2"

5.

Volta:

v100$,#

4 or 8

38

(8*4 and 30*8)

32GB

366GB

and

745GB

"--gres=gpu:v100:4" and "--gres=gpu:v100:8"

Note: The allocation of CPU memory can be done with `--mem=###G` constraint in SLURM job scripts. The amount of memory depends on the job characteristization. A good starting place would be at least as much as the GPU memory they will use. For example: 2 x v100 GPUs would allocate at least `--mem=64G` for the CPUs.

 

# Newly added into IBEX cluster.

$ The glogin node has a single Quadro K6000 GPU for compilation of the source code.

^ The usable node memory represents the available memory for job execution.

 

Continous efforts has been made for fair share allocation of resources on Ibex, the following partitions has been implemented seamlessly to our users.

gpu_wide for jobs with 4+ gpus per node

gpu_wide24 wide jobs with time limit less than 24 hours

gpu4  for short GPU jobs (less than 4 hours)

Note: Users can't specify those partitions in their scripts. This is done automatically by SLURM.
 
 
 
 
For further info or send us a query using the Contact Us page.

Alternatively, send an email:

  1. Application installation/failure/support: 
  2. System issues/failure/support: