Neser - Pre/Post Processing System

top

On September 16th KSL introduces a brand-new service to its researchers. A computing cluster called "Neser" is offered to Shaheen users for pre- and post-processing data, allowing them to input data and process the output in a very fast manner. By providing a swift and efficient access to Shaheen storage, Neser distinguishes itself from a general-purpose cluster IBEX. Previously, Shaheen users needed to transfer their data to local computers first for analysis, which was an inconvenient and time-consuming process. With Neser as its sidekick, the need to move large amounts of data for pre-and post-processing is reduced. The ease of retrieving Shaheen data will enable faster scientific results for the High-Performance Computing community.

Neser System Overview

top

Cray CS500 Cluster composed of:

  • Red Hat Enterprise Linux Server release 8.6
  • 792 physical compute cores (1584 logical cores with Hyper-Threading enabled)
  • 16 Nvidia Tesla K80
  • 4.9 TB of compute memory
  • Peak performance about 50 TF
  • Connected to Shaheen parallel Lustre file system for /project and /scratch
  • 20 compute nodes
  • 2 login nodes
  • Configured with FDR Infiniband.

Compute Node Details

top

19 nodes composed of:

  • Two Skylake Intel Xeon(R) Gold 6138 CPU 2.0 GHz
  • 40 physical cores on each node (80 logical cores if using Hyper-Threading)
  • 2 nodes are equipped with 768 GB of memory, 5 nodes with 256 GB of memory and 12 nodes with 192 GB of memory
  • 12TB of local disk per node

One node composed of 2 Intel Xeon(R) Haswell 16 cores 2.3 GHz with 256 GB of RAM and 16 Nvidia K80 Tesla cards.

Login Node Details

top

Two login nodes ( neser1, neser2) with two Skylake Intel Xeon(R) Gold 6134 CPU 3.4 GHz

Access and Allocation

top

Access is granted via SSH using double authentication with OTP: ssh neser.hpc.kaust.edu.sa

Access to Neser is granted to existing Shaheen users upon request, justifying their use of Neser for pre/post processing. Users should contact the KSL helpdesk (help@hpc.kaust.edu.sa) with a short description of their proposed workflow. The new cluster uses the same core-hour accounting system as Shaheen, hence it is important to be a member of an active project.

Quick Start Guide

top

Compiling

How to load the modules for MPI:

  • module load PrgEnv-cray/1.0.4
  • export IMPI_VERSION=2018.2.199
  • module load cray-impi

To Use Cray compiler

  • cc or ftn for C or Fortran

To use GNU compiler with Intel MPI:

  • module load gcc/6.1.0
  • mpicc , mpif90 are the compiler for C and Fortran

To use Intel compiler with Intel MPI:

  • module load gcc/6.1.0
  • module load intel
  • mpiicc, mpiifort are the compiler for C and Fortran

Software

All installed libraries are available via module.

Compiler and MPI 
PrgEnv-cray/1.0.4
cce/8.7.0 
gcc/4.9.1 
gcc/6.1.0 
intel/18.0.1.163
intel/18.0.2.199
java/jdk1.8.0_131
intel_mpi/18.0.2.199
cray-mvapich2-gnu/2.2rc1
cray-impi/2
openmpi/2.1.1/intel-2018
cdt/18.04 
craype/2.5.14

Math Libraries

 
cray-fftw/3.3.6.4
cray-libsci/17.12.1
cray-fftw_impi/3.3.6.4

Performance tools and debugging

 
advisor/2018.2.0.551025
papi/5.6.0.1 perftools/7.0.1
perftools-base/7.0.1
perftools-lite/7.0.1 vtune_amplifier/2018.2.0.551022
cray-lgdb/3.0.7
craypkg-gen/1.3.6
cray-ccdb/3.0.3
cray-cti/1.0.6

Data Analytics libraries

 
arwpost/3 
grads/2.2.1 
grib2_gnu/3.1.0 
grib2_intel/3.1.0 
hdf5_gnu/1.10.2 
hdf5_intel/1.10.2
ncview_gnu/2.1.7
ncview_intel/2.1.7
netcdf_gnu/4.6.1
netcdf_intel/4.6.1
paraview/5.4.1
pnetcdf_gnu/1.10.0
pnetcdf_intel/1.10.0
python/2.7.14<
r/3.5.1                         zlib_gnu/1.2.11
zlib_intel/1.2.11
Others 
advisor/2018.2.0.551025
advisor/2019.5.0.602216
ansys/v182
ansys/v191
ansys/v191-fluids
ansys/#v19R1-fluids#
ansys/v19R1-fluids
arwpost/3
cadence/ic618
calibre/2013
cdo/1.9.5
ferret/7.4.3(default)
grads/2.2.1
grib2_gnu/3.1.0
grib2_intel/3.1.0
hdf5_gnu/1.10.2
hdf5_intel/1.10.2
intel/18.0.1.163
intel/18.0.2.199(default)
intel/19.0.1.144
intel/19.0.5.281
intel_mpi/18.0.2.199
java/jdk1.8.0_131
mathematica/11.3.0
matlab/R2018b
miniconda/2.7.15
miniconda/3.7
ncl/6.5.0
ncview_gnu/2.1.7
ncview_intel/2.1.7
netcdf_gnu/4.6.1
netcdf_intel/4.6.1
openmpi/2.1.1/gcc-6.1.0
paraview/5.6.0-linux-64bit
pnetcdf_gnu/1.10.0
pnetcdf_intel/1.10.0
python/2.7.14
r/3.5.1
singularity/2.5.1
singularity/3.1.1
tecplot/v2011
tecplot/v2018(default)
tecplot/v2018.anperc
tecplot/v2018.aramco
tecplot/v2018.wdrc
tensorflow/1.7.0-cudnn7.1-cuda9.0-py3.6
vasp/5.4.4/ompi211-intel1602
vtune_amplifier/2018.2.0.551022
zlib_gnu/1.2.11
zlib_intel/1.2.11

File Systems

Neser is connected directly to Shaheen Lustre Parallel filesystem

Running Jobs

Two queues partitions:

  • default workq ( similar to Shaheen)
  • tesla for GPU node

FAQs

  • To launch jobs inside your job scripts, you can use srun and mpirun. Make sure you use the same mpirun launcher from the MPI library used.
  • The compute nodes are exclusive, meaning that even when all the resources within a node are not utilized by a given job, another job will not have access to these resources.
  • The jobs are limited to a maximum of 4 nodes with up to 2 jobs running. The remaining jobs will stay in the queue.
  • To access large memory node, please pecify in the job script #SBATCH --mem=200G or larger
  • To use Paraview, please type: module use /sw/csk/modulefiles and then load paraview