Neser - Pre/Post Processing System
On September 16th KSL introduces a brand-new service to its researchers. A computing cluster called "Neser" is offered to Shaheen users for pre- and post-processing data, allowing them to input data and process the output in a very fast manner. By providing a swift and efficient access to Shaheen storage, Neser distinguishes itself from a general-purpose cluster IBEX. Previously, Shaheen users needed to transfer their data to local computers first for analysis, which was an inconvenient and time-consuming process. With Neser as its sidekick, the need to move large amounts of data for pre-and post-processing is reduced. The ease of retrieving Shaheen data will enable faster scientific results for the High-Performance Computing community.
Neser System Overview
Cray CS500 Cluster composed of:
- Red Hat Enterprise Linux Server release 8.6
- 792 physical compute cores (1584 logical cores with Hyper-Threading enabled)
- 16 Nvidia Tesla K80
- 4.9 TB of compute memory
- Peak performance about 50 TF
- Connected to Shaheen parallel Lustre file system for /project and /scratch
- 20 compute nodes
- 2 login nodes
- Configured with FDR Infiniband.
Compute Node Details
19 nodes composed of:
- Two Skylake Intel Xeon(R) Gold 6138 CPU 2.0 GHz
- 40 physical cores on each node (80 logical cores if using Hyper-Threading)
- 2 nodes are equipped with 768 GB of memory, 5 nodes with 256 GB of memory and 12 nodes with 192 GB of memory
- 12TB of local disk per node
One node composed of 2 Intel Xeon(R) Haswell 16 cores 2.3 GHz with 256 GB of RAM and 16 Nvidia K80 Tesla cards.
Login Node Details
Two login nodes ( neser1, neser2) with two Skylake Intel Xeon(R) Gold 6134 CPU 3.4 GHz
Access and Allocation
Access is granted via SSH using double authentication with OTP: ssh neser.hpc.kaust.edu.sa
Access to Neser is granted to existing Shaheen users upon request, justifying their use of Neser for pre/post processing. Users should contact the KSL helpdesk (help@hpc.kaust.edu.sa) with a short description of their proposed workflow. The new cluster uses the same core-hour accounting system as Shaheen, hence it is important to be a member of an active project.
Quick Start Guide
Compiling
How to load the modules for MPI:
- module load PrgEnv-cray/1.0.4
- export IMPI_VERSION=2018.2.199
- module load cray-impi
To Use Cray compiler
- cc or ftn for C or Fortran
To use GNU compiler with Intel MPI:
- module load gcc/6.1.0
- mpicc , mpif90 are the compiler for C and Fortran
To use Intel compiler with Intel MPI:
- module load gcc/6.1.0
- module load intel
- mpiicc, mpiifort are the compiler for C and Fortran
Software
All installed libraries are available via module.
Compiler and MPI |
PrgEnv-cray/1.0.4 cce/8.7.0 gcc/4.9.1 gcc/6.1.0 intel/18.0.1.163 intel/18.0.2.199 java/jdk1.8.0_131 intel_mpi/18.0.2.199 cray-mvapich2-gnu/2.2rc1 cray-impi/2 openmpi/2.1.1/intel-2018 cdt/18.04 craype/2.5.14 |
Math Libraries |
cray-fftw/3.3.6.4 cray-libsci/17.12.1 cray-fftw_impi/3.3.6.4 |
Performance tools and debugging |
advisor/2018.2.0.551025 papi/5.6.0.1 perftools/7.0.1 perftools-base/7.0.1 perftools-lite/7.0.1 vtune_amplifier/2018.2.0.551022 cray-lgdb/3.0.7 craypkg-gen/1.3.6 cray-ccdb/3.0.3 cray-cti/1.0.6 |
Data Analytics libraries |
arwpost/3 grads/2.2.1 grib2_gnu/3.1.0 grib2_intel/3.1.0 hdf5_gnu/1.10.2 hdf5_intel/1.10.2 ncview_gnu/2.1.7 ncview_intel/2.1.7 netcdf_gnu/4.6.1 netcdf_intel/4.6.1 paraview/5.4.1 pnetcdf_gnu/1.10.0 pnetcdf_intel/1.10.0 python/2.7.14< r/3.5.1 zlib_gnu/1.2.11 zlib_intel/1.2.11 |
Others |
advisor/2018.2.0.551025 advisor/2019.5.0.602216 ansys/v182 ansys/v191 ansys/v191-fluids ansys/#v19R1-fluids# ansys/v19R1-fluids arwpost/3 cadence/ic618 calibre/2013 cdo/1.9.5 ferret/7.4.3(default) grads/2.2.1 grib2_gnu/3.1.0 grib2_intel/3.1.0 hdf5_gnu/1.10.2 hdf5_intel/1.10.2 intel/18.0.1.163 intel/18.0.2.199(default) intel/19.0.1.144 intel/19.0.5.281 intel_mpi/18.0.2.199 java/jdk1.8.0_131 mathematica/11.3.0 matlab/R2018b miniconda/2.7.15 miniconda/3.7 ncl/6.5.0 ncview_gnu/2.1.7 ncview_intel/2.1.7 netcdf_gnu/4.6.1 netcdf_intel/4.6.1 openmpi/2.1.1/gcc-6.1.0 paraview/5.6.0-linux-64bit pnetcdf_gnu/1.10.0 pnetcdf_intel/1.10.0 python/2.7.14 r/3.5.1 singularity/2.5.1 singularity/3.1.1 tecplot/v2011 tecplot/v2018(default) tecplot/v2018.anperc tecplot/v2018.aramco tecplot/v2018.wdrc tensorflow/1.7.0-cudnn7.1-cuda9.0-py3.6 vasp/5.4.4/ompi211-intel1602 vtune_amplifier/2018.2.0.551022 zlib_gnu/1.2.11 zlib_intel/1.2.11 |
File Systems
Neser is connected directly to Shaheen Lustre Parallel filesystem
Running Jobs
Two queues partitions:
- default workq ( similar to Shaheen)
- tesla for GPU node
FAQs
- To launch jobs inside your job scripts, you can use srun and mpirun. Make sure you use the same mpirun launcher from the MPI library used.
- The compute nodes are exclusive, meaning that even when all the resources within a node are not utilized by a given job, another job will not have access to these resources.
- The jobs are limited to a maximum of 4 nodes with up to 2 jobs running. The remaining jobs will stay in the queue.
- To access large memory node, please pecify in the job script #SBATCH --mem=200G or larger
- To use Paraview, please type: module use /sw/csk/modulefiles and then load paraview