CLE7 Transition

On the week commencing 19thJanuary 2020, Shaheen was upgraded with the latest version of the operating system (CLE7/SLES15).

The change in the operating system, has introduced a new default software environment, CDT/19.12. As a result, some of the application and software compiled with the Cray programming environment prior to CDT/19.03 may no longer work. Therefore, rebuilding (or at least relinking) of this software will be required. We recommend that all software is rebuilt regardless. With the OS upgrade to CLE7 SLES 15 it is recommended that in-house code should be recompiled.

Below, please find the list of the scientific libraries and applications available as of 22nd January 2020. The CS team is working in building more software. Please feel free to contact us at help@hpc.kaust.edu.sa to request missing packages.

The default version of Cray supported compilers and libraries have changed to the following:

Major changes

• Default CDT is 19.12
  • Default linking is dynamic with cdt/19.12.
  • To build statically linked executables, just add the -static flag to the command and link lines, or set CRAYPE_LINK_TYPE=static in the environment,
  • Only 2 CDT are available on Shaheen. 19.12 by default and 19.03
    • Swapping to the prior cdt/19.03 will result in static linking by default; while swapping to the 19.12 version (or later) of craype will result in dynamic linking by default.
• Cray Compiler Environment is based now on clang/LLVM with some additional Cray enhancements.
  • Most of the compiler flags have changed, and some capabilities available in CCE 8 and previous versions are no longer available in CCE 9
  • Cray provides a migration guide for users switching from CCE 8 to CCE 9.
  • OpenMP has been disabled by default in the C, C++, and Fortran compilers. To enable OpenMP, one can use the following flags:
    • C/C++: -fopenmp
    • Fortran: -h omp
  • To use the classical CCE, do module swap cce cce/9.1.1-classic
    • "classic" CCE is now considered "legacy," and that future versions of CCE will likely be based on Clang/LLVM
  • If you swap to or from a CCE Clang compiler with CCE-classic,you must reload the cray-mpich module to get correct libraries.
  • Unlike CCE Fortran and Classic C and C++ compilers, all optimizations are disabled by default for the Clang-based compiler.
    • The -ffast-math option implies -ffp=3. Cray suggests using the -Ofast optimization level with Cray Clang to obtain the best performance on most codes.
    • The -help command line option is a good resource, along with the craycc(1) and crayCC(1) man pages.
• Due to the change of the default software development environments, the Slurm jobscripts of some applications (especially for chemistry/materials sciences/molecular dynamics) have also been adjusted accordingly.
  • Please find the updated Slurm jobscript examples in the respective example folders: /sw/xc40cle7/<appname>/<appversion>/<appcompilation>/example.

CLE7 Tips:

• Avoid using full path in your scripts in makefile or submission when using Cray and KSL supported libraries.
  • All the CS team supported software are installed in /sw/xc40cle7 now. So please be careful just in case you use absolute paths in your installation scripts and/or SLURM job scripts.
  • To look for a specific library/application that is installed on Shaheen, type: module avail -S apps
• Cray-python calls the OpenMP threaded version of cray-libsci. It is recommended to set the number of desired threads with the OMP_NUM_THREADS environment variable.

• If you have any issues, please contact us through help@hpc.kaust.edu.sa.

CLE7 known issues

• In some runs, you might see the following harmless error message. It is not affecting your runs, we are working on it to remove the message) : XALT_EXCEPTION(xalt_run_submission.py):  [Errno 2] No such file or directory, it's not relate
• Using CCE-Clang, you will see the detailed linking libraries