KAUST Supercomputing Laboratory Newsletter 10th December 2015

RCAC Meeting

The next scheduled RCAC (Research Computing Allocation Committee) meeting is Thursday 17th December.

Retirement of Neser and Associated Scratch Storage

As previously announced, Neser will be decommissioned on January and the last date that jobs will be scheduled to run is 31st December 2015.

We will continue to have the Shaheen I/Neser ‘home' and ‘project' filesystems available until at least 31st July 2016. However, please note that the ‘scratch' filesystem will be taken off-line and deleted on the 1st February 2016.

For data that is needed for projects on Shaheen II, rather than copying it yourself, please contact us and we can assist in moving the data as we have dedicated systems for this that have direct access to both storage subsystems.

Scratch filesystem purge policy

We would like to remind all users that /scratch files not modified AND not accessed in the last 60 days will be deleted.

Neser Last Day of Service

Please note this system will be decommissioned in January, the last day that any job will be able to run is 31st December 2015.

Tip of the Week:Best practices on Lustre

  • Edit/build code in user home area whenever possible
  • Open files as read-only whenever possible
  • Read small, shared files from a single task
  • Placing too many files in a single directory negatively impacts performance.
  • Avoid using ls -l
  • Avoid frequently opening files in append mode, writing small amounts of data, and closing the file.
  • Instead of reading a small file from every task, read the entire file from one task and broadcast the contents to all other tasks.
  • If possible, use a high-level I/O library ( HDF5, NetCDF)
  • The default I/O striping width is 1 on the Shaheen II parallel filesystem. This is optimal for many cases especially when every MPI process writes to its own file resulting in as many files as number of processes used.
  • The Lustre filesystem stripe count can be modified using the following command with a maximum stripe count of 144: lfs setstripe --count [stripe-count] filename/directory

For more details, please refer to https://www.hpc.kaust.edu.sa/training/2015/cray/material/4.01_IO.pdf

Follow us on Twitter

Follow all the latest news on HPC within the Supercomputing Lab and at KAUST, on Twitter @KAUST_HPC.

Previous Announcements

Previous Tips