Shaheen users should keep in mind that /project space is usually NOT recommended for running jobs. You should be running your jobs on /scratch and after post processing, you should copy ONLY the necessary files back to /project. This is a standard practice in most supercomputing centres. Project filesystem's performance on Shaheen is 5x slower than /scratch, however the data on /project is backed-up to tape. On the other hand, data that isn't accessed on /scratch will be deleted automatically after a 60-day period of inactivity.
Those of you who are used to running all jobs from /project, with the deployment of the new /project filesystem, it is time to start running your jobs on /scratch. The new project storage is mounted read-only on the compute nodes, which was not possible before, with the old project storage. This change should improve the way how the user data is handled on Shaheen and ensure that all essential files are properly backed up to the tape system.
Please note this change will not drastically change the way you run your jobs. The idea is to have a directory of your read-only input files on /project, and the job script should automatically move the necessary read-write files to /scratch, run the job, delete the temporary files, run analytics/postprocessing, and copy ONLY essential files back to /project.
Do not hesitate to contact us at firstname.lastname@example.org if you need any further help.
Reducing unnecessary data on /project reduces the load on the backup system and makes it more efficient, which in turn helps all Shaheen users.