Efficient I/O is critical for performance with data-intensive applications since parallel file systems are usually a substantial bottleneck on HPC systems.
Here are some simple guidelines that can be used for almost any type of I/O on KSL HPC systems:
- Avoid frequently opening and closing the same file during code execution.
- Avoid creating directory hierarchies with thousands of files which causes significant overhead.
- Aggregate small amounts of data into larger reads and writes.
- Avoid using ASCII representations of your data since they usually require much more space to store, and require conversion to/from binary when reading/writing.
- Don’t re-invent the wheel. Using Parallel I/O libraries such as MPI-IO, HDF5 and netCDF can help you to parallelise, aggregate and efficiently manage I/O operations. HDF5 and netCDF use binary file formats that support complex data models and provide portability across multiple systems.