MPI deﬁnes four “levels” of thread safety and it is supported for all three programming environments (intel, cray and gnu) on Shaheen. Cray-MPICH offers improved support for multi-threaded applications that perform MPI operations within threaded regions. Currently, this feature is available as a separate version of the Cray-MPICH library that is invoked by using a new compiler driver option, -craympich-mt. This is used when MPI code MPI_Init_threads is called instead of MPI_Init. The maximum thread support level is returned by the MPI_Init_thread() call in the "provided"
On Shaheen II, here are the steps for using it:
- Compile your code as follows:
cc -craympich-mt -o mpi_mt_code.x mpi_mt_test.c
- In your job script, before the srun command, add the following:
- Please note that MPICH_MAX_THREAD_SAFETY specifies the maximum allowable thread-safety level that is returned by MPI_Init_thread() in the provided argument. This allows the user to control the maximum level of threading allowed. The 4 legal values are:
- MPI_THREAD_SINGLE: only one thread will execute.
- MPI_THREAD_FUNNELED: the process may be multi-threaded, but only the main thread will make MPI calls (all MPI calls are funneled to the main thread).
- MPI_THREAD_SERIALIZED: the process may be multi- threaded, and multiple threads may make MPI calls, but only one at a time: MPI calls are not made concurrently from two distinct threads (all MPI calls are serialized ).
- MPI_THREAD_MULTIPLE: Multiple threads may call MPI, with no restrictions.
|MPICH_MAX_THREAD_SAFETY value||MPI_Init_thread() returns|
More information is available in the man page for intro_mpi.