Multi-threaded MPI

MPI defines four “levels” of thread safety and it is supported for all three programming environments (intel, cray and gnu) on Shaheen. Cray-MPICH  offers improved support for multi-threaded applications that perform MPI operations within threaded regions. Currently, this feature is available as a separate version of the Cray-MPICH library that is invoked by using a new compiler driver option, -craympich-mt. This is used when MPI code MPI_Init_threads is called instead of MPI_Init. The maximum thread support level is returned by the MPI_Init_thread() call in the "provided"

On Shaheen II, here are the steps for using it:

• Compile your code as follows:

             cc -craympich-mt -o mpi_mt_code.x mpi_mt_test.c

• In your job script, before the srun command, add the following:

              export MPICH_MAX_THREAD_SAFETY=multiple

• Please note that MPICH_MAX_THREAD_SAFETY specifies the maximum allowable thread-safety level that is returned by MPI_Init_thread() in the provided argument. This allows the user to control the maximum level of threading  allowed. The 4  legal values are:
  • MPI_THREAD_SINGLE: only one thread will execute.  
  • MPI_THREAD_FUNNELED: the process may be multi-threaded, but only the main thread will make  MPI calls (all MPI calls are funneled to the main thread).  
  • MPI_THREAD_SERIALIZED: the process may be multi- threaded, and multiple threads may make  MPI calls, but only one at a time: MPI calls are not made concurrently  from two distinct threads (all MPI calls are serialized ).
  • MPI_THREAD_MULTIPLE: Multiple threads may call MPI, with no restrictions.

More information is available in the man page for  intro_mpi.