Low-level MPI interfaces

MPI API Support

MPI routines	KokkosComm::mpi:: namespace	Kokkos::View support
MPI_Send	send or send<CommMode::Standard>	✓
MPI_Rsend	send<CommMode::Ready>	✓
MPI_Ssend	send<CommMode::Synchronous>	✓
MPI_Isend	isend or isend<CommMode::Standard>	✓
MPI_Irsend	isend<CommMode::Ready>	✓
MPI_Issend	isend<CommMode::Synchronous>	✓
MPI_Recv	recv	✓
MPI_Allgather	allgather	✓
MPI_Allgather (in-place)	allgather	✓
MPI_Iallgather	iallgather	✓
MPI_Reduce	reduce	✓
MPI_Ireduce	ireduce	✓
MPI_Bcast	broadcast	✓
MPI_Ibcast	ibroadcast	✓
MPI_Alltoall	alltoall	✓
MPI_Ialltoall	ialltoall	✓
MPI_Allreduce	allreduce	✓
MPI_Iallreduce	iallreduce	✓
MPI_Scan	inclusive_scan	✓
MPI_Exscan	exclusive_scan	✓
MPI_Barrier	barrier	✓

Point-to-point

template<KokkosView SendView>
auto send(const SendView &sv, int dest, int tag, MPI_Comm comm) -> void

Initiates a blocking send operation.

Template Parameters:

SendView – The type of the view to be sent.

Parameters:

• sv – The view to be sent.

• dest – The destination rank.

• tag – The message tag.

• comm – The MPI communicator.

template<CommMode SendMode = CommMode::Default, KokkosExecutionSpace ExecSpace, KokkosView SendView>
auto send(const ExecSpace &space, const SendView &sv, int dest, int tag, MPI_Comm comm) -> void

Initiates a blocking send operation with a specified execution space and communication mode.

Template Parameters:

• SendMode – The communication mode (default is CommMode::Default).

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

Parameters:

• space – The execution space.

• sv – The view to be sent.

• dest – The destination rank.

• tag – The message tag.

• comm – The MPI communicator.

template<CommMode SendMode, KokkosExecutionSpace ExecSpace, KokkosView SendView>
auto isend(Communicator<MpiSpace, ExecSpace> &h, const SendView &sv, int dest, int tag) -> Request<MpiSpace>

Initiates a non-blocking send operation.

Template Parameters:

• SendMode – The communication mode.

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

Parameters:

• h – The handle for the execution space and MPI.

• sv – The view to be sent.

• dest – The destination rank.

• tag – The message tag.

Returns:

A request object for the non-blocking send operation.

template<KokkosView RecvView>
auto recv(const RecvView &rv, int src, int tag, MPI_Comm comm, MPI_Status *status) -> void

Initiates a blocking receive operation.

Template Parameters:

RecvView – The type of the view to be received.

Parameters:

• rv – The view to be received.

• src – The source rank.

• tag – The message tag.

• comm – The MPI communicator.

• status – The MPI status object for the blocking receive operation.

template<KokkosExecutionSpace ExecSpace, KokkosView RecvView>
auto recv(const ExecSpace &space, RecvView &rv, int src, int tag, MPI_Comm comm) -> void

Initiates a blocking receive operation with a specified execution space.

Template Parameters:

• ExecSpace – The execution space.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• rv – The view to be received.

• src – The source rank.

• tag – The message tag.

• comm – The MPI communicator.

template<KokkosView RecvView>
auto irecv(const RecvView &rv, int src, int tag, MPI_Comm comm, MPI_Request &req) -> void

Initiates a non-blocking receive operation.

Template Parameters:

RecvView – The type of the view to be received.

Parameters:

• rv – The view to be received.

• src – The source rank.

• tag – The message tag.

• comm – The MPI communicator.

• req – The MPI request object for the non-blocking receive operation.

Throws:

std::runtime_error – If the view is not contiguous.

template<KokkosView SendView>
auto isend(const SendView &sv, int dest, int tag, MPI_Comm comm, MPI_Request &req) -> void

MPI_Isend with a Kokkos::View.

Template Parameters:

SendView – The type of the view to be sent.

Parameters:

• sv – The view to be sent (must be contiguous).

• dest – The destination rank.

• tag – The message tag.

• comm – The MPI communicator.

• req – The MPI request.

Collectives

template<KokkosView SendView, KokkosView RecvView>
auto allgather(const SendView &sv, const RecvView &rv, MPI_Comm comm) -> void

Performs an allgather operation, gathering data from all processes and distributing it to all processes.

Template Parameters:

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• sv – The view to be sent.

• rv – The view to be received.

• comm – The MPI communicator.

template<KokkosView RecvView>
auto allgather(const RecvView &rv, MPI_Comm comm) -> void

Performs an in-place allgather operation, gathering data from all processes and distributing it to all processes.

Template Parameters:

RecvView – The type of the view to be received.

Parameters:

• rv – The view to be received.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SendView, KokkosView RecvView>
auto allgather(const ExecSpace &space, const SendView &sv, const RecvView &rv, MPI_Comm comm) -> void

Performs an allgather operation with a specified execution space, gathering data from all processes and distributing it to all processes.

Template Parameters:

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent.

• rv – The view to be received.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SView, KokkosView RView>
auto iallgather(const ExecSpace &space, const SView sv, RView rv, MPI_Comm comm) -> Request<MpiSpace>

MPI_Iallgather with Kokkos::View arguments.

Template Parameters:

• ExecSpace – The execution space.

• SView – The type of the view to be sent.

• RView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous).

• comm – The MPI communicator.

Returns:

A request object for the non-blocking all-gather.

template<KokkosView SendView, KokkosView RecvView>
auto reduce(const SendView &sv, const RecvView &rv, MPI_Op op, int root, MPI_Comm comm) -> void

Performs a reduction operation, combining data from all processes and distributing the result to the root process.

Template Parameters:

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• sv – The view to be sent.

• rv – The view to be received.

• op – The MPI operation to be applied.

• root – The rank of the root process.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SendView, KokkosView RecvView>
auto reduce(const ExecSpace &space, const SendView &sv, const RecvView &rv, MPI_Op op, int root, MPI_Comm comm) -> void

Performs a reduction operation with a specified execution space, combining data from all processes and distributing the result to the root process.

Template Parameters:

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent.

• rv – The view to be received.

• op – The MPI operation to be applied.

• root – The rank of the root process.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SView, KokkosView RView>
auto ireduce(const ExecSpace &space, const SView &sv, RView &rv, MPI_Op op, int root, MPI_Comm comm) -> Request<MpiSpace>

MPI_Ireduce with Kokkos::View arguments.

Template Parameters:

• ExecSpace – The execution space.

• SView – The type of the view to be sent.

• RView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent.

• rv – The view to be received (valid only on root).

• op – The MPI operation to be applied.

• root – The rank of the root process.

• comm – The MPI communicator.

Returns:

A request object for the non-blocking reduction.

template<KokkosView View>
auto broadcast(const View &v, int root, MPI_Comm comm) -> void

MPI_Bcast with a Kokkos::View.

Template Parameters:

View – The type of the view to be broadcast.

Parameters:

• v – The view to be broadcast (must be contiguous).

• root – The rank of the root process.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView View>
auto broadcast(const ExecSpace &space, const View &v, int root, MPI_Comm comm) -> void

MPI_Bcast with a Kokkos::View and execution space.

Template Parameters:

• ExecSpace – The execution space.

• View – The type of the view to be broadcast.

Parameters:

• space – The execution space.

• v – The view to be broadcast (must be contiguous).

• root – The rank of the root process.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView View>
auto ibroadcast(const ExecSpace &space, View &v, int root, MPI_Comm comm) -> Request<MpiSpace>

MPI_Ibcast with a Kokkos::View.

Template Parameters:

• ExecSpace – The execution space.

• View – The type of the view to be broadcast.

Parameters:

• space – The execution space.

• v – The view to be broadcast (must be contiguous).

• root – The rank of the root process.

• comm – The MPI communicator.

Returns:

A request object for the non-blocking broadcast.

template<KokkosExecutionSpace ExecSpace, KokkosView SendView, KokkosView RecvView>
auto alltoall(const ExecSpace &space, const SendView &sv, const size_t sendCount, const RecvView &rv, const size_t recvCount, MPI_Comm comm) -> void

MPI_Alltoall with Kokkos::View arguments.

Template Parameters:

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• sendCount – The number of elements to send to each process.

• rv – The view to be received (must be contiguous).

• recvCount – The number of elements to receive from each process.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView RecvView>
auto alltoall(const ExecSpace &space, const RecvView &rv, const size_t recvCount, MPI_Comm comm) -> void

MPI_Alltoall (in-place) with a Kokkos::View.

Template Parameters:

• ExecSpace – The execution space.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• rv – The view to be received (must be contiguous).

• recvCount – The number of elements to receive from each process.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SView, KokkosView RView>
auto ialltoall(const ExecSpace &space, const SView sv, RView rv, int count, MPI_Comm comm) -> Request<MpiSpace>

MPI_Ialltoall with Kokkos::View arguments.

Template Parameters:

• ExecSpace – The execution space.

• SView – The type of the view to be sent.

• RView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous).

• count – The number of elements sent to (and received from) each process.

• comm – The MPI communicator.

Returns:

A request object for the non-blocking all-to-all.

template<KokkosView SendView, KokkosView RecvView>
auto allreduce(const SendView &sv, const RecvView &rv, MPI_Op op, MPI_Comm comm) -> void

MPI_Allreduce with Kokkos::View arguments.

Template Parameters:

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosView View>
auto allreduce(const View &v, MPI_Op op, MPI_Comm comm) -> void

MPI_Allreduce (in-place) with a Kokkos::View.

Template Parameters:

View – The type of the view to be reduced.

Parameters:

• v – The view to be reduced (must be contiguous).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SendView, KokkosView RecvView>
auto allreduce(const ExecSpace &space, const SendView &sv, const RecvView &rv, MPI_Op op, MPI_Comm comm) -> void

MPI_Allreduce with Kokkos::View arguments and an execution space.

Template Parameters:

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView View>
auto allreduce(const ExecSpace &space, const View &v, MPI_Op op, MPI_Comm comm) -> void

MPI_Allreduce (in-place) with a Kokkos::View and an execution space.

Template Parameters:

• ExecSpace – The execution space.

• View – The type of the view to be reduced.

Parameters:

• space – The execution space.

• v – The view to be reduced (must be contiguous).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosView SView, KokkosView RView, KokkosExecutionSpace ExecSpace>
auto iallreduce(const ExecSpace &space, const SView sv, RView rv, MPI_Op op, MPI_Comm comm) -> Request<MpiSpace>

MPI_Iallreduce with Kokkos::View arguments.

Template Parameters:

• SView – The type of the view to be sent.

• RView – The type of the view to be received.

• ExecSpace – The execution space.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

Returns:

A request object for the non-blocking all-reduce.

template<KokkosView SendView, KokkosView RecvView>
auto inclusive_scan(const SendView &sv, const RecvView &rv, MPI_Op op, MPI_Comm comm) -> void

MPI_Scan with Kokkos::View arguments.

Template Parameters:

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• sv – The view to be sent (must be contiguous, rank ≤ 1).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SendView, KokkosView RecvView>
auto inclusive_scan(const ExecSpace &space, const SendView &sv, const RecvView &rv, MPI_Op op, MPI_Comm comm) -> void

MPI_Scan with Kokkos::View arguments and an execution space.

Template Parameters:

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosView SendView, KokkosView RecvView>
auto exclusive_scan(const SendView &sv, const RecvView &rv, MPI_Op op, MPI_Comm comm) -> void

MPI_Exscan with Kokkos::View arguments.

Template Parameters:

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• sv – The view to be sent (must be contiguous, rank ≤ 1).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

template<KokkosExecutionSpace ExecSpace, KokkosView SendView, KokkosView RecvView>
auto exclusive_scan(const ExecSpace &space, const SendView &sv, const RecvView &rv, MPI_Op op, MPI_Comm comm) -> void

MPI_Exscan with Kokkos::View arguments and an execution space.

Template Parameters:

• ExecSpace – The execution space.

• SendView – The type of the view to be sent.

• RecvView – The type of the view to be received.

Parameters:

• space – The execution space.

• sv – The view to be sent (must be contiguous).

• rv – The view to be received (must be contiguous, same size as sv).

• op – The MPI operation to be applied.

• comm – The MPI communicator.

inline auto barrier(MPI_Comm comm) -> void

Blocks until all processes in the communicator have reached this routine.

Parameters:

comm – The MPI communicator.

Related Types

enum class CommMode

A scoped enum to specify the mode of an operation. Buffered mode is not supported.

enumerator Standard

The MPI implementation decides whether outgoing messages will be buffered. Send operations can be started whether or not a matching receive has been started. They may complete before a matching receive is started. Standard mode is non-local: successful completion of the send operation may depend on the occurrence of a matching receive.

enumerator Ready

Send operations may be started only if the matching receive is already started.

enumerator Synchronous

Synchronous mode: Send operations complete successfully only if a matching receive is started, and the receive operation has started to receive the message sent.

enumerator Default

Default mode is an alias for Standard mode, but lets users override the behavior of operations at compile-time using the KOKKOSCOMM_FORCE_SYNCHRONOUS_MODE pre-processor define. This forces Synchronous mode for all “default-mode” operations, which can be useful for debugging purposes, e.g. for asserting that the communication scheme is correct.