Skip to main content

High Performance Computing

Introduction to th...

Communicating... [mpi]

High Performance Computing

Introduction to th...

Collective Co... [mpi]

Point-to-Point Communication
"Introduction to the Message Passing Interface" course by the Southampton RSG

"Introduction to the Message Passing Interface" course by the Southampton RSG

Creative Commons License

Point-to-Point Communication

In the previous episode we introduced the various types of communication in MPI. In this section we will use the MPI library functions MPI_Send() and MPI_Recv(), which employ point-to-point communication, to send data from one rank to another.
Sending data from one rank to another using MPI_SSend and MPI_Recv()
Let's look at how MPI_Send() and MPI_Recv()are typically used:
  • Rank A decides to send data to rank B. It first packs the data to send into a buffer, from which it will be taken.
  • Rank A then calls MPI_Send() to create a message for rank B. The underlying MPI communication is then given the responsibility of routing the message to the correct destination.
  • Rank B must know that it is about to receive a message and acknowledge this by calling MPI_Recv(). This sets up a buffer for writing the incoming data when it arrives and instructs the communication device to listen for the message.
Note that MPI_Send and MPI_Recv() are often used in a synchronous manner, meaning they will not return until communication is complete on both sides. However, as mentioned in the previous episode, MPI_Send() may return before the communication is complete, depending on the implementation and message size.

Sending a Message: MPI_Send()

The MPI_Send() function is defined as follows:
int MPI_Send( const void *data, int count, MPI_Datatype datatype, int destination, int tag, MPI_Comm communicator )
*data:Pointer to the start of the data being sent. We would not expect this to change, hence it's defined as const
count:Number of elements to send
datatype:The type of the element data being sent, e.g. MPI_INTEGER, MPI_CHAR, MPI_FLOAT, MPI_DOUBLE, ...
destination:The rank number of the rank the data will be sent to
tag:An message tag (integer), which is used to differentiate types of messages. We can specify 0 if we don't need different types of messages
communicator:The communicator, e.g. MPI_COMM_WORLD as seen in previous episodes
For example, if we wanted to send a message that contains "Hello, world!\n" from rank 0 to rank 1, we could state (assuming we were rank 0):
char *message = "Hello, world!\n"; MPI_Send(message, 14, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
So we are sending 14 elements of MPI_CHAR() one time, and specified 0 for our message tag since we don't anticipate having to send more than one type of message. This call is synchronous, and will block until the corresponding MPI_Recv() operation receives and acknowledges receipt of the message.

MPI_Ssend(): an Alternative to MPI_Send()

MPI_Send() represents the "standard mode" of sending messages to other ranks, but some aspects of its behaviour are dependent on both the implementation of MPI being used, and the circumstances of its use. There are three scenarios to consider:
  1. The message is directly passed to the receive buffer, in which case the communication has completed
  2. The send message is buffered within some internal MPI buffer but hasn't yet been received
  3. The function call waits for a corresponding receiving process
In scenarios 1 & 2, the call is able to return immediately, but with 3 it may block until the recipient is ready to receive. It is dependent on the MPI implementation as to what scenario is selected, based on performance, memory, and other considerations.
A very similar alternative to MPI_Send() is to use MPI_Ssend() - synchronous send - which ensures the communication is both synchronous and blocking. This function guarantees that when it returns, the destination has categorically started receiving the message.

Receiving a Message: MPI_Recv()

Conversely, the MPI_Recv() function looks like the following:
int MPI_Recv( void *data, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm communicator, MPI_Status *status )
*data:Pointer to where the received data should be written
count:Maximum number of elements to receive
datatype:The type of the data being received
source:The number of the rank sending the data
tag:A message tag, which must either match the tag in the sent message, or if MPI_ANY_TAG is specified, a message with any tag will be accepted
communicator:The communicator (we have used MPI_COMM_WORLD in earlier examples)
*status:A pointer for writing the exit status of the MPI command, indicating whether the operation succeeded or failed
Continuing our example, to receive our message we could write:
char message[15] = {0}; /* Initialise teh buffer to zeros */ MPI_Status status; MPI_Recv(message, 14, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &status); message[14] = '\0';
Here, we create a buffer message to store the received data and initialise it to zeros ({0}) to prevent any garbage content. We then call MPI_Recv() to receive the message, specifying the source rank (0), the message tag (0), and the communicator (MPI_COMM_WORLD). The status object is passed to capture details about the received message, such as the actual source rank or tag, though it is not used in this example. To ensure safe string handling, we explicitly null-terminate the received message by setting message[14] = '\0'.
Let's put this together with what we've learned so far. Here's an example program that uses MPI_Send() and MPI_Recv() to send the string "Hello World!" from rank 0 to rank 1:
#include <stdio.h> #include <mpi.h> int main(int argc, char** argv) { int rank, n_ranks; // First call MPI_Init MPI_Init(&argc, &argv); // Check that there are two ranks MPI_Comm_size(MPI_COMM_WORLD,&n_ranks); if( n_ranks != 2 ){ printf("This example requires exactly two ranks\n"); MPI_Finalize(); return 1; } // Get my rank MPI_Comm_rank(MPI_COMM_WORLD,&rank); if( rank == 0 ){ const char *message = "Hello, world!\n"; MPI_Send(message, 14, MPI_CHAR, 1, 0, MPI_COMM_WORLD); } if( rank == 1 ){ char message[15] = {0}; MPI_Status status; MPI_Recv(message, 14, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &status); message[14] = `\0`; printf("%s",message); } // Call finalise at the end return MPI_Finalize(); }
When using MPI_Recv() to receive string data, ensure that your buffer is large enough to hold the message and includes space for a null terminator. Explicitly initialising the buffer and adding the null terminator avoids undefined behavior or garbage output.

MPI Data Types in C

In the above example we send a string of characters and therefore specify the type MPI_CHAR. For a complete list of types, see the MPICH documentation.

Try It Out

Compile and run the above code. Does it behave as you expect?

What Happens If...

Try modifying, compiling, and re-running the code to see what happens if you...
  1. Change the tag integer of the sent message. How could you resolve this where the message is received?
  2. Modify the element count of the received message to be smaller than that of the sent message. How could you resolve this in how the message is sent?

Many Ranks

Change the above example so that it works with any number of ranks. Pair even ranks with odd ranks and have each even rank send a message to the corresponding odd rank.

Hello Again, World!

Modify the Hello World code below so that each rank sends its message to rank 0. Have rank 0 print each message.
#include <stdio.h> #include <mpi.h> int main(int argc, char **argv) { int rank; char message[30]; // First call MPI_Init MPI_Init(&argc, &argv); // Get my rank MPI_Comm_rank(MPI_COMM_WORLD, &rank); // Print a message using snprintf and then printf snprintf(message, 30, "Hello World, I'm rank %d", rank); printf("%s\n", message); // Call finalise at the end return MPI_Finalize(); }
Note: In MPI programs, every rank runs the same code. To make ranks behave differently, you must explicitly program that behavior based on their rank ID. For example:
  • Use conditionals like if (rank == 0) to define specific actions for rank 0.
  • All other ranks can perform different actions in an else block.
If you don't require the additional information provided by MPI_Status, such as source or tag, you can use MPI_STATUS_IGNORE in MPI_Recv calls. This simplifies your code by removing the need to declare and manage an MPI_Status object. This is particularly useful in straightforward message-passing scenarios.

Blocking

Try the code below and see what happens. How would you change the code to fix the problem?
Note: If you are using the MPICH, this example might automagically work. With OpenMPI it shouldn't!
#include <mpi.h> #define ARRAY_SIZE 3 int main(int argc, char **argv) { MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); const int comm_tag = 1; int numbers[ARRAY_SIZE] = {1, 2, 3}; MPI_Status recv_status; if (rank == 0) { // synchronous send: returns when the destination has started to // receive the message MPI_Ssend(&numbers, ARRAY_SIZE, MPI_INT, 1, comm_tag, MPI_COMM_WORLD); MPI_Recv(&numbers, ARRAY_SIZE, MPI_INT, 1, comm_tag, MPI_COMM_WORLD, &recv_status); } else { MPI_Ssend(&numbers, ARRAY_SIZE, MPI_INT, 0, comm_tag, MPI_COMM_WORLD); MPI_Recv(&numbers, ARRAY_SIZE, MPI_INT, 0, comm_tag, MPI_COMM_WORLD, &recv_status); } return MPI_Finalize(); }

Ping Pong

Write a simplified simulation of Ping Pong according to the following rules:
  • Ranks 0 and 1 participate
  • Rank 0 starts with the ball
  • The rank with the ball sends it to the other rank
  • Both ranks count the number of times they get the ball
  • After counting to 1 million, the rank is bored and gives up
  • There are no misses or points