4. Nonblocking Send and Receive.

Purpose:

One of the means to improve the performance on many systems by overlapping communication and computation.

Basic operations:

Nonblocking Send: MPI_Isend
Nonblocking Receive:  MPI_Irecv
Check for Completion: MPI_Test; MPI_Wait
 

A nonblocking send call initiates the send operation, but does not complete it.

The send start call will return before the message is copied out of the send buffer. A separate send complete call is needed to complete the communication, i.e., to verify that the data have been copied out of the send buffer.
 

Syntax:

MPI_Isend(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request);
 

• the first six arguments have the same meaning as in the blocking send.

• Request is a communication request handle that you can use later to query the status of the communication or to wait for its completion.

Calling the nonblocking send indicates that the system may start copying data out of the send buffer. The sender should not access any part of the send buffer after a nonblocking send operation is called until the send completes.
 
 

A nonblocking receive start call initiates the receive operation, but does not complete it.

The call will return before a message is stored into the receive buffer. An MPI_Wait call is needed to complete the receive operation.
 

Syntax:

MPI_Irecv(void* buf, int count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm, MPI_Request *request);
 

• source is the rank of the processor that sent the message.

• the remaining arguments are as previously discussed.

The receive buffer stores count consecutive elements of the type specified by datatype, starting at the address in buf.

The length of the received message must be less than or equal to the length of the
receive buffer. An overflow error occurs if all incoming data does not fit, without truncation, into the receive buffer.
 
 

To check the status of a nonblocking send or receive, one can call:
 

MPI_Test(MPI_Request *request, int *flag,MPI_Status *status);
 

• request is a communication request (from a call to MPI_Isend, for example)

• flag is set to TRUE if the operation is completed

• status is a status object containing information on the transaction returned by the call.

 

The MPI_Test call returns immediately.
 
 
 

A call to MPI_Wait returns when the operation identified by request is complete.

        MPI_Wait(MPI_Request *request, MPI_Status *status);
 

• all arguments are the same as for MPI_Test.

 
 
 

Order: Nonblocking communication operations occur in the same order as the execution of the calls that initiate the communication. In both blocking and nonblocking communication, operations are non-overtaking.
 

An example of a nonblocking send and receive with an MPI_Wait call:

        MPI_Comm_rank(comm,&rank);
        if (rank == 0) {

           MPI_Isend(sendbuf,count,MPI_REAL,1,tag,comm,&request);

****** do some computation to mask latency ******

           MPI_Wait(&request,&status);
        }
        else (!(rank == 1)) {

           MPI_Irecv(recvbuf,count,MPI_REAL,0,tag,comm,&request);
****** do some computation to mask latency ******

           MPI_Wait(&request,&status);
        }
 

A request object can be deadlocked without waiting for the associated communication to complete. One way to avoid any deadlock caused by a request object is to use a different request for each pair of sends and receives.

For example:

        MPI_Comm_rank(comm,&rank);

        if (rank == 0) {

           MPI_Isend(sbuf1,count,MPI_REAL,1,tag,comm,&req1);
           MPI_Isend(sbuf2,count,MPI_REAL,1,tag,comm,&req2);

        }

        else (!(rank == 1) {

           MPI_Irecv(rbuf1,count,MPI_REAL,0,tag,comm,&req1);
           MPI_Irecv(rbuf2,count,MPI_REAL,0,tag,comm,&req2);

        }
           MPI_Wait(&req1,&status);
           MPI_Wait(&req2,&status);
 
 

A call to MPI_Wait that completes a receive will eventually terminate and return if a matching send has been started, unless the send is satisfied by another receive.

In particular, if the matching send is nonblocking, then the receive should complete even if no call is executed by the sender to complete the send.

Similarly, a call to MPI_Wait that completes a send will eventually return if a matching receive has been started, unless the receive is satisfied by another send, even if no call is executed to complete the receive.
 
 
 

The MPI_Probe and MPI_Iprobe operations check for incoming messages without actually receiving them.

After using these operations, the user can decide how to receive the messages, based on the information returned by the probe.

In particular, the user may allocate adequate memory for the receive buffer according to the length of the probed message.

Syntax:

    MPI_Probe(int source, int tag, MPI_Comm comm,
            MPI_Status *status);

  MPI_Iprobe(int source, int tag, MPI_Comm comm,
            int *flag, MPI_Status *status);
 

• source and tag are selection criteria for messages.

• comm is the communicator and status contains information about a waiting message.

In the nonblocking call, MPI_Iprobe, flag returns TRUE if a message
matching the criteria is waiting.

The blocking call, MPI_Probe, waits until such a message is available for
receipt.

Note that there are times when the length of the incoming message is not known a priori. In this case, one can use MPI_Probe or MPI_Iprobe with MPI_Get_count:

  MPI_Probe(source,tag,comm,&status);
 MPI_Get_count(&status,datatype,&count);
 

• count returns the number of the items contained in the incoming message.

Example 1: ring.c .

Communicates among the nodes in the MPI_COMM_WORLD in ring fashion using both blocking and nonblocking communications.

Example 2: pi.c .

computes the value of pi.

Example 3: brecv.c .

This program demonstrates that replacing a blocking receive with a non-blocking receive earlier in the program can decrease the synchronization time on the corresponding send.