Parallel Computing on Distributed Systems using MPI

Parallel Computing:

Definition: Utilize multiple processors concurrently to solve a computational problem  with each processor working on a part of problem

Why? Certain classes of probelms are either too large or too long to execute on available

                        uniprocessor machines
 

Promise of Parallel Computing:

• Cheaper, in terms of Price/Performance
• Faster than equivalently expensive

uniprocessor machine
• Scalable: the performance of a

particular program may be improved by
execution on a larger machine
• Reliable? In theory if some processors fail we can simply use others.
• Handle bigger problems

How?

• Parallel Machines: MIMD system

(Multiple-Instruction-Multiple-Data)
Individual processors work on different data under the control of their own program
• Parallel Algorithms: Find sections of problem which can be executed simultaneously with other sections without changing the final results.
• Parallel Software: Ideal parallel compilers that automatically parallelize sequential algorithms (program) remain limited in their

applicabality; Most likely we must identify and exploit potential parallelism in applications by ourself.

Definition: A distributed computing system consists of multiple autonomousprocessors

that do not share primary memory, but cooperate by communications over network.

It is a distributed-memory MIMD system.

Available system: Computers in year 3 Lab. falcon1 - falcon20 which are connected via a local area network (LAN).

Communication: message passing on network.

A library which can be called from C, C++ and Fortran 77, used to implement message-passing model on distributed system, hopefully with efficiency, portability, and functionality.

-

Example of a simple C program with MPI:

-

#include "mpi.h"

#include <stdio.h>

int main( argc, argv )

int argc;

char **argv;

{

MPI_Init( &argc, &argv );

printf( "Hello world\n" );

MPI_Finalize();

return 0;

}

 

#include "mpi.h" provides basic MPI definitions and types
MPI_Init starts MPI
MPI_Finalize exits MPI
Note that all non-MPI routines are local; thus the printf run on each process
 

A Second Example:
 
 

#include "mpi.h"
#include <stdio.h>

int main( argc, argv )
int argc;
char **argv;
{
int rank, size;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Comm_size( MPI_COMM_WORLD, &size );
printf( "Hello world! I'm %d of %d\n",
        rank, size );
MPI_Finalize();
return 0;
}

 
 

MPI_Comm_rank finds out which process I am (For example: the ID of a processor in a parallel computer cluster).
MPI_Comm_size finds out how many processes there are.

These sample programs have been kept as simple as possible by assuming that all processes can do output. Not all parallel systems provide this feature, and MPI provides a way to handle this case.

 

 
 

Another example of a C program with MPI for a very simple problem.

Summary of basic MPI routines (functions):

MPI_Init                      Initialize MPI
MPI_Comm_size         Find out how many processes there are
MPI_Comm_rank        Find out which processor I am
MPI_Bcast                   Send data to all other processors
MPI_Reduce                 collect and further process data from all processors
MPI_Send                    Send a message
MPI_Recv                    Recieve a message
MPI_Wtime                 Give the timing of a process
MPI_Finalize               Terminate MPI
 

Runing a C program on our Department Falcon clusters:

• Add /usr/local/mpi/bin to your path by adding the following line to either .cshrc or .login file:
set path=($path . /usr/local/bin  /usr/local/mpi/bin)
 
• Create a .rhosts file in your home directory with the following content:

falcon2 your_user_name
falcon3 your_user_name
.....
falcon20 your_user_name

Then excute: chmod 600 .rhosts

• Commands to run a program (Example: your_source_code.c)

mpicc your_source_code.c -lm
mpirun -np  4 (# of processors) a.out

General Procedure:

 
For simple programs, special compiler commands can be used. For large projects, it is best to use a standard Makefile.

The MPICH implementation of MPI provides the commands mpicc and mpif77 as well as Makefile examples in:
 

/usr/local/mpi/examples/Makefile.in

Note: There are many implementations of MPI. MPICH is a freely available and portable implementation.
 
 

The commands
 

mpicc -o first first.c
mpif77 -o firstf firstf.f

may be used to build simple programs when using MPICH.

These provide special options that exploit the profiling features of MPI
 

-mpilog
    Generate log files of MPI calls
-mpitrace
    Trace execution of MPI calls
-mpianim
    Real-time animation of MPI (not available on all systems)

 

There are specific to the MPICH implementation; other implementations may provide similar commands (e.g., mpcc and mpxlf on IBM SP2).
 

Using Makefiles:
 

The file Makefile.in is a template Makefile. The program (script) mpireconfig translates this to a Makefile for a particular system. This allows you to use the same Makefile for a network of workstations and a massively parallel computer, even when they use different compilers, libraries, and linker options.
 

mpireconfig Makefile

 

Note that you must have mpireconfig in your PATH.
 
 

 
A Sample Makefile.in file

 

Running MPI programs:
 
 

mpirun -np 2 hello

mpirun is not part of the standard, but some version of it is common with several MPI implementations. The version shown here is for the MPICH implementation of MPI.

Just as Fortran does not specify how Fortran programs are started, MPI does not specify how MPI programs are started.

The option -t shows the commands that mpirun would execute; you can use this to find out how mpirun starts programs on your system. The option -help shows all options to mpirun.
 

Finding out about the environment:

Two of the first questions asked in a parallel program are: How many processes are there? and Who am I?

How many is answered with MPI_Comm_size and who am I is answered with MPI_Comm_rank.

The rank is a number between zero and size-1.

 
 
 Homework1