Open MPI Quick Reference

This document serves as a reference for getting started using Open MPI in C++.

Compiling MPI Applications
C++ MPI applications should be built using the mpiCC command: $ mpiCC -c program.cpp -o program.o $ mpiCC -o program program.o

Note that mpiCC is a wrapper around g++, so all arguments that work with g++ will work with mpiCC. The purpose of mpiCC is to include the required header and library files in a standard way. For more information, please see or the latest version of the Open MPI documentation.

Running MPI Applications using mpirun
The mpirun program is a powerful tool used to run Open MPI applications. It allows the developer to choose how many processes to use, which computers to run the job on, and even which specific cores processes and ranks are bound to. The basic use of mpirun is: $ mpirun -np NUM_PROCESSES This runs the specified MPI application on the local machine using the specified number of processes (an integer >= 1). Another (perhaps more common) use is: $ mpirun -np NUM_PROCESSES -H host1,host2,host3 This command is similar to the first, except for that it defines a list of hosts on which to run the job. In this case the job is not limited to running on the local machine. Another way to achieve similar behavior is to define a hostfile and use it instead of a host list: $ mpirun -np NUM_PROCESSES --hostfile

Hostfiles
Hostfiles are used by mpirun to determine which hosts to run a job on. A similar file supported by Open MPI is the rankfile, which is like a host file but is more specific (see manpages for more information). To demonstrate hostfiles, here is a hostfile that defines helios:

h01 slots=2 h02 slots=2 h03 slots=2 h04 slots=2 h05 slots=2 h06 slots=2 h07 slots=2 h08 slots=2 h09 slots=2 h10 slots=2 h11 slots=2 h12 slots=2 h13 slots=2 h14 slots=2 h15 slots=2 h16 slots=2

Keep in mind that other options may be used to further specify how hosts should be used. Hostfiles are mainly a convenient way to reuse a list of hosts (such as queues in SGE).

Running MPI Applications using SGE
Sun Grid Engine (SGE) is a job and resource manager for cluster computing. From a user-perspective, it is used to intelligently decide which nodes to run jobs on, automatically collect job output, and provide information about the status of the job. In particular it helps to distribute the load on a cluster when many jobs are running at the same time. Jobs may be run through SGE by providing a submit script. A typical submit script looks like this:

# File: job.sh #!/bin/sh #$ -S /bin/sh #$ -pe ompi 8 #$ -cwd #$ -j y #$ -N TestJob export LD_LIBRARY_PATH=/usr/local/lib mpirun /home/username/jobdir/job

Note that mpirun is being used, but no information is being provided to it aside from the executable. This is because SGE is able to give mpirun information about which hosts to use (etc.) directly. In the above script, lines denoted by a #$ are SGE configuration lines. The -pe line specifies which parallel environment to use, and how many processes to launch. The -cwd line specifies that the current working directly should be used. The -j line merges the files for standard output and standard error, and finally, the -N line names the job. Once a submit script is in place, the job may be run by entering the command:

$ qsub job.sh

Running jobs may be monitored either by using the qstat command, or checking the job control section of qmon (the SGE graphical management system). Job output is placed in either the user's home directory, or the CWD if specified.

MPI Basic Program
The following program is a basic MPI application that demonstrates initializing and finalizing MPI, and getting information about each of the running processes.


 * 1) include "mpi.h"
 * 2) include

using std::cout; using std::endl;

int main(int argc, char ** argv) {   int rank = 0; int numberOfProcesses = 0; int nameLength = 0; char processorName[MPI_MAX_PROCESSOR_NAME];

MPI::Init(argc, argv); rank = MPI::COMM_WORLD.Get_rank; numberOfProcesses = MPI::COMM_WORLD.Get_size; MPI::Get_processor_name(processorName, nameLength);

cout << "Node " << rank << " of " << numberOfProcesses << " With processor: " << processorName << endl;

MPI::Finalize; return 0; }