MPI is a C program that can run at multiple processors, and the processors can be at multiple machines. It will SSH into the remote machines and run the task on the processor
Install MPI
sudo apt install openmpi-bin openmpi-common libopenmpi-dev
MPI has its own compiler and runtime:
mpicc --version
mpiexec --version
Compile MPI
mpicc -o hello-mpi hello-mpi.c
Run MPI
mpiexec hello-mpi
(This will execute hello with 16 processors because my machine has 16 processors. If ./hello-mpi is invoked directly, hello will be executed just once.)
Create a hostfile:
localhost slots=4
Execute the hostfile to run with just 4 processors:
mpirun --hostfile hostfile hello-mpi
This will run in 4 processors.
You can directly run with just 4 processors without specifying a host file:
mpirun -np 4 hello-mpi
If the number of processors exceeds the system has, the request will error out. For example:
mpirun -np 17 hello-mpi
(This resulted in errors on my machine with "There are not enough slots available in the system to satisfy the 17 slots that were requested by the application." Similar error would happen if -np and --hostfile is combined and -np exceeds the slots on a machine defined in hostfile.）
mpirun and mpiexec can be used exchangeable.
How to Get Result from Remote
Results on different machines can be reduced with MPI_Reduce(), usually on machine 0. For example: to sum the result:
double local_result = compute_local_result();
double global_result;
MPI_Reduce(&local_result, &global_result, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Allreduce will reduce the result and send to all machines.
Use MPI_Gather to collect results from all nodes as an array of items into a node (node 0) without reducing.
double local_result = compute_local_result();
double all_results[num_processes];
MPI_Gather(&local_result, 1, MPI_DOUBLE, all_results, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
MPI_Allgather will collect results in an array and make available in all machines.
You can also customize the communication among the nodes. For example, to collect results from all other MPI nodes in node 0.
if (rank != 0) {
MPI_Send(&local_result, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD);
} else {
for (int i = 1; i < num_processes; i++)
MPI_Recv(&worker_result, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
How to trigger Python with MPI?
Instead of running a C program, use mpiexec to call a python program. For example,
mpirun -np 4 python 1.py
Of course, your python script needs MPI support MPI; otherwise, it would run the same content. Install mpi4py by:
sudo apt-get update
sudo apt-get install libopenmpi-dev
conda install python=3.10
conda install mpi4py
Copy and run the basic example in https://mpi4py.readthedocs.io/en/stable/tutorial.html, and run:
mpiexec -np 2 python 2.py
References:
https://mpitutorial.com/tutorials/mpi-hello-world/
https://mpitutorial.com/tutorials/mpi-reduce-and-allreduce/
https://mpi4py.readthedocs.io/en/stable/tutorial.html
