Raspberry Pi cluster with MPICH3 and MPI4Py

As there are more out there I created a Raspberry Pi cluster as well. And like the others it starts of with a shopping list:

  • 1 Raspberry Pi 3B (just because I already had one)
  • 4 Raspberry Pi 3b+
  • 1 6 port USB charger
  • 5 ethernet patch cables
  • 5 micro USB cables
  • 1 (cheap) ethernet switch
  • 1 (cheap) WiFi router with internet access

Each other the RPIs will connect to the ethernet switch using the patch cables but each of them will also connect to the WiFi router (as they all have an ethernet port and are WiFi enabled).

One of the RPIs (the 3B as it is one of its kind in this setup) will act as main RPI or controller. It will have a full Stretch install (so including a UI). The other four will be based upon Stretch Lite. The controller will be configured so that it can be accessed via a VNC client as well as via SSH. The controller will also act as the DHCP server for the ethernet based network. The controller will be named rpi3-0. The others will be called rpi3-1, rpi3-2, rpi3-3 and rpi3-4.

I could repeat the base install of the RPIs but I guess there are enough samples available on the web, hereby just one of them and an other one.

This document is just there to provide the code/settings for some of the steps in setting up your cluster. You can change the number of RPIs if you like…

Setting up the ethernet network

To create a network you will need a DHCP server. In this cluster the WiFi network has access to the internet. This is important as the DNS settings for the ethernet network are based upon Google DNS servers.

Update the controller RPI (you will be installing the DHCP server once) – rpi3-0:

sudo apt-get update
sudo apt-get upgrade

Install the DHCP server software:

sudo apt-get install isc-dhcp-server

Configure the DHCP server:

sudo nano /etc/dhcp/dhcpd.conf

Add the following text:

subnet aaa.bbb.ccc.ddd netmask 255.255.255.0 {
range aaa.bbb.ccc.ddd aaa.bbb.ccc.ddd;
option broadcast-address aaa.bbb.ccc.ddd;
option routers aaa.bbb.ccc.ddd;
max-lease-time 7200;
default-lease-time 600;
option domain-name "rpi3";
option domain-name-servers 8.8.8.8, 8.8.4.4;
}

Be aware: I have added pseudo code for the actual ip addresses. I am assuming you can come up with the correct number ranges for your cluster. To give a hint: option routers should referring to the fixed ip address for rpi3-0. Further on you will see the configuration for that. The option broadcast-address I set to aaa.bbb.ccc.255, so the last ip address in your range.

Restart the DHCP server:

sudo service isc-dhcp-server restart

Configure the DHCP server:

sudo nano /etc/default/isc-dhcp-server

Add the following text:

DHCPD_CONF=/etc/dhcp/dhcp.conf
DHCPD_PID=/var/run/dhcpd.pid
INTERFACES="eth0"

Setup your network interfaces:

sudo nano /etc/network/interfaces

Add the following text:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address aaa.bbb.ccc.ddd
netmask 255.255.255.0
post-up iptables-restore < /etc/iptables.ipv4.nat

allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-ssid "ssid"
wpa-psk "password"
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Basically there are three parts: The first one is a network loopback – This enables you to find yourselves. The second part sets the static ip address for rpi3-0 itself. There is also a line referring to a file called iptables.ipv4.nat. Further on you will see the configuration for that. The last part enables you to access the WiFi network. You need to use your SSID and password here.

Some more configuration:

sudo nano /etc/sysctl.conf

Uncomment the next line to enable packet forwarding for IPv4:

net.ipv4.ip_forward=1

The firewall needs some basic configuration (which will result into the setup of the iptables.ipv4.nat file):

sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED
sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT
sudo iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o wlan0 -j MASQUERADE
sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i wlan0 -o eth0 -m state --state RELATED
-A FORWARD -i eth0 -o wlan0 -j ACCEPT
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Although using DHCP on the ethernet network, it would be nice to have fixed ip addresses for the other RPIs. You will need the MAC addresses of the others to do so.

as the rpi3-0 is hosting the DHCP server, you can find out who it is serving. A way to find out what MAC addresses the RPIs have is starting with setting up the DHCP server but not yet configure the fixed ip addresses. Each RPI will connect to rpi3-0 via the ethernet network. You can find out what leases (including MAC addresses) have been given to the RPIs via the following command on rpi3-0:

cat /var/lib/dhcp/dhcpd.leases

Edit the fixed ip addresses for the RPIs:

sudo nano /etc/dhcp/dhcpd.conf

Add the following text:

host rpi3-1 {
  hardware ethernet aa:aa:aa:aa:aa:aa;
  fixed-address aaa.bbb.ccc.ddd;
}

host rpi3-2 {
  hardware ethernet bb:bb:bb:bb:bb:bb;
  fixed-address aaa.bbb.ccc.ddd;
}

host rpi3-3 {
  hardware ethernet cc:cc:cc:cc:cc:cc;
  fixed-address aaa.bbb.ccc.ddd;
}

host rpi3-4 {
  hardware ethernet dd:dd:dd:dd:dd:dd;
  fixed-address aaa.bbb.ccc.ddd;
}

Restart the DHCP server:

sudo service isc-dhcp-server restart

Check the status of the DHCP server:

sudo systemctl status isc-dhcp-server

Edit the hosts file to assign the hostnames to the ip addresses of all RPIs:

sudo nano /etc/hosts

Add the following text:

aaa.bbb.ccc.ddd rpi3-0
aaa.bbb.ccc.ddd rpi3-1
aaa.bbb.ccc.ddd rpi3-2
aaa.bbb.ccc.ddd rpi3-3
aaa.bbb.ccc.ddd rpi3-4

You need to add your assigned ip address. I am not sure if you need to do this on rpi3-0 as it assigns the ip address to the hosts but you need this file on the other RPIs anyway.

Do a reboot: Just because you can!

sudo reboot

Setting up the other cluster nodes

I will not be going into all details but in a nutshell (you will do this for all others RPIs):

sudo raspi-config

Set all to us, ssh on, change memory and no predictable network names.

Update the RPIs – rpi3-1, rpi3-2, rpi3-3 and rpi3-4:

sudo apt-get update
sudo apt-get upgrade

Setup your network interfaces:

sudo nano /etc/network/interfaces

Add the following text:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-ssid "ssid"
wpa-psk "password"
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Basically there are three parts: The first one is a network loopback – This enables you to find yourselves. The second part sets the dynamic ip address for the RPIs. The last part enables you to access the WiFi network. You need to use your SSID and password here.

Edit the hosts file to assign the hostnames to the ip addresses of all RPIs:

sudo nano /etc/hosts

Add the following text:

aaa.bbb.ccc.ddd rpi3-0
aaa.bbb.ccc.ddd rpi3-1
aaa.bbb.ccc.ddd rpi3-2
aaa.bbb.ccc.ddd rpi3-3
aaa.bbb.ccc.ddd rpi3-4

You will need this file use the hostnames of the RPIs to access them instead of the ip addresses.

Do a reboot: Just because you can!

sudo reboot

Passwordless SSH access

All the RPIs in the cluster will access each other via SSH. This communication needs to be passwordless aka they need to access only by using a user id. You will need to this for all combinations of RPIs. In this case each RPI can potentially access the 4 other RPIs, so you need to this per RPI for all other RPIs.

I will explain this for rpi3-0 only but is to be repeated for the others.

Generate your SSH keys (for more information, look here). Be aware: No passphrass:

ssh-keygen

Copy a key to the hosts you want to access (for more information, look here):

ssh-copy-id pi@rpi3-1

You need to repeat this for the rpi3-2, rpi3-3 and rpi3-4 combinations with rpi3-0 and for all the other combinations.

MPICH3

As I am new to this technology I can not say much about it but go visit www.mpich.org. As it is not really difficult to install, it takes some time. So be patient and it needs to be installed on all RPIs.

First download and install:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo mkdir mpich3
cd ~/mpich3
sudo wget http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz
sudo tar xfz mpich-3.2.1.tar.gz
sudo mkdir /home/rpimpi/
sudo mkdir /home/rpimpi/mpi-install
mkdir /home/pi/mpi-build
sudo apt-get install gfortran
sudo /home/pi/mpich3/mpich-3.2.1/configure -prefix=/home/rpimpi/mpi-install
sudo make
sudo make install

Make it findable:

cd ..
nano .bashrc

Add this at the end:

PATH=$PATH:/home/rpimpi/mpi-install/bin

Do a reboot: Just because you can!

sudo reboot

Do a test:

mpiexec -n 1 hostname

mpiexec1

MPI4Py

As I am new to this technology I can not say much about it but go visit MPI4Py. This too needs to be installed on all RPIs.
First download and install:

wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-2.0.0.tar.gz
sudo tar -zxf mpi4py-2.0.0.tar.gz
cd mpi4py-2.0.0
sudo aptitude install python-dev
python setup.py build
sudo python setup.py install
export PYTHONPATH=/home/pi/mpi4py-2.0.0

Do a test:

mpiexec -n 5 python demo/helloworld.py

mpiexec2

After having done this for all cluster RPIs, you have a bunch of standalone RPIs. You need to make them aware of the each other. For that you need to create a machinefile. Basically it is a list of all RPIs in the cluster and each RPI will need this file.

Create the file:

cd
nano machinefile

Add these lines:

rpi3-0
rpi3-1
rpi3-2
rpi3-3
rpi3-4

So now you have the list of cluster RPIs on all RPIs.

Do a test:

mpiexec -f machinefile -n 5 hostname

mpiexec3

Do another test:

mpiexec -f machinefile -n 5 python /home/pi/mpi4py-2.0.0/demo/helloworld.py

mpiexec4

As you can see, now all 5 RPIs in the cluster are participating!

 

 

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s