3. piw-slave

The piw-slave script is intended to be run on a standalone machine to build packages on behalf of the piw-master service. It is intended to be run as an unprivileged user with a clean home-directory. Any build dependencies you wish to use must already be installed. The script will run until it is explicitly terminated, either by Ctrl+C, SIGTERM (see signal(7)), or by the remote piw-master script.

3.1. Synopsis

piw-slave [-h] [--version] [-c FILE] [-q] [-v] [-l FILE] [--debug]
          [-m HOST] [-t DURATION] [-d DIR] [-L STR]

3.2. Description

-h, --help

Show this help message and exit


Show program’s version number and exit

-c FILE, --configuration FILE

Specify a configuration file to load

-q, --quiet

Produce less console output

-v, --verbose

Produce more console output

-l FILE, --log-file FILE

Log messages to the specified file


Set logging to debug level

-m HOST, --master HOST

The IP address or hostname of the master server (default: localhost)


The time to wait before assuming a build has failed (default: 3h)

-d DIR, --dir DIR

The temporary directory to use when building wheels (default: /tmp)

-L STR, --label STR

The label to transmit to the master identifying this build slave (default: the local hostname)

3.3. Deployment

Our typical method of deployment is to spin up a new Pi as a build slave (through Mythic Beasts’ control panel) then execute a script to install the piwheels code, and all the build dependencies that we feel are reasonable to support under various Raspbian versions. The deployment script can be found in the root of the piwheels repository:


set -eu

if [ $# -ne 2 ]; then
    echo "Usage: deploy_slave.sh HOSTNAME MASTER_IP"
    exit 1

echo $1 > /etc/hostname
echo "[slave]" > /etc/piwheels.conf
echo "master=$2" >> /etc/piwheels.conf


sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config

source /etc/os-release

PYTHON2_PACKAGES="ipython python-pip python-dev python-scipy python-matplotlib \
    python-pandas python-yaml libpng-dev python-lxml python-cffi python-bs4 \
    python-click python-sqlalchemy python-pil python-pymongo python-django \
    python-flask python-cherrypy python-tornado python-redis python-dateutil \
    python-dnspython python-sphinx python-boto python-gi python-gi-cairo \
    python-cairocffi python-numpy"

if [ $VERSION_ID -eq 10 ]; then
elif [ $VERSION_ID -eq 11 ]; then

apt update
apt -y upgrade
apt -y install vim wget curl ssh-import-id tree byobu htop pkg-config cmake time pandoc \
    gfortran ipython3 git qt5-qmake python3-dev python3-pip python3-apt \
    zlib1g-dev libpq-dev libffi-dev libxml2-dev libhdf5-dev libldap2-dev \
    libjpeg-dev libbluetooth-dev libusb-dev libhidapi-dev libfreetype6-dev \
    liblcms2-dev libzbar-dev libbz2-dev libblas-dev liblapack-dev \
    liblapacke-dev libcurl4-openssl-dev libgmp-dev libgstreamer1.0-dev \
    libsdl2-dev libsdl2-image-dev libsdl2-mixer-dev libsdl2-ttf-dev libssl-dev \
    libsasl2-dev libldap2-dev libavcodec-dev libavformat-dev libswscale-dev \
    libv4l-dev libxvidcore-dev libx264-dev libgtk2.0-dev libgtk-3-dev \
    libatlas-base-dev python3-numpy python3-cairocffi libsdl-image1.2-dev \
    libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsdl1.2-dev libportmidi-dev \
    libtiff5-dev libx11-6 libx11-dev xfonts-base xfonts-100dpi xfonts-75dpi \
    xfonts-cyrillic fluid-soundfont-gm libsystemd-dev libusb-1.0-0-dev \
    libudev-dev libopus-dev libvpx-dev libc-bin libavdevice-dev libadios-dev \
    libavfilter-dev libavutil-dev libcec-dev lsb-release pybind11-dev \
    libsnappy-dev libpcap0.8-dev swig libzmq5 portaudio19-dev libqpdf-dev \
    coinor-libipopt-dev libsrtp2-dev default-libmysqlclient-dev golang \
    $PYTHON2_PACKAGES $QMAKE libgphoto2-dev libsqlite3-dev libsqlcipher-dev \
    ninja-build $FPRINT

apt purge python3-cryptography -y

pip3 install setuptools --upgrade
pip3 install pip --upgrade
hash -r

pip3 install pypandoc versioneer kervi scikit-build cython numpy scipy \
    setuptools_rust conan cbor2 \
    --upgrade --extra-index-url https://www.piwheels.org/simple --prefer-binary

getent passwd piwheels && userdel -fr piwheels
getent group piwheels || groupadd piwheels
getent passwd piwheels || useradd -g piwheels -m -s /bin/bash piwheels
passwd -d piwheels

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
cat .cargo/env >> /home/piwheels/.bashrc

if [ -d piwheels ]; then
    cd piwheels
    git pull
    pip3 uninstall -y piwheels
    git clone https://github.com/piwheels/piwheels
    cd piwheels

cp piwheels-slave.service /etc/systemd/system/
systemctl enable piwheels-slave.service
pip3 install .[slave]

fallocate -x -l 1G /swapfile
chmod 0600 /swapfile
mkswap /swapfile
echo "/swapfile none swap x-systemd.makefs,nofail 0 0" >> /etc/fstab
systemctl daemon-reload

rm -f /etc/pip.conf


# chmod +x deploy_slave.sh
# ./deploy_slave.sh myname 1234:abcd::1

However, you will very likely wish to customize this script for your own purposes, e.g. to support a different set of dependencies, or to customize the typical build environment.

Once the script is complete, the machine should reboot, automatically connect to the master node and start building packages as and when the build queue becomes populated. If you need to manually execute the application (assuming the master’s IP address is 1234:abcd::1:

# su - piwheels
$ piw-slave -m 1234:abcd::1

3.4. Automatic start

If you wish to ensure that the build slave starts on every boot-up, you may wish to define a systemd unit for it. Example units can be also be found in the root of the piwheels repository:

Description=The piwheels slave service
After=local-fs.target network.target

ExecStart=/usr/local/bin/piw-slave -v

# systemctl daemon-reload
# systemctl enable piwheels-slave
# systemctl start piwheels-slave


Be aware that this example unit forces a reboot in the case that the build slave fails (as occasionally happens with excessively complex packages).

Because of this you must ensure that the slave executes successfully prior to installing the unit, otherwise you’re liable to leave your build slave in permanent reboot cycle. This isn’t a huge issue for a build slave that’s physically in front of you (from which you can detach and tweak the storage), but it may be an issue if you’re dealing with a cloud builder.