Vlbuildbot (bot) maintenance

Maintenance on the bot master.

The buildbot master needs to be restarted when at least one of the following events take place.

  • Adding or removing an application from the manifest.
  • Adding or removing a slave from the bot system.
  • Changes to the git polling frequency intervals.
  • Changes in the master’s master.cfg configuration

Changes are only applied when the bot master is restarted.

Adding a new slave to the master.

To add a new slave on the vlbuildbot system, the bot administrator needs to follow these guidelines:

Edit slavenodes on the master’s home directory following the existing syntax. Each line represents one slave and is setup to contain four fields. node_name | password | build_type | build_capacity.

  • node_name represents the name of the slave.
  • password sets the password the slave will use to authorize with the master
  • build_type is one of “pkg” or “iso”. Slaves set to “pkg” will build packages while others set to “iso” will build ISO images from packages.
  • build_capacity represents the number of simultaneous builds this slave can run at the same time.

Note

The buildbot’s home directory is the directory containing the master.cfg file

Maintenance on the slave.

The slaves almost never need to be maintained, however, the status of each of the attached slaves can be seen at http://vlcore.vectorlinux.com/buildbot/buildslaves

That page presents a table 7 columns. The column to the far right indicates the current status for each slave. You may need to scroll down the page to see the status because we have a lot of builders listed. If the status says offline for any of the slaves, usually a slave restart will take care of this.

Updating the build environments on the slaves.

Buildslaves are setup to automatically check the VLCore/vlbuildbot git tree for available updates. Just about any update can be performed by pushing the necessary instructions to the git tree.

The updated directory on that git tree contains a file named update.conf. This file is used to define in simple BASH script how the update is rolled out.

Only (2) parts of the file MUST be changed for an update to be executed.

  1. REMOTEVERSION. This is normally set to a date value (ie 20130423). The update system compares this value to the existing value in the build environment and determines if an update is needed.
  2. function update_slave(). This bash function contains the instructions that make up the update. Wether the update consists of renaming a file, or updating the entire build environment with new ISO’s, it’s all done here.

Rolling out updates this way helps a lot with maintenance on the buildbot system for VectorLinux. Since every slave is watching this git repository, all slaves will be updated simultaneously.

Note

If a slave is busy at the time the update is detected, the slave will request a graceful shutdown from the master. This means the slave will shut itself down when the running build finishes (pass or fail). Then the bot will update itself and come back online.

Note

Build slaves are setup to check for updates every hour, so not all slaves will be updated at the exact same time.

Preparing a new slave.

Having multiple slaves attached to the buildbot master allows for simultaneous maintenance of all tracked packages and spreading the workload between all the active buildslaves. When more than one slave is suited to build the requested package, the buildbot will pick one at random to build. The following steps can be followed to prepare a new slave and get it ready for connecting to the buildbot master.

Minimun Requirements (on the slave host)

To host a slave instance, you will need 64-bit hardware and at the very least meet the following requiements.

  • Hardware
  • 1024M RAM.
  • 30GB available disk space.
  • 2GB swap space.
  • Software (must be installed and tested to work inside the isolated environment)
  • procmail
  • git
  • dev-base (metapackage)
  • kernel-headers
  • setuptools
  • lftp
  • wget

Preparing the slave host

Build slaves normally run in an isolated environment. This can be either a virtual machine or a LXC container. This guide will not cover how to set that up, but will assume the procedure is performed within an isolated environment. For a quick guide on LXC containers on VectorLinux or VLocity, see http://vlcoredocumentation.readthedocs.org/en/latest/manuals/lxc-containers.html

The deployment directory of the VLCore/vlbuildbot git tree contains the necessary tools to deploy a new slave. The following procedure must be carried out to deploy a new slave

  1. git clone http://bitbucket.org/VLCore/vlbuildbot

  2. cd vlbuildbot/deployment

  3. Edit mkslave.sh and fill in the following fields:

    • SLAVE_NAME: Should be the node name on the master for this slave
    • SLAVE_PASSWORD: Password used to authorize this slave at the master.
    • MASTER_HOST: IP or URL to the bot master
    • MASTER_SLAVE_PORT: Port on which the master expects slave connections.

  4. cp ../slave/etc/vlbuildslave/slavehost.conf .

  5. Edit slavehost.conf and set the following fields as follows:

    • SLAVES_ROOT: Path to where the slave will live.

    • JAILS: Path to directory where the jails will be kept.

      Note

      This directory will require lots of disk space. This is where the read-only areas are kept for clean builds. This could require up to 20GB of disk space (subject to change)

    • REPOS_HOME: This is where the git clones of the SlackBuild repositories will be kept.

    • TOOLSDIR: This should always be set to /usr/local/bin

Launching the slave deployment process

After all the above steps have been taken, you will need to launch the deployment process from the same directory your mkslave.sh is at.

  1. Make sure the correct timezone has been set on your host. This can be checked
by making sure /etc/localtime and /etc/localtime-copied-from point to a valid symlink. If they do not, fix it before you start the deployment process.
  1. export VLBB_REPO=$PWD/..
  2. export CMD_MKCHROOT=$VLBB_REPO/slave/sbin/mkchrootSB
  3. export SLAVECONF=$PWD/slavehost.conf
  4. sh mkslave.sh

After deployment

After the slave has been deployed, you will need to do a couple of things before the slave can be used by the master.

  1. source /etc/vlbuildslave/slavehost.conf
  2. cd $REPOS_HOME
  3. git clone http://bitbucket.org/VLCore/vl70
  4. git clone http://bitbucket.org/VLCore/vl71
  5. git clone http://bitbucket.org/VLCore/vl72
  6. You should also provide the bot master’s administrator with a set of your root’s ssh public keys. This is needed to allow your slave to upload built packages.

Test your setup

You can test your new slave by manually launching a build as follows.

/usr/local/bin/vlbb-jailedbuild -p htop -i 1 -a i586 -v veclinux-7.1

This will trigger a local build of htop for 7.1 32-bit on your isolated environment. Repeat the step by issuing the command and changing the i586 to x86_64 and the veclinux-7.1 to veclinux-7.0 or veclinux-7.2.

As a rule of thumb, a slave should be tested with the above procedure before connecting it to the master. The slave should be tested to build a package for every release for every architecture supported. If any one build process fails, the slave is not working and cannot be connected to the master.

When testing a slave, use a simple package to build. Htop is a good package to use because it does not take much to build. Normally, if a slave cannot build htop, it will not work at all.

If all tests pass, then the slave is ready to be connected to the master and to begin receiving build tasks.

Starting and Stopping the slave

After your slave has been setup and tested to be working, and you have been given the assigned node name and password for your slave, you may start the slave by using the provided rc script

/etc/rc.d/rc.vlbuildslave [ start | stop ] will start or stop the slave.

Note

The deployment process will begin by first downloading all of the necessary ISO images to create the required jails. This will take time, bandwidth and disk space.

Note

In the VectorLinux buildbot system, all buildslaves are required to be able to build for both the x86 and x86_64 architectures. This can be easily acheived by setting up the 32 and 64b chroots properly. Setting the slave up in a virtual machine of any kind allows more flexibility and independence from the real hardware.