============================= Vlbuildbot (bot) maintenance ============================= Maintenance on the bot master. ============================== The buildbot master needs to be restarted when at least one of the following events take place. * Adding or removing an application from the manifest. * Adding or removing a slave from the bot system. * Changes to the git polling frequency intervals. * Changes in the master's master.cfg configuration Changes are only applied when the bot master is restarted. Adding a new slave to the master. --------------------------------- To add a new slave on the vlbuildbot system, the bot administrator needs to follow these guidelines: Edit ``slavenodes`` on the master's home directory following the existing syntax. Each line represents one slave and is setup to contain four fields. ``node_name | password | build_type | build_capacity``. * ``node_name`` represents the name of the slave. * ``password`` sets the password the slave will use to authorize with the master * ``build_type`` is one of "pkg" or "iso". Slaves set to "pkg" will build packages while others set to "iso" will build ISO images from packages. * ``build_capacity`` represents the number of simultaneous builds this slave can run at the same time. .. note:: The buildbot's home directory is the directory containing the master.cfg file Maintenance on the slave. ========================= The slaves almost never need to be maintained, however, the status of each of the attached slaves can be seen at http://vlcore.vectorlinux.com/buildbot/buildslaves That page presents a table 7 columns. The column to the far right indicates the current status for each slave. You may need to scroll down the page to see the status because we have a lot of builders listed. If the status says ``offline`` for any of the slaves, usually a slave restart will take care of this. Updating the build environments on the slaves. ---------------------------------------------- Buildslaves are setup to automatically check the ``VLCore/vlbuildbot`` git tree for available updates. Just about any update can be performed by pushing the necessary instructions to the git tree. The ``updated`` directory on that git tree contains a file named ``update.conf``. This file is used to define in simple ``BASH`` script how the update is rolled out. Only (2) parts of the file **MUST** be changed for an update to be executed. #. ``REMOTEVERSION``. This is normally set to a date value (ie 20130423). The update system compares this value to the existing value in the build environment and determines if an update is needed. #. ``function update_slave()``. This bash function contains the instructions that make up the update. Wether the update consists of renaming a file, or updating the entire build environment with new ISO's, it's all done here. Rolling out updates this way helps a lot with maintenance on the buildbot system for VectorLinux. Since every slave is watching this git repository, all slaves will be updated simultaneously. .. note:: If a slave is busy at the time the update is detected, the slave will request a graceful shutdown from the master. This means the slave will shut itself down when the running build finishes (pass or fail). Then the bot will update itself and come back online. .. note:: Build slaves are setup to check for updates every hour, so not all slaves will be updated at the exact same time. Preparing a new slave. ====================== Having multiple slaves attached to the buildbot master allows for simultaneous maintenance of all tracked packages and spreading the workload between all the active buildslaves. When more than one slave is suited to build the requested package, the buildbot will pick one at random to build. The following steps can be followed to prepare a new slave and get it ready for connecting to the buildbot master. Minimun Requirements (on the slave host) --------------------------------------- To host a slave instance, you will need 64-bit hardware and at the very least meet the following requiements. - Hardware - 1024M RAM. - 30GB available disk space. - 2GB swap space. - Software (must be installed and tested to work inside the isolated environment) - procmail - git - dev-base (metapackage) - kernel-headers - setuptools - lftp - wget Preparing the slave host ------------------------ Build slaves normally run in an isolated environment. This can be either a virtual machine or a LXC container. This guide will not cover how to set that up, but will assume the procedure is performed within an isolated environment. For a quick guide on LXC containers on VectorLinux or VLocity, see http://vlcoredocumentation.readthedocs.org/en/latest/manuals/lxc-containers.html The ``deployment`` directory of the ``VLCore/vlbuildbot`` git tree contains the necessary tools to deploy a new slave. The following procedure must be carried out to deploy a new slave #. ``git clone http://bitbucket.org/VLCore/vlbuildbot`` #. ``cd vlbuildbot/deployment`` #. Edit ``mkslave.sh`` and fill in the following fields: * ``SLAVE_NAME``: Should be the node name on the master for this slave * ``SLAVE_PASSWORD``: Password used to authorize this slave at the master. * ``MASTER_HOST``: IP or URL to the bot master * ``MASTER_SLAVE_PORT``: Port on which the master expects slave connections. #. ``cp ../slave/etc/vlbuildslave/slavehost.conf .`` #. Edit ``slavehost.conf`` and set the following fields as follows: * ``SLAVES_ROOT``: Path to where the slave will live. * ``JAILS``: Path to directory where the jails will be kept. .. note:: This directory will require lots of disk space. This is where the read-only areas are kept for clean builds. This could require up to 20GB of disk space (subject to change) * ``REPOS_HOME``: This is where the git clones of the SlackBuild repositories will be kept. * ``TOOLSDIR``: This should always be set to ``/usr/local/bin`` Launching the slave deployment process -------------------------------------- After all the above steps have been taken, you will need to launch the deployment process from the same directory your mkslave.sh is at. #. Make sure the correct timezone has been set on your host. This can be checked by making sure ``/etc/localtime`` and ``/etc/localtime-copied-from`` point to a valid symlink. If they do not, fix it before you start the deployment process. #. ``export VLBB_REPO=$PWD/..`` #. ``export CMD_MKCHROOT=$VLBB_REPO/slave/sbin/mkchrootSB`` #. ``export SLAVECONF=$PWD/slavehost.conf`` #. ``sh mkslave.sh`` After deployment ---------------- After the slave has been deployed, you will need to do a couple of things before the slave can be used by the master. #. ``source /etc/vlbuildslave/slavehost.conf`` #. ``cd $REPOS_HOME`` #. ``git clone http://bitbucket.org/VLCore/vl70`` #. ``git clone http://bitbucket.org/VLCore/vl71`` #. ``git clone http://bitbucket.org/VLCore/vl72`` #. You should also provide the bot master's administrator with a set of your root's ssh public keys. This is needed to allow your slave to upload built packages. Test your setup --------------- You can test your new slave by manually launching a build as follows. ``/usr/local/bin/vlbb-jailedbuild -p htop -i 1 -a i586 -v veclinux-7.1`` This will trigger a local build of htop for 7.1 32-bit on your isolated environment. Repeat the step by issuing the command and changing the ``i586`` to ``x86_64`` and the ``veclinux-7.1`` to ``veclinux-7.0`` or ``veclinux-7.2``. As a rule of thumb, a slave should be tested with the above procedure before connecting it to the master. The slave should be tested to build a package for every release for every architecture supported. If any one build process fails, the slave is not working and cannot be connected to the master. When testing a slave, use a simple package to build. Htop is a good package to use because it does not take much to build. Normally, if a slave cannot build htop, it will not work at all. If all tests pass, then the slave is ready to be connected to the master and to begin receiving build tasks. Starting and Stopping the slave ------------------------------- After your slave has been setup and tested to be working, and you have been given the assigned node name and password for your slave, you may start the slave by using the provided rc script ``/etc/rc.d/rc.vlbuildslave [ start | stop ]`` will start or stop the slave. .. note:: The deployment process will begin by first downloading all of the necessary ISO images to create the required jails. This will take time, bandwidth and disk space. .. note:: In the VectorLinux buildbot system, all buildslaves are *required* to be able to build for both the x86 and x86_64 architectures. This can be easily acheived by setting up the 32 and 64b chroots properly. Setting the slave up in a virtual machine of any kind allows more flexibility and independence from the real hardware.