=================================================== Packaging guidelines for the VectorLinux buildbot =================================================== Introduction: ============= The packaging procedures for use with the VectorLinux Buildbot are a bit different, although the principles are still the same and some of the steps on the procedure remain unchanged. In the past, one would normally generate a build script (SlackBuild in most cases), then one would get the source code for the application as well as a fitting slack-desc file. When it was built, one would upload a complete source directory containing a tarball, description, and build script as well as the binary package itself (.txz as of VectorLinux 7.0). With the buildbot system, things change a little bit. You still need to generate a script (See Buildscript Guidelines below), and you still need a description file. However, the only thing you upload is the build script and any supporting files it needs (slack-desc, patches, etc). See the guide for submitting a package below. Why use a buildbot? =================== * Allows us to build packages for multiple architectures virtually parallel to each other. This means that the x86_64 version of every package in our repositories will be available at the same time the x86 package is ready, without the need for anyone to do any extra work. * Makes package maintenance easier. * Would allow us to release security fixes in a timely manner. * Makes the job of the repo maintainer easier. * Enforces consistency during the development of the distro by forcing developers and contributors to re-use the existing tools and resources as much as possible. * Allows us to see which packages are failing and why so we can address the changes and get them fixed. Submitting a new package ======================== The following steps should be taken to submit a new package to the database. #. Generate the build script with sbbuilder. (See Buildscript Guidelines below). #. Get or write a slack-desc file for the application you are building. You can usually find them online. If they don't exist, you need to create one. #. Execute the script to make sure it builds on your box. Mainly, this will make sure the LINK and build procedures are correct on the SlackBuild. Make any corrections to the SlackBuild as necessary. #. Submit your SlackBuild, slack-desc and any other supporting files (patches, icons, .desktop files, etc) to the appropriate git repository for the vector release it should be built for. ie, for vector 7.1, upload your work to http://bitbucket.org/VLCore/vl71 SlackBuild Guidelines: ====================== **All** SlackBuilds (with very few exceptions approved by a buildbot maintainer) submitted for inclusion on the buildbot system **must** meet the following requirements: * Must be generated by sbbuilder. (some modifications for sbbuilder will be released soon for including stuff useful for the bot) * Must include a value in the ``LINK`` array. * Must list any and all dependencies required *to build the package* in an array named ``MAKEDEPENDS``. This is an array separated by a single space character. See 'Handling Dependencies' below. * The script must be sourceable. Meaning, we should be able to source the script from a shell without running it. This can be accomplished by adding a ``if [ "$NORUN" != 1 ]; then`` at any point after the BUILD ENVIRONMENT has been set in the script and the source is extracted. The ``if`` must be closed at the end of the script. * Must be tested by the contributor to make sure it is free of syntax errors and to make sure it downloads its source package correctly. * Must be accompanied by a slack-desc and any other files the SlackBuild needs to run successfully. This includes any patches, icons, .desktop files, etc. All of these apply to every single submission. There will be special cases where an exception to the rules needs to be made. In such cases, communication between the package maintainer and the buildbot maintainer will determine how it will be implemented. Editing existing (vectorlinux) scripts for compatibility with vlbuildbot ======================================================================== Existing vectorlinux build scripts can be adapted for compatibility with vlbuildbot. Most scripts only require 2 small adjustments. * Fill in the ``LINK`` array. * Add the ``if [ "$NORUN" != 1 ]`` condition. .. note:: DO NOT hard code package names and version numbers into ``LINK``. The string in ``LINK`` must read something like: ``http://somehost.com/$NAME/$NAME-$VERSION.tar.gz`` instead of: ``http://somehost.com/foo/foo-1.0.tar.gz``. After this, simply test the script to make sure it does indeed download the source code, and submit Handling Dependencies ===================== The buildbot is equipped with a small tool to resolve any dependencies needed to build your package. But you must list them in the ``MAKEDEPENDS`` array of your SlackBuild. To solve these, the bot will first look in the official VectorLinux repositories. If a dependency is not in the repositories, it will look in the source repository for the release it's working for to see if we have build scripts that will produce the package needed. If a dependency is not found at either location, your build will fail immediately. .. note:: The ``MAKEDEPENDS`` array should only list dependencies needed **to build** your package, not to run it. Requiredbuilder will list all the dependencies needed to run the application you just packaged. .. note:: Listing packages in ``MAKEDEPENDS`` forces the build slaves to always use the latest versions of the listed packages available. How to list your build-time dependencies ---------------------------------------- Consider the following scenario. You are building package 'foo'. But when you run the SlackBuild locally (for testing before submission), you find that the configure step needs 'libbar'. You would do the following. * Edit your SlackBuild to add 'libbar' to your ``MAKEDEPENDS``. * Install libbar on your local system so you can continue to test the SlackBuild. * Run the SlackBuild again. Repeat those steps for every build dependency you need to list for your SlackBuild. If you need to depend on a package that is currently not in the official repositories, you will have to submit buildscripts for your dependency too. .. note:: In rare cases, the package you are building may conflict with another package, or will not build if a certain package is installed. To have the build slave remove a package from the build environment before building your package, list your conflicting package with a "!" prefix (ie ``"MAKEDEPENDS="!foobar"``). .. note:: The default bahaviour for resolving build-time dependencies is as follows to first attempt to install the package from the repositories (via slapt-get). If that fails, then the package listed in ``MAKEDEPENDS`` will be built from source, and installed in the build environment before the requested package is built. **If the package fails to install from the online repos and from source code, the build for the requested package will fail until the dep is question is resolved.** The life of a package ===================== The following explains how a package is created and how it ends up in the repositories. #. A contributor (packager) submits a buildscript to the git repository. #. The buildbot master will detect the change on the git commit logs and instruct the build slaves to build the new package. #. The build slave receives notification from the master that a new package needs to be built, and runs the script submitted by the contributor. #. The build slave notifies the master of the build results. If the build was successful, the binary package and source code are uploaded to http://vlcore.vectorlinux.com/pkg/ #. Once the package makes it to vlcore.vectorlinux.com, it is picked up by other tools for processing and submission to the official VectorLinux repositories. How the bot works ================= The VectorLinux buildbot is a special configuration of the buildbot tool found at http://buildbot.net It consists of at least 2 parts. One master, and at least one slave. The Master ---------- The buildbot master is responsible for the following tasks: * Monitor the git source repository for changes. * Assign tasks for each build slave as necessary. * Collect results data and build logs from the build slaves. * Expose build results and logs via a web ui. * Provide an IRC bot as another way to force builds and notify the dev channel of events taking place. * (currently disabled) Provide email notification of triggered events. The Slave --------- The buildbot slave(s) are responsible for the following tasks: * Receive instructions from the build master. * Carry out the actual build process when instructed by the master. * Relay build results and log files back to the master. * Upload the resulting binary package and source tree to the pool location.