Package Testing

This guide explains how to use the built-in package testing facilities inside the vlbuildbot build system.

How it works

The vlbuildbot system offers the ability to perform tests on the resulting package(s) created by the .SlackBuild file. The testing system is pretty much wide open for the contributor to write their own tests for the package they are contributing. In a nutshell, this is how the process goes.

  • The .SlackBuild file is executed. If this exits in error, the build is marked as failed, and everything halts there.
  • After the .SlackBuild exits, the bot will look for a file named <package_name>.tests.sh, where <package_name> refers to the name of the program you are building. For example, the htop/src directory will contain htop.SlackBuild and htop.tests.sh if testing is desired. If the .tests.sh file is found, the bot will install the package it just created and execute the <package_name>.tests.sh file. If this program returns an error, the build is marked as failed, and the package is discarded.

How to write tests

The syntax to a tests.sh file is just plain shell script. You can assume the resulting package has been installed by the time this script is executed.

The idea behind te tests file is to verify that a package works as it should, so you could run any number of tests against the package. For instance

  • Check the existance|permissions|version of a file after the package is installed.
  • Use the installed program to perform a simple task to make sure it works as it should.

Sample use case

When testing packages like python, perl, lua or any other programming language package for example, you could also place a sample program written in that language and execute it from your tests.sh file.

Requirements on the tests script

Generally, the tests script is wide open to whatever you want to test for using shell syntax. In order to properly work though, the following criteria must be met by the .tests.sh program.

  1. The tests program must follow the same naming convention used throughout the vlbuildbot system. For instance, the htop program will have a htop.SlackBuild and a htop.tests.sh file is testing is desired after the build completes.

  2. The script must return a value (ie, if it errors out, return a value greater than 1).

    Not having this will produce false positives and the render testing useless

  3. The script must be placed in the same directory as the .SlackBuild

  4. All resources required by the .tests.sh must be provided and placed

on the same location as the tests file, ie, a test hello world program to be compiled as a test.

Note

The tests are performed in the same environment that was left after the package was built. All packages installed as MAKEDEPENDS of the .SlackBuild are still present when the tests run. Any run-time dependencies your package may need, and that are not present on the default building environment can be installed via slapt-get from the tests.sh program.

Known Limitations

As of the time of this writing, the package testing has the following known limitations.

  1. X applications cannot be launched from the .tests.sh file. This is because the X Window System is not reachable by the docker container where the tests are ran. This limits the ability for instance to launch a GUI application and take a screenshot.

Note

Due to the fact that this is the first attempt at automatic package testing, more limitations or caveats to this rough-draft implementation may surface. In a case like that, a bug report should be filed at http://bitbucket.org/VLCore/vlbuildbot