Package Testing¶
This guide explains how to use the built-in package testing facilities inside the vlbuildbot build system.
How it works¶
The vlbuildbot system offers the ability to perform tests on the resulting
package(s) created by the .SlackBuild
file. The testing system is pretty
much wide open for the contributor to write their own tests for the package
they are contributing. In a nutshell, this is how the process goes.
- The
.SlackBuild
file is executed. If this exits in error, the build is marked as failed, and everything halts there. - After the
.SlackBuild
exits, the bot will look for a file named<package_name>.tests.sh
,where <package_name>
refers to the name of the program you are building. For example, thehtop/src
directory will containhtop.SlackBuild
andhtop.tests.sh
if testing is desired. If the.tests.sh
file is found, the bot will install the package it just created and execute the<package_name>.tests.sh
file. If this program returns an error, the build is marked as failed, and the package is discarded.
How to write tests¶
The syntax to a tests.sh
file is just plain shell script. You can assume
the resulting package has been installed by the time this script is executed.
The idea behind te tests file is to verify that a package works as it should, so you could run any number of tests against the package. For instance
- Check the existance|permissions|version of a file after the package is installed.
- Use the installed program to perform a simple task to make sure it works as it should.
Sample use case¶
When testing packages like python
, perl
, lua
or any other programming
language package for example, you could also place a sample program written in
that language and execute it from your tests.sh
file.
Requirements on the tests script¶
Generally, the tests script is wide open to whatever you want to test for using
shell syntax. In order to properly work though, the following criteria must be
met by the .tests.sh
program.
The tests program must follow the same naming convention used throughout the vlbuildbot system. For instance, the
htop
program will have ahtop.SlackBuild
and ahtop.tests.sh
file is testing is desired after the build completes.- The script must return a value (ie, if it errors out, return a value greater than 1).
Not having this will produce false positives and the render testing useless
The script must be placed in the same directory as the
.SlackBuild
All resources required by the
.tests.sh
must be provided and placed
on the same location as the tests file, ie, a testhello world
program to be compiled as a test.
Note
The tests are performed in the same environment that was left after the package
was built. All packages installed as MAKEDEPENDS
of the .SlackBuild
are still present when the tests run. Any run-time dependencies your package
may need, and that are not present on the default building environment can be
installed via slapt-get
from the tests.sh
program.
Known Limitations¶
As of the time of this writing, the package testing has the following known limitations.
- X applications cannot be launched from the
.tests.sh
file. This is because the X Window System is not reachable by the docker container where the tests are ran. This limits the ability for instance to launch a GUI application and take a screenshot.
Note
Due to the fact that this is the first attempt at automatic package testing, more limitations or caveats to this rough-draft implementation may surface. In a case like that, a bug report should be filed at http://bitbucket.org/VLCore/vlbuildbot