================================ Package Testing ================================ This guide explains how to use the built-in package testing facilities inside the vlbuildbot build system. How it works ============ The vlbuildbot system offers the ability to perform tests on the resulting package(s) created by the ``.SlackBuild`` file. The testing system is pretty much wide open for the contributor to write their own tests for the package they are contributing. In a nutshell, this is how the process goes. - The ``.SlackBuild`` file is executed. If this exits in error, the build is marked as failed, and everything halts there. - After the ``.SlackBuild`` exits, the bot will look for a file named ``.tests.sh``, ``where `` refers to the name of the program you are building. For example, the ``htop/src`` directory will contain ``htop.SlackBuild`` and ``htop.tests.sh`` if testing is desired. If the ``.tests.sh`` file is found, the bot will install the package it just created and execute the ``.tests.sh`` file. If this program returns an error, the build is marked as failed, and the package is discarded. How to write tests ================== The syntax to a ``tests.sh`` file is just plain shell script. You can assume the resulting package has been installed by the time this script is executed. The idea behind te tests file is to verify that a package works as it should, so you could run any number of tests against the package. For instance - Check the existance|permissions|version of a file after the package is installed. - Use the installed program to perform a simple task to make sure it works as it should. Sample use case --------------- When testing packages like ``python``, ``perl``, ``lua`` or any other programming language package for example, you could also place a sample program written in that language and execute it from your ``tests.sh`` file. Requirements on the tests script ================================ Generally, the tests script is wide open to whatever you want to test for using shell syntax. In order to properly work though, the following criteria must be met by the ``.tests.sh`` program. #. The tests program must follow the same naming convention used throughout the vlbuildbot system. For instance, the ``htop`` program will have a ``htop.SlackBuild`` and a ``htop.tests.sh`` file is testing is desired after the build completes. #. The script must return a value (ie, if it errors out, return a value greater than 1). Not having this **will produce false positives and the render testing useless** #. The script must be placed in the same directory as the ``.SlackBuild`` #. All resources required by the ``.tests.sh`` must be provided and placed on the same location as the tests file, ie, a test ``hello world`` program to be compiled as a test. .. note :: The tests are performed in the same environment that was left after the package was built. All packages installed as ``MAKEDEPENDS`` of the ``.SlackBuild`` are still present when the tests run. Any run-time dependencies your package may need, and that are not present on the default building environment can be installed via ``slapt-get`` from the ``tests.sh`` program. Known Limitations ================= As of the time of this writing, the package testing has the following known limitations. #. X applications cannot be launched from the ``.tests.sh`` file. This is because the X Window System is not reachable by the docker container where the tests are ran. This limits the ability for instance to launch a GUI application and take a screenshot. .. note :: Due to the fact that this is the first attempt at automatic package testing, more limitations or caveats to this rough-draft implementation may surface. In a case like that, a bug report should be filed at http://bitbucket.org/VLCore/vlbuildbot