Links
OpenDNSSEC Developer Wiki
OpenDNSSEC Documentation
SoftHSM Developer Wiki
SoftHSM Documentation
Current location:
This is a step by step HowTo develop tests locally so you can test the tests before committing them. It assumes you want to develop a test for OpenDNSSEC trunk, are running Ubuntu Server 10.04.3 and have installed required packages described here plus subversion. You can use any distribution specified in Platforms & HSMs, if you use one that is not specified there you might run in to problems.
You do not have to be root, in fact, you SHOULD NOT be root !
Using bash -x
The scripts below use bash in debug mode (bash -x). This generates a lot of output which can according to the issue make debugging easier or harder. Bash on its own (not in debug mode) will work if you want less debug output.
Setting up the workspace
These instructions assume you want to set up a SQLite workspace. If you want to use MySQL then you must use the "bash -x testing/build-opendnssec-mysql.sh" command in the last step instead!!
Preparations
Clear any environment variables that might confuse the tests, e.g.
$ unset SOFTHSM_CONF
Setting up a workspace
Jenkins and the framework both depends on a top level directory called 'workspace' so we make that first.
$ mkdir ~/workspace $ cd ~/workspace
Checking out code bases
We need to clone OpenDNSSEC and softHSM to build all requirements before we can run tests.
$ git clone https://github.com/opendnssec/SoftHSMv1.git $ git clone https://github.com/opendnssec/OpenDNSSEC.git
Checkout the relevant code base.
cd ~/workspace/SoftHSMv1 git checkout develop cd ~/workspace/OpenDNSSEC git checkout 1.4/develop
Install tag
Decide on a install tag, this tag will be used as a directory under INSTALL_ROOT where everything will be installed into.
$ export INSTALL_TAG=local-test
Building requirements
LDNS
To build LDNS needed by OpenDNSSEC we change to the OpenDNSSEC testing directory, set the WORKSPACE to the current directory, fake a SVN_REVISION and start the build, -x is used so you can follow all commands that bash runs so if you don't want all that output you can skip that option.
$ cd ~/workspace/OpenDNSSEC/testing $ export WORKSPACE=`pwd` $ export SVN_REVISION=1 $ bash -x build-ldns.sh
Note that the build-ldns.sh script is no longer provided. Make sure you configure ldns with --enable-stderr-msgs. Else some of the tests will fail.
SoftHSM
To build SoftHSM needed by OpenDNSSEC we change to the SoftHSM directory, set the WORKSPACE to the current directory, fake a SVN_REVISION and start the build, -x is used so you can follow all commands that bash runs so if you don't want all that output you can skip that option.
$ cd ~/workspace/SoftHSMv1 $ export WORKSPACE=`pwd` $ export SVN_REVISION=1 $ bash -x testing/build-softhsm.sh
OpenDNSSEC
To build OpenDNSSEC needed for our tests we change to the OpenDNSSEC directory, set the WORKSPACE to the current directory, fake a SVN_REVISION and start the build, -x is used so you can follow all commands that bash runs so if you don't want all that output you can skip that option.
$ cd ~/workspace/OpenDNSSEC $ export WORKSPACE=`pwd` $ export SVN_REVISION=1 $ bash -x testing/build-opendnssec.sh
Copy&Paste Workspace setup commands
unset SOFTHSM_CONF mkdir ~/workspace cd ~/workspace git clone https://github.com/opendnssec/SoftHSMv1.git git clone https://github.com/opendnssec/OpenDNSSEC.git cd ~/workspace/SoftHSMv1 git checkout develop cd ~/workspace/OpenDNSSEC git checkout 1.4/develop export INSTALL_TAG=local-test cd ~/workspace/OpenDNSSEC/testing export WORKSPACE=`pwd` export SVN_REVISION=1 bash -x build-ldns.sh cd ~/workspace/SoftHSMv1 export WORKSPACE=`pwd` export SVN_REVISION=1 bash -x testing/build-softhsm.sh cd ~/workspace/OpenDNSSEC export WORKSPACE=`pwd` export SVN_REVISION=1 bash -x testing/build-opendnssec.sh
Running existing tests (optional)
All tests
You can run all the existing tests in a given test directory if you want:
cd ~/workspace/OpenDNSSEC/testing export INSTALL_TAG=local-test export WORKSPACE=`pwd` export SVN_REVISION=1 bash -x test-opendnssec.sh
You may want to re-direct the output as it is very long and verbose!
One test
If the script OpenDNSSEC/testing/test-this.sh exists in the branch you are working on then you can simple cd into a given test directory (e.g. OpenDNSSEC/testing/<test-dir>/<test-name>) and run
../../test-this.sh
Or (manual option) just run a single test and leave the log files in place:
cd ~/workspace/OpenDNSSEC/testing/<test-dir>/<test-name> export INSTALL_TAG=local-test export WORKSPACE=`pwd` export SVN_REVISION=1 bash -x source ../../lib.sh source ../../functions-opendnssec.sh init log_cleanup && syslog_cleanup ods_find_softhsm_module ods_pre_test syslog_trace source ./test.sh
To re-run the same test if you remain within the bash shell simply do the following:
syslog_stop ods_post_test start-and-stop "$test_status" log_cleanup && syslog_cleanup ods_find_softhsm_module ods_pre_test syslog_trace source ./test.sh
NOTE: If you have set up a MYSQL workspace then you also need to export the following variable before running the test:
export HAVE_MYSQL="YES"
Making a new test
This will describe step by step how to make the 00-start-and-stop test that exists today.
Quick overview
Tests are made up of a main test script, test-opendnssec.sh, thats sets up the environment, checks requirements and runs the tests. The run_tests() function is used on a rc/init style directory called test-cases.d. Each test case inside the directory is a directory itself containing a test.sh. Tests are executed in ascending order and all tests will be executed. If any test fails then run_tests() will return false and the test job will be marked failed.
Figuring out what to do
So we want to test OpenDNSSEC by just starting and stopping it so we can start with setting up the framework and manually starting and stopping OpenDNSSEC just to see if it works.
First we get our own bash instance and run it with -x to see what it does. Then load up the framework and initialize it.
$ cd ~/workspace/OpenDNSSEC/testing $ export WORKSPACE=`pwd` $ export SVN_REVISION=1 $ bash -x $ source ./lib.sh $ init
You can now check that ods-control exists in the right path:
$ which ods-control + which ods-control /home/jerry/workspace/root/local-test/sbin/ods-control
Lets try and start OpenDNSSEC:
$ ods-control start + ods-control start Starting enforcer... OpenDNSSEC ods-enforcerd started (version 1.4.0-trunk), pid 24402 Starting signer engine... Starting signer... DEBUG: timeshift mode enabled, but not set. OpenDNSSEC signer engine version 1.4.0-trunk Engine running.
Okey that worked, now lets stop it.
$ ods-control stop + ods-control stop Stopping enforcer... Stopping signer engine.. Engine shut down.
Good, so if we wanted this could be our test.sh that we place in test-cases.d/00-start-and-stop/ .
#!/usr/bin/env bash ods-control start && ods-control stop && return return 1
This would run ods-control 2 times, one to start and one to stop and if those program does not fail we would return success else it would return failure.
But maybe ods-control returns success even if it does not start the Signer or Enforcer so lets go ahead and change test.sh to log the output of ods-control and check it with the log_this() / log_grep() functions.
#!/usr/bin/env bash log_this ods-control-start ods-control start && log_this ods-control-stop ods-control stop && return return 1
So now we can run the test manually to see what it logs (stderr logs will contain bash -x output also but I've stripped that here):
$ cd ~/workspace/OpenDNSSEC/testing/test-cases.d/00-start-and-stop $ source ./test.sh $ cat _log*ods-control-start*stdout Starting enforcer... OpenDNSSEC ods-enforcerd started (version 1.4.0-trunk), pid 2319 Starting signer engine... Starting signer... DEBUG: timeshift mode enabled, but not set. OpenDNSSEC signer engine version 1.4.0-trunk Engine running. DEBUG: timeshift mode enabled, but not set. OpenDNSSEC signer engine version 1.4.0-trunk $ cat _log*ods-control-start*stderr $ cat _log*ods-control-stop*stdout Stopping enforcer... Stopping signer engine.. Engine shut down. $ cat _log*ods-control-stop*stderr $ log_cleanup
NOTE 2012-02-09: It might be a bug that the ods-signerd outputs timeshift and version on commands sent, will be looked at.
Now we can add some log_grep() to our test.sh to check for some of the relevant strings:
#!/usr/bin/env bash log_this ods-control-start ods-control start && log_grep ods-control-start stdout "OpenDNSSEC ods-enforcerd started" && log_grep ods-control-start stdout "Engine running" && log_this ods-control-stop ods-control stop && # log_grep ods-control-stop stdout 'Engine shut down' && # This did not work on all platforms, thus have been removed return return 1
So now we check that the Enforcer and Signer have started but there can still be errors after starting elsewhere like configuration errors.
Both Enforcer and Signer output a lot to the syslog and we can use that information to make more checks that both program started okay and ended okay.
Lets start logging from syslog and running the test again manually:
$ cd ~/workspace/OpenDNSSEC/testing/test-cases.d/00-start-and-stop $ syslog_trace $ source ./test.sh $ syslog_stop $ cat _syslog* Feb 9 11:17:04 ubuntu ods-enforcerd: opendnssec starting... Feb 9 11:17:04 ubuntu ods-enforcerd: opendnssec Parent exiting... Feb 9 11:17:04 ubuntu ods-enforcerd: opendnssec forked OK... Feb 9 11:17:04 ubuntu ods-enforcerd: opendnssec started (version 1.4.0-trunk), pid 2591 Feb 9 11:17:04 ubuntu ods-enforcerd: HSM opened successfully. Feb 9 11:17:04 ubuntu ods-enforcerd: Reading config "/home/jerry/workspace/root/local-test/etc/opendnssec/conf.xml" Feb 9 11:17:04 ubuntu ods-enforcerd: Reading config schema "/home/jerry/workspace/root/local-test/share/opendnssec/conf.rng" Feb 9 11:17:04 ubuntu ods-enforcerd: Communication Interval: 3600 Feb 9 11:17:04 ubuntu ods-enforcerd: No DS Submit command supplied Feb 9 11:17:04 ubuntu ods-enforcerd: SQLite database set to: /home/jerry/workspace/root/local-test/var/opendnssec/kasp.db Feb 9 11:17:04 ubuntu ods-enforcerd: Log User set to: local0 Feb 9 11:17:04 ubuntu ods-enforcerd: Switched log facility to: local0 Feb 9 11:17:04 ubuntu ods-enforcerd: Connecting to Database... Feb 9 11:17:04 ubuntu ods-enforcerd: Policy default found. Feb 9 11:17:04 ubuntu ods-enforcerd: Key sharing is Off. Feb 9 11:17:04 ubuntu ods-enforcerd: No zones on policy default, skipping... Feb 9 11:17:04 ubuntu ods-enforcerd: Purging keys... Feb 9 11:17:04 ubuntu ods-enforcerd: zonelist filename set to /home/jerry/workspace/root/local-test/etc/opendnssec/zonelist.xml. Feb 9 11:17:04 ubuntu ods-enforcerd: Disconnecting from Database... Feb 9 11:17:04 ubuntu ods-enforcerd: Sleeping for 3600 seconds. Feb 9 11:17:05 ubuntu ods-signerd: [hsm] libhsm connection opened succesfully Feb 9 11:17:05 ubuntu ods-signerd: [engine] signer started Feb 9 11:17:06 ubuntu ods-enforcerd: Received SIGTERM, exiting... Feb 9 11:17:06 ubuntu ods-enforcerd: all done! hsm_close result: 0 Feb 9 11:17:07 ubuntu ods-signerd: [engine] signer shutdown $ log_cleanup $ syslog_cleanup
NOTE: You do not have to start logging syslog inside the test, this is automatically done by the framework before each test.
There are plenty of things you can check here but we keep it simple for this exercise, to the bat-test.sh-mobile:
#!/usr/bin/env bash log_this ods-control-start ods-control start && log_grep ods-control-start stdout 'OpenDNSSEC ods-enforcerd started' && log_grep ods-control-start stdout 'Engine running' && syslog_waitfor 60 'ods-signerd: .*\[engine\] signer started' && syslog_grep 'ods-enforcerd: .*Sleeping for' && log_this ods-control-stop ods-control stop && syslog_waitfor 60 'ods-signerd: .*\[engine\] signer shutdown' && syslog_grep 'ods-enforcerd: .*all done' && return return 1
NOTE: .* after process name in the syslog_grep is for Solaris compatibility.
Project specific functions
Beside the framework, projects might have project specific functions that can be used. Please see testing page for each project for more details.
OpenDNSSEC comes with a few of these functions and one is to reset the environment, ods_reset_env(), that should be used in every test so we add it:
#!/usr/bin/env bash ods_reset_env && log_this ods-control-start ods-control start && log_grep ods-control-start stdout 'OpenDNSSEC ods-enforcerd started' && log_grep ods-control-start stdout 'Engine running' && syslog_waitfor 60 'ods-signerd: .*\[engine\] signer started' && syslog_grep 'ods-enforcerd: .*Sleeping for' && log_this ods-control-stop ods-control stop && syslog_waitfor 60 'ods-signerd: .*\[engine\] signer shutdown' && syslog_grep 'ods-enforcerd: .*all done' && return ods-control stop return 1
There are also pre and post hooks that the OpenDNSSEC project uses so if we are going to run this test manually we do this:
$ cd ~/workspace/OpenDNSSEC/testing/test-cases.d/00-start-and-stop $ source ../../functions-opendnssec.sh $ ods_pre_test start-and-stop $ syslog_trace $ source ./test.sh $ test_status="$?" $ syslog_stop $ ods_post_test start-and-stop "$test_status" $ log_cleanup $ syslog_cleanup
Or we can use the run_test() function:
$ cd ~/workspace/OpenDNSSEC/testing $ source functions-opendnssec.sh $ export PRE_TEST=ods_pre_test $ export POST_TEST=ods_post_test $ run_test start-and-stop test-cases.d/00-start-and-stop
The finale
Now you should have a workable test, lets try it one last time before committing:
$ cd ~/workspace/OpenDNSSEC/testing $ export WORKSPACE=`pwd` $ export SVN_REVISION=2 $ bash -x test-opendnssec.sh
If everything works, commit and wait for Jenkins to break! 
Overview
Content Tools
