Links
OpenDNSSEC Developer Wiki
OpenDNSSEC Documentation
SoftHSM Developer Wiki
SoftHSM Documentation
Current location:
This page serves to capture some process related to adding and developing test cases
On this Page
Adding a new test case
- Make sure you have run the test case locally before submitting
- Please try to comment the test case as fully as possible
- Send an email to the developers list to get someone else to review the test case. It is easy to miss things (especially &&)!
- Please update the test coverage page when a new case is added
- If the test does not run reliably by the end of the day, please disable the test completely or partially (i.e for certain platforms) or test it via the sandbox (see below) so it does not affect the overnight build.
- Sometimes test cases fail intermittently - if so the test should be disabled, modified or moved to the sandbox until it does run reliably
SANDBOX:
- In each branch in jenkins there is a sandbox test job called "test-opendnssec-sandbox-<branch>" with a matching test script and test directory in the repository. The idea is that these jobs/directories can be used to try out tests that have been developed (locally on one machine) across all the platforms without having to have the job turned on. They are configured not to generate any emails, and run only over the sqlite backend (mysql jobs can be added if needed).
- To have a test included in the sandbox job, it needs to exist in one of the main test directories (test-cases.d or test-cases-daily.d) and it has to include a file called 'sandbox' in the test directory.
- Then either wait (the jobs poll every 10 mins for changes to 'sandbox' files) or log in to jenkins and run the test manually.
- Note that the presence (or lack of) of an 'off' file in the test directory is ignored by the sandbox test script - it runs all 'sandbox' jobs regardless. So the sandbox status and 'normal' on/off status of the test are completely orthogonal.
- Note the sandbox job isn't linked to the build job so if code changes are needed then the build job will need to be run via the timer or manually before the sandbox test job can then run properly
Which directory should it go in?
test-cases.d: For core functionality tests (enforcing, signing and command line tools) that cover default or typical configuration success cases. Try not to add tests that make the total test run time longer than 20 mins.
test-cases-daily.d: For everything else until the total test time gets too long...
test-cases-weekly.d: Doesn't exist yet but will be created if the daily tests get to long.
Naming test cases
The test case names are important in two ways. Firstly they should describe what the test does so this can be seen at a glance. Secondly they are used by JUnit to categorise the test and then generate all the detailed output of the individual tests.
The test cases should have names split into 3 parts separated by dots as follows:
application.area.test_short_name
The words in the test name should be separated by underscores and can have an optional number added at the end. The three parts are:
- top level category of something like enforcer/signer/general
- followed by a second level category for the sub area of the test
- and then either
- a short text description [try to use a common word at the start for related tests so they group together] or
- the issue number if it tests a specific JIRA issue (note the JIRA issue can also be tracked by adding a comment - see below)
So for example we might have tests called:
enforcer.keys.export_as_ds
enforcer.keys.export_as_dnskey_rr
signer.adaptors.bind_stress_test-01
general.repository.opendnssec_XXX
The correct format is important is as it is used to categories the tests in JUnit!
Category suggestions
| Top Level | Coverage |
|---|---|
| Enforcer | All things enforcer including:
|
| Signer | All things signer including:
|
| General | Everything else...
|
Overview
Content Tools
