Validation of Test Specifications
A standardized conformance or interoperability test specification should bevalidated using similar methods to those for other standards. Such aspecification typically comprises:
- a set of Test Purposes;
- the definition of a Test Suite Structure and of the implementation conformance/interoperability statement and the extra information for testing (often defined textually);
- an Abstract Test Suite (ATS) which is most often specified in TTCN-3.
Validation of Test Purposes
The formalized design review method of validation can be quite effective invalidating the Test System Structure and the Test Purposes as they generally usea mixture of textual and graphical specification techniques. Such a review willeasily highlight gaps in the test coverage of the base requirements as well asany unnecessary testing of unspecified functions. It will also validate that thegeneral approach to testing, the test configurations and the individual testmethods implied by the Test Purposes are all valid.
Test Suites, however, cannot easily be validated by means of a design reviewas they are written in TTCN-3 or some other programming language. It is, ofcourse, possible to partially validate the structure and coverage of a testsuite by such a manual method but the correctness and effectiveness of a TTCN-3test suite can only be validated by execution of the programming code in eithera simulation or a controlled testbed.
Validation of Test Suites
In order to produce high quality executable test suites some form ofpractical validation must be done before they are published as ETSI standards.The following definitions of validation levels for standardized TTCN-3 testspecifications have been developed by the ETSI Centre for Testing andInteroperability (CTI). The intention here is to provide guidance to TechnicalBodies on:
- the selection of a validation level for a particular test suite to be developed;
- the completion of the validation aspects within the Terms of Reference (ToR) for a Specialist Task Force (STF) to develop the test suite;
- the construction of a validation report
There are three levels of validation defined for TTCN-3 test specifications,as follows:
Level 1:The test suite successfully analyses/compiles onone or more TTCN-3 test tools. Optionally, a design review ofthe test suite is performed.
Level 2:The tests are executed on one test platform againstat least one System Under Test (SUT). Where possible, tests willbe run to completion. Optionally, a back-to-back validation(mirror test cases) and/or a data-driven codec validation areperformed.
Level 3:The tests are executed on several test platformsagainst several SUTs. Tests will be run to completion and traceanalysis will be done.
Further details on the validation processes for the three levels are given inAnnex B ofEG 201 015.
Each validation level carries its own overlapping set of responsibilitiesthat need to be considered and these are identified in the table below.
| Rigorous Validation (Level 3) |
|---|
| Strong Validation (Level 2) | |
|---|
| Basic Validation (Level 1) | | |
|---|
| Provision of TTCN-3 analyser/compilers | | |
| Compilation of the test suite | | |
| Reporting of errors in the tools | | |
| Reporting of errors in the ATS | | |
| Updating of ATS | | |
| Compilation of validation report | | |
| | Setting-up and maintaining the test platform | |
| | Provision and installation of SUT(s) | |
| | Execution of the tests | |
| | Reporting of errors in the platform | |
| | Validation of test verdicts | |
| | | Analysis of output traces |