Conformance Testing
The purpose of conformance testing is to determine to what extent a singleimplementation of a particular standard conforms to the individual requirementsof that standard.
The important factors which characterize conformance testing are as follows:
- the System or Implementation Under Test (SUT or IUT) defines the boundaries (open interfaces) for testing;
- the conformance test system is a specialized tool (system) built for the purpose of testing and on which specific test scripts can be run;
- the SUT comes from a single supplier (or, at least, a single product line);
- the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all communications from the SUT;
- the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user (i.e. they are specified at the protocol level);
- the tests are specified at the detailed protocol level and are not usually based on functionality as experienced by a user.
Because the conformance tester maintains a high degree of control over thesequence and contents of the protocol messages sent to the IUT it is able toexplore a wide range of both expected and unexpected (invalid) behaviour.
Development of ETSI Conformance Test Specifications
Most ETSI test specifications are developed according to the well-provenmethodology defined in ISO/IEC 9646. This standard, which covers the entire testdevelopment process, provides an excellent basis for the production ofhigh-quality test frameworks and specifications.
The following diagram shows the development of test specifications process,acronyms are explained below.
Building a coherent set of test specifications from disperse requirementssources can be simplified by gathering the requirements together into a singlecatalogue. A Requirements Catalogue should list all implementation requirementsfrom the various sources and organize them into an appropriate structure.
The Implementation Conformance Statement (ICS) is a checklist of thecapabilities defined in the standard. When filled in by the user, the ICS provides an overview of the features and options that are implemented by anygiven IUT. The ICS can be used to select and parameterize test cases and as anindicator for basic interoperability between different products.
The Implementation eXtra Information forTesting (IXIT) contains additionalinformation (specific addresses, timer values etc.) necessary for testing.
In the specific case of protocol standards the ICS is known as the ProtocolICS or PICS and the IXIT is known as the Protocol IXIT or PIXIT.
Test Purposes (TP) provide an informal, easy-to-read self-containeddescription of each test, concentrating on what is to be tested rather than howthe actual test may be achieved. They are derived from the relevant basestandards or from the appropriate Requirements Catalogue. The set of all TestPurposes is grouped into a logical Test Suite Structure (TSS) according tosuitable criteria (basic interconnection, error handling, functionality etc.).The standardized notationTPLAN is a good way to write consistent Test Purposes.
The Abstract Test Suite (ATS) is the entire collection of Test Cases,sometimes known as test scripts. Each Test Case specifies the detailed coding ofthe Test Purposes, written in the standardized test specification languageTTCN-3.The actual Executable Test Suite (ETS) can be quickly and easily implementedfrom the ATS using the TTCN compilers available on many test tool platforms.
Finding ETSI Test Specifications
ETSI test specifications can be downloaded via theETSI Work Programme application.
Keywords can be added to facilitate your search, the following are related to testing:
- ATS (Abstract Test Suite)
- TTCN PICS (Protocol Implementation Conformance Statement)
- ICS (Implementation Conformance Statement)
- TSS&TP (Test suite structure and Test purposes)
- Testing
- Conformance
To select more than one, hold the Ctrl button and click on the keywordsrequired. When more than one keyword is selected, you can combine them eitherwith the OR function ("one of") or the AND function ("all").
The following is a list of recently published and frequently downloadedstandards. Please use theETSI WorkProgramme to find further related standards.