Repository files navigation

Thegemtest framework makes it easy to write metamorphic relations in Python, from where the framework derivesmultiple metamorphic test cases.Metamorphic test cases are then executed as apytest test suite.

An example of a simple metamorphic relation:

# content of test_sin_metamorphic.pyimportgemtestasgmtimportmathmr_1=gmt.create_metamorphic_relation(name='mr_1',data=range(100))@gmt.transformation(mr_1)defexample_transformation(source_input:float)->float:returnsource_input+2*math.pi@gmt.relation(mr_1)defexample_relation(source_output:float,followup_output:float)->bool:returngmt.relations.approximately(source_output,followup_output)@gmt.system_under_test(mr_1)deftest_example_sut(input:float)->float:returnmath.sin(input)

To execute it:

$pytest test_sin_metamorphic.py=============== test session starts ===============platform linux -- Python 3.10.12, pytest-8.3.4,pluggy-1.5.0rootdir: /home/user/gemtestplugins: typeguard-2.13.3, html-3.2.0,metadata-3.1.1, xdist-3.6.1, gemtest-1.0.0,cov-4.1.0, hypothesis-6.113.0collected 100 itemstest_sin_metamorphic.py ....................................................................................................=============== 100 passed in 0.28s ===============

• pytest --string-report <test-file path>: Enables custom string report output on console.
• pytest --html-report <test-file path>: Enables custom html report includingvisualization of in- and outputs if a visualization function is provided, additionallytest results are stored in an SQLite database and can be viewed with thegemtest-webapp.

A simple metamorphic relation consists of 4 parts:

1. The creation of the metamorphic relation. Every metamorphic relation requires a name and a data source fromwhich the metamorphic test cases are created.

<mr1_name>=gmt.create_metamorphic_relation(name='mr_1',data=range(100))

2. A function annotated with@transformation which takes a single source input and creates asingle followup input. A transformation can be registered to a metamorphic relation byspecifying the name of the metamorphic relation in the@transformation annotation. Atransformation is registered to all metamorphic relations of a test file if nometamorphic relation is explicitly specified in the@transformation annotation. Everymetamorphic relation can only have one registered transformation.

@gmt.transformation(<mr1_name,mr2_name, ...>)def<transformation_function_name>(source_input:Input)->Input:<applycustomtransformationtoInput>

3. A function annotated with@relation which takes a single source output and followupoutput and return a boolean value. Registering a relation to a metamorphic relation worksidentically to the registration of a transformation. Everymetamorphic relation can only have one registered relation.

@gmt.relation(<mr1_name,mr2_name, ...>)def<relation_function_name>(source_output:Output,followup_output:Output)->boolean:<applycustomrelationtoOutputs>

4. A function annotated by@system_under_test whose name must begin with test, take asingle input and return a single output. Registering a system under test to a metamorphicrelation works identical to the registration for a transformation.

@gmt.system_under_test(<mr1_name,mr2_name, ...>)deftest_<system_name>(input:Input)->Output:<applycustomsystemfunctionalitytoInput>

To usegemtest, one must first define one's metamorphic relationsusing thecreate_metamorphic_relation() function. This function takes in various arguments,such as the name of the relation, the data to be transformed, and the number of test casesto generate:

defcreate_metamorphic_relation(name:str,data:Sequence,testing_strategy:str=TestingStrategy.EXHAUSTIVE,number_of_test_cases:int=1,number_of_sources:int=1,parameters:Optional[Dict]=None,system_under_test:Optional[System]=None,transform:Optional[Transform]=None,general_transform:Optional[GeneralTransform]=None,relation:Optional[Relation]=None,general_relation:Optional[GeneralRelation]=None,valid_input:Optional[Input]=None)->MR_ID:

Parameters

• name: Name of the metamorphic relation.
• data: A sequence of input data that is used to generate metamorphic test cases.
• testing_strategy: Specifies the testing strategy to use for generatingmetamorphic test cases. Can take the values TestingStrategy.SAMPLE or TestingStrategy.EXHAUSTIVE. Default value is TestingStrategy.EXHAUSTIVE.
• number_of_test_cases: An integer that specifies the number of metamorphic test cases togenerate. Default value is 1.
• number_of_sources: An integer that specifies the number of input sources to use forgenerating metamorphic test cases. Default value is 1.
• parameters: Optional dictionary of test parameters. Can be used to define multiple similar tests with different parameters.
• system_under_test: The system under test whose functionality is to be verified. Defaults to None.
• transform: Optional transformation function to apply to the input data.
• general_transform: An optional callable that represents the general transformation function to apply to the input data.
• relation: Optional relation function evaluating the metamorphic test case.
• general_relation: An optional callable that represents the general relation function evaluating the metamorphic test case.
• valid_input: A list of functions returning a bool that are used to validate the input to the system under test. The metamorphic test case is skipped if function returns false

Functions for the propertiessystem_under_test,transform,general_transform,relation,general_relation, andvalid_input can be added to a metamorphic relation with annotationsafter it is created, as seen in the example above. Thegemtest framework also contains predefinedfunctions that can be added to a metamorphic relation during creation.

Next to the simple functionality provided by@transformation and@relation decorated functions,gemtest also supports a more general approach for defining metamorphic relations using the@general_transformation and@general_relation decorators.

importgemtestasgmtimportmathmr_2=gmt.create_metamorphic_relation(name='mr_2',data=range(10),testing_strategy=gmt.TestingStrategy.SAMPLE,number_of_test_cases=10,number_of_sources=2)@gmt.general_transformation(mr_2)defshift(mtc:gmt.MetamorphicTestCase):followup_input_1=mtc.source_inputs[0]+2*math.pifollowup_input_2=mtc.source_inputs[1]-2*math.pireturnfollowup_input_1,followup_input_2@gmt.general_relation(mr_2)defapproximately_equals(mtc:gmt.MetamorphicTestCase)->bool:return (gmt.approximately(mtc.source_outputs[0],mtc.followup_outputs[0])andgmt.approximately(mtc.source_outputs[1],mtc.followup_outputs[1]))@gmt.system_under_test(mr_2)deftest_dummy_sut(input:float)->float:returnmath.sin(input)

A general metamorphic relation consists of 4 parts:

1. The creation of the metamorphic relation. Every metamorphic relation requires a name and a data sourcefrom which the metamorphic test cases are created.

<mr1_name>=gmt.create_metamorphic_relation(name='mr_1',data=range(100))

2. A function annotated with@general_transformation must take aMetamorphicTestCase objectand return a single or multiple followup inputs as a tuple. Registering ageneral_transformation to a metamorphic relation works identically to the registration of atransformation. Every metamorphic relation can only have one registeredgeneral_transformation or transformation. A metamorphic relation may have a registeredgeneral_transformation and a registered relation if the functionality of ageneral_relation is not required.

@gmt.general_transformation(<mr1_name,mr2_name, ...>)def<transformation_function_name>(mtc:MetamorphicTestCase)->Input:<accesssinglesource_input>source_input:Input=mtc.source_input<accessmultiplesource_inputs>source_inputs:List[Input]=mtc.source_inputs<applycustomtransformationtoInput>returnfollowup_input_1,followup_input_2, ... ,followup_input_n

3. A function annotated with@general_relation must take aMetamorphicTestCase object andreturn a boolean value. Registering a general_relation to a metamorphic relation worksidentically to the registration of a relation. Every metamorphic relation can only haveone registered general_relation or relation.

@gmt.general_relation(<mr1_name,mr2_name, ...>)def<relation_function_name>(mtc:MetamorphicTestCase)->boolean:<applycustomrelationtoattributesofMetamorphicTestCase>

4. A function annotated with@system_under_test whose name must begin with test, take asingle input and return a single output.

Using theMetamorphicTestCase object allows general transformations to have anynumber of sources and create any number of followups. There is also the possibility to usesource inputs and source outputs to create followup inputs. A general relation can also usemultiple sources and followups as and additionally consider source and followup inputs andoutputs when evaluating if the relation holds for aMetamorphicTestCase.

TheMetamorphicTestCase class holds one concrete instance of a metamorphic test case for ametamorphic relation. The testing strategy is used to createMetamorphicTestCase objects from theprovided data object.Pytest tests are executed on instances of aMetamorphicTestCase. If allpytest tests for theMetamorphicTestCase objects of a metamorphic relation pass, the relation holdsfor the provided data.

Properties of classMetamorphicTestCase

• source_inputs: list of the source inputs for the Metamorphic Test Case
• source_input: convenience property to access the single source input if there is only one
• followup_inputs: list of the followup inputs for the Metamorphic Test Case
• followup_input: convenience property to access the single followup input if there is only one
• source_outputs: list of the source outputs for the Metamorphic Test Case
• source_output: convenience property to access the single source output if there is only one
• followup_outputs: list of the followup outputs for the Metamorphic Test Case
• followup_output: convenience property to access the single followup output if there is only one
• parameters: dictionary containing previously specified parameters

If you find thegemtest framework useful in your research or projects, please consider citing it:

@inproceedings{speth2025,    author = {Speth, Simon and Pretschner, Alexander},    title = {{GeMTest: A General Metamorphic Testing Framework}},    booktitle = "Proceedings of the 47th International Conference on Software Engineering, (ICSE-Companion)",    pages = {1--4},    address = {Ottawa, ON, Canada},    year = {2025},}

MIT License