Repository files navigation

This repository contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their validators.

It is meant to be language agnostic and should require only a JSON parser.The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer.

The recommended workflow of this test suite is to clone themain branch of this repository as agit submodule orgit subtree. Themain branch is always stable.

All JSON Schema specification releases should be well covered by this suite, including versions draft-2020-12, draft-2019-09, draft-07, draft-06, draft-04 and draft-03.Versions draft-04 and draft-03 are considered "frozen" in that less effort is put in to backport new tests to these versions.

Additional coverage is always welcome, particularly for bugs encountered in real-world implementations.If you see anything missing or incorrect, please feel free tofile an issue orsubmit a PR.

@gregsdennis has also started a separatetest suite that is modelled after this suite to cover third-party vocabularies.

The tests in this suite are contained in thetests directory at the root of this repository.Inside that directory is a subdirectory for each released version of the specification.

The structure and contents of each file in these directories is described below.

In addition to the version-specific subdirectories, one additional directory is present:

1. latest/: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)

Inside each version directory there are a number of.json files each containing a collection of related tests.Often the grouping is by property under test, but not always.In addition to the.json files, each version directory contains one or more special subdirectories whose purpose isdescribed below, and which contain additional.json files.

Each.json file consists of a single JSON array of test cases.

For clarity, we first define this document's usage of some testing terminology:

An example illustrating this structure is immediately below, and a JSON Schema containing a formal definition of the contents of test cases can be foundalongside this README.

Here is a singletest case, containing one or more tests:

{"description":"The test case description","schema": {"type":"string" },"tests": [        {"description":"a test with a valid instance","data":"a string","valid":true        },        {"description":"a test with an invalid instance","data":15,"valid":false        }    ]}

There is currently only one additional subdirectory that may exist within each specification version test directory.

This is:

1. optional/: Contains tests that are considered optional.

Note, theoptional/ subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (inwhich case they are not truly optional in such a language).In the future this directory structure will be made richer to reflect these differences more clearly.

The test suite structure was describedabove.

If you are authoring a new validator implementation, or adding support for an additional version of the specification, this section describes:

1. How to implement a test runner which passes tests to your validator
2. Assumptions the suite makes about how the test runner will configure your validator
3. Invariants the test suite claims to hold for its tests

Presented here is a possible implementation of a test runner.The precise steps described do not need to be followed exactly, but the results of your own procedure should produce the same effects.

To test a specific version:

• For 2019-09 and later published versions, implementations that are able to detect the version of each schema via$schema SHOULD be configured to do so

• For draft-07 and earlier, v1 (not yet released), and implementations unable to detect via$schema, implementations MUST be configured to expect the version matching the test directory name

• Load any remote referencesdescribed below and configure your implementation to retrieve them via their URIs

• Walk the filesystem tree for that version's subdirectory and for each.json file found:

  • if the file is located in the root of the version directory:

    • for each test case present in the file:

      • load the schema from the"schema" property

      • load (or log) the test case description from the"description" property for debugging or outputting

      • for each test in the"tests" property:

        • load the instance to be tested from the"data" property

        • load (or log) the individual test description from the"description" property for debugging or outputting

        • use the schema loaded above to validate whether the instance is considered valid under your implementation

        • if the result from your implementation matches the value found in the"valid" property, your implementation correctly implements the specific example

        • if the result does not match, or your implementation errors or crashes, your implementation does not correctly implement the specific example

  • otherwise it is located in a special subdirectory as described above.Follow the additional assumptions and restrictions for the containing subdirectory, then run the test case as above.

If your implementation supports multiple versions, run the above procedure for each version supported, configuring your implementation as appropriate to call each version individually.

1. The suite, notably in itsrefRemote.json file in each specification version directory, expects a number of remote references to be configured.These are JSON documents, identified by URI, which are used by the suite to test the behavior of the$ref keyword (and related keywords).Depending on your implementation, you may configure how to "register" theseeither:

   • by directly retrieving them off the filesystem from theremotes/ directory, in which case you should load each schema with a retrieval URI ofhttp://localhost:1234 followed by the relative path from the remotes directory -- e.g. a$ref tohttp://localhost:1234/foo/bar/baz.json is expected to resolve to the contents of the file atremotes/foo/bar/baz.json

   • or alternatively, by executingbin/jsonschema_suite remotes using the executable in thebin/ directory, which will output a JSON object containing all of the remotes combined, e.g.:

   $  bin/jsonschema_suite remotes

   {"http://localhost:1234/baseUriChange/folderInteger.json": {"type":"integer"    },"http://localhost:1234/baseUriChangeFolder/folderInteger.json": {"type":"integer"    }}

2. Test cases found withinspecial subdirectories may require additional configuration to run.In particular, when running tests within theoptional/format subdirectory, test runners should configure implementations to enable format validation, where the implementation supports it.

The test suite guarantees a number of things about tests it defines.Any deviation from the below is generally considered a bug.If you suspect one, pleasefile an issue:

1. All files containing test cases are valid JSON.

2. The contents of the"schema" property in a test case are always validJSON Schemas under the corresponding specification.

   The rationale behind this is that we are testing instances in a test's"data" element, and not the schema itself.A number of testsdo test the validity of a schema itself, but do so by representing the schema as an instance inside a test, with the associated meta-schema in the"schema" property (via the"$ref" keyword):

   {"description":"Test the\"type\" schema keyword","schema": {"$ref":"https://json-schema.org/draft/2019-09/schema"     },"tests": [        {"description":"Valid: string","data": {"type":"string"            },"valid":true        },        {"description":"Invalid: null","data": {"type":null            },"valid":false        }    ]}

   See below for someknown limitations.

This suite expresses its assertions about the behavior of an implementationwithin JSON Schema itself.Each test is the application of a schema to a particular instance.This means that the suite of tests can test against any behavior a schema can describe, and conversely cannot test against any behavior which a schema is incapable of representing, even if the behavior is mandated by the specification.

For example, a schema can require that a string is aURI-reference and even that it matches a certain pattern, but even though the specification containsrecommendations about URIs being normalized, a JSON schema cannot today represent this assertion within the core vocabularies of the specifications, so no test covers this behavior.

This suite is being used by:

• jinx
• json-schema

• jsck

• json-schema

• Blaze
• Modern C++ JSON schema validator
• Valijson

• json_schema

• ex_json_schema

• jesse

• gojsonschema
• validate-json

• aeson-schema
• hjsonschema

• json-schema-validation-comparison (Comparison site for JVM-based validator implementations)
• json-schema-validator
• everit-org/json-schema
• networknt/json-schema-validator
• Justify
• Snow
• jsonschemafriend
• OpenAPI JSON Schema Generator

• json-schema-benchmark
• direct-schema
• is-my-json-valid
• jassi
• JaySchema
• json-schema-valid
• Jsonary
• jsonschema
• request-validator
• skeemas
• tv4
• z-schema
• jsen
• ajv
• djv

• json-schema-validation-comparison (Comparison site for JVM-based validator implementations)

For node.js developers, the suite is also available as annpm package.

Node-specific support is maintained in aseparate repository which also welcomes your contributions!

• JsonSchema.Net
• Newtonsoft.Json.Schema

• Test::JSON::Schema::Acceptance (a wrapper of this test suite)
• JSON::Schema::Modern
• JSON::Schema::Tiny

• opis/json-schema
• json-schema
• json-guard

• postgres-json-schema
• is_jsonb_valid

• jsonschema
• fastjsonschema
• hypothesis-jsonschema
• jschon
• OpenAPI JSON Schema Generator

• json-schema
• json_schemer

• jsonschema
• valico

• json-schema-validation-comparison (Comparison site for JVM-based validator implementations)
• typed-json

• JSONSchema
• swift-json-schema

If you use it as well, please fork and send a pull request adding yourself tothe list :).

If you see something missing or incorrect, a pull request is most welcome!

There are some sanity checks in place for testing the test suite. You can runthem withbin/jsonschema_suite check ortox. They will be run automaticallybyGitHub Actionsas well.

This repository is maintained by the JSON Schema organization, and will be governed by the JSON Schema steering committee (once it exists).