Repository files navigation

Overview •Quick start •Debug •Containers •DCAT-AP info

Requirements:

• Docker

ckan-mqa offers a Docker Compose solution for performingMetadata Quality Assessment (MQA) on both CKAN endpoints and the European Data Portal catalogs. MQA is a crucial process to ensure the accuracy, completeness, and reliability of metadata, enhancing the overall data interoperability and accessibility.

This Docker Compose configuration enhances a Python MQA software¹ to integrates the powerful MQA toolset seamlessly with CKAN endpoints and European Data Portal catalogs, enabling users to perform in-depth assessments of metadata quality effortlessly. The setup provides an efficient way to run comprehensive quality checks on various metadata attributes, including data relevance, schema compliance, data format consistency, and adherence to standard vocabularies.

The MQA measures the quality of various indicators, each indicator is explained in the tables below. The results of the checks are stored as Data Quality Vocabulary (DQV) . DQV is a specification of the W3C that is used to describe the quality of a dataset.

The dimensions are derived from the FAIR principles:

• FindabilityThe following table describes the metrics that help people and machines in finding datasets. A maximum of 100 points can be scored in this area.

• AccessibilityThe following table describes which metrics are used to determine whether access to the data referenced by the distributions is guaranteed. A maximum of 100 points can be scored in this area.

• InteroperabilityThe following table describes the metrics used to determine whether a distribution is considered interoperable. According to the assumption 'identical content with several distributions', only the distribution with the highest number of points is used to calculate the points. A maximum of 110 points can be scored in this area

• ReusabilityThe following table describes which metrics are used to check the reusability of the data. A maximum of 75 points can be scored in this area.

• ContextualityThe following table show some light weight properties, that provide more context to the user. A maximum of 20 points can be scored in this area.

The final rating happens via four rating groups. The mapping of the points to the rating category is shown in the table below. The representation of the rating in the MQA is expressed exclusively via the rating categories. This enables providers to achieve the highest rating even with a slight deduction of points.

First copy the.env.example template as.env and configure by changing theCKAN_CATALOG_URL, as well as the DCAT-AP Profile version (DCATAP_FILES_VERSION), if needed.

cp .env.example .env

Custom ennvars:

• CKAN_CATALOG_URL: URL of the CKAN catalog to be downloaded (i.e.http://localhost:5000/catalog.rdf?q=organization:test).
• APP_DIR: Path to the application folder in Docker.
• TZ: Timezone.
• DCATAP_FILES_VERSION: DCAT-AP version (Avalaibles: 2.0.1, 2.1.0, 2.1.1).
• UPDATE_VOCABS: Update vocabs from the EU Publications Office at start (True orFalse).
• CKAN_METADATA_TYPE: CKAN Metadata elements type:ckan_uris for GeoDCAT-AP schema with all elements described by URIs (e.g.dct:format =http://publications.europa.eu/resource/authority/file-type/XML) orckan if used a CKAN default schema with label metadata elements (e.g.dct:format = "XML").

To deploy the environment,docker compose will build the latest image (ghcr.io/mjanez/ckan-mqa:latest).

git clone https://github.com/mjanez/ckan-mqacd ckan-mqadocker compose up --build# Or detached modedocker compose up -d --build

docker compose -f docker-compose.dev.yml up --build

 docker build -t target_name xxxx/

Dependencies:

python3 -m pip install --user pipxpipx install pdmpdm install --no-self

Run:

pdm run python ckan2mqa/ckan2mqa.py

1. Build and run container.
2. Attach Visual Studio Code to container
3. Start debugging onckan2mqa.py Python file (Debug the currently active Python file).

List ofcontainers:

The different cases to validate in theDCAT-AP Validator are based on the level of completeness of the checks and the incorporation of background knowledge (vocabularies). Each case is designed for a specific data exchange scenario.The following describes each case and recommends which one you should use for a CKAN catalog:

Includes all constraints required for technical coherence, excluding range class membership constraints and controlled vocabulary usage.

SHACL Profiles:

• 2.1.1
• 3.0.0

Includes all range class membership constraints.

SHACL Profiles:

• 2.1.1
• 3.0.0

Extends Case 1 with background knowledge, including all vocabularies used in DCAT-AP.

SHACL Profiles:

• 2.1.1:shapes andimports
• 3.0.0:shapes andimports

Extends Case 2 with background knowledge, adding validation of range class membership and vocabulary standards compliance.

SHACL Profiles:

• 2.1.1:range andimports
• 3.0.0:range andimports

Includes all constraints related to recommended properties.

SHACL Profiles:

• 2.1.1:shapes recommended andimports
• 3.0.0:shapes recommended andimports

Includes all constraints related to controlled vocabularies.

SHACL Profiles:

• 2.1.1:vocabularies shape andimports
• 3.0.0:vocabularies shape andmdr imports

The union of Cases 3, 4, 5, and 6.

SHACL Profiles:

• 2.1.1:shapes,shapes recommended,imports,range anddeprecateduris
• 3.0.0:shapes,shapes recommended,imports,range anddeprecateduris

For most use cases,Case 3: DCAT-AP Base (with background knowledge) is recommended. It provides comprehensive validation of basic coherence and vocabulary standards compliance.If your CKAN catalog uses controlled vocabularies, consider usingCase 6: DCAT-AP Controlled Vocabularies orCase 7: DCAT-AP Full (with background knowledge) for more exhaustive validation.Remember, the choice of the appropriate validation case depends on your specific needs and data exchange context.

Copyright (c) the respective contributors.It is open and licensed under the GNU Affero General Public License (AGPL) v3.0 whose full text may be found at:http://www.fsf.org/licensing/licenses/agpl-3.0.html