Repository files navigation

This CKAN extension provides a way to configure and share metadata schemas using aYAML or JSON schema description. Custom validation and template snippets for editingand display are supported.

Table of contents:

1. Requirements
2. Installation
3. Configuration
   • Schema Types
   • Example Schemas
   • Storing non-string data
   • Common Schema Keys
     • scheming_version
     • about_url
   • Dataset Schema Keys
     • dataset_type
     • dataset_fields,resource_fields
     • before_validators,after_validators
   • Group / Organization Schema Keys
     • group_type
     • organization_type
     • fields
     • before_validators,after_validators
   • Field Keys
     • field_name
     • label
     • repeating_subfields
     • start_form_page
     • required
     • choices
     • choices_helper
     • default
     • default_jinja2
     • preset
     • form_snippet
     • display_snippet
     • display_property
     • validators
     • output_validators
     • create_validators
     • help_text
4. Action API Endpoints
5. Running the Tests

This plugin is compatible with CKAN 2.9 or later.

You can install the extension with the following shell commands:

cd$CKAN_VENV/src/pip install -e"git+https://github.com/ckan/ckanext-scheming.git#egg=ckanext-scheming"

Set the schemas you want to use with configuration options:

# Each of the plugins is optional depending on your useckan.plugins = scheming_datasets scheming_groups scheming_organizations#   module-path:file to schemas being usedscheming.dataset_schemas = ckanext.spatialx:spatialx_schema.yaml                           ckanext.spatialx:spatialxy_schema.yaml#   will try to load "spatialx_schema.yaml" and "spatialxy_schema.yaml"#   as dataset schemas#   For group and organization schemas (replace myplugin with your custom plugin)scheming.group_schemas = ckanext.scheming:group_with_bookface.json                         ckanext.myplugin:/etc/ckan/default/group_with_custom_fields.jsonscheming.organization_schemas = ckanext.scheming:org_with_dept_id.json                                ckanext.myplugin:org_with_custom_fields.json##   URLs may also be used, e.g:## scheming.dataset_schemas = http://example.com/spatialx_schema.yaml#   Preset files may be included as well. The default preset setting is:scheming.presets = ckanext.scheming:presets.json#   The is_fallback setting may be changed as well. Defaults to false:scheming.dataset_fallback = false

With this plugin, you can customize the group, organization, and dataset entities in CKAN. Adding and enabling a schema will modify the forms used to update and create each entity, indicated by the respectivetype property at the root level. Such asgroup_type,organization_type, anddataset_type. Non-default types are supported properly as is indicated throughout the examples.

Dataset schemas:

• default dataset schema
• camel photos schema
• subfields schema
• form pages schema

These schemas are included in ckanext-scheming and may be enabledwith e.g:scheming.dataset_schemas = ckanext.scheming:camel_photos.yaml

These schemas usepresets defined inpresets.json.

Group schemas:

• Default group schema with field modifications
• Group with custom type

Organization schemas:

• Default organization schema with field modifications
• Organization with custom type

Set to2. Future versions of ckanext-scheming may use a largernumber to indicate a change to the schema format.

about_url:https://github.com/link-to-my-project

about_url is a link to human-readable information about this schema.ckanext-schemingautomatically publishes yourschema and this link allows users to learn more about it.

dataset_type:camel-photos

This is the "type" field stored in the dataset.It is also used to set the URL for searching this type of dataset.

Normal datasets would be available under the URL/dataset, but datasets withthecamel_photos.json schema above would appear under/camel-photos instead.

dataset_fields:-field_name:titlelabel:Titlepreset:title-field_name:namelabel:URLpreset:dataset_slug...

Fields are specified in the order youwould like them to appear in the dataset and resource editingpages.

Fields you exclude will not be shown to the end user, and will notbe accepted when editing or updating this type of dataset.

before_validators:validator_nameafter_validators:validator_name

Runs validator functions before and after thedataset_type package is created/updated.

group_type:group

is used for modifying the default group schema

group_type:theme

is an example of defining a custom group type, as seen in theexample schemas above

Likedataset_type, agroup_type ofgroup allows you to customize the default group schema under the URL/group, such as the modified schema ingroup_with_bookface.json, but a schema with a custom type such ascustom_group_with_status.json schema above would appear under/theme instead, because itsgroup_type field is set to "theme".

organization_type:organization

is used for modifying the default organization schema

organization_type:publisher

is an example of defining a custom organization type, as seen in theexample schemas above

fields:-field_name:titlelabel:Nameform_snippet:large_text.htmlform_attrs:data_module:slug-preview-targetform_placeholder:My Organization...

A singlefields list replaces thedataset_fields andresource_fields schema properties doin dataset schemas.

before_validators:validator_nameafter_validators:validator_name

Runs validator functions before and after theorganization_type/group_type group is created/updated.

Thefield_name value is the name of an existing CKAN dataset or resourcefield or a new extra field. Existing datasetfield names include:

• name - the URI for the dataset
• title
• notes - the dataset description
• author
• author_email
• maintainer
• maintainer_email

New field names should follow the current lowercase_with_underscoresnaming convention. Don't name your fieldmySpecialField, usemy_special_field instead.

Thelabel value is a human-readable label for this field asit will appear in the dataset editing form.This label may be a string or an object providing multiplelanguage versions:

-field_name:titlelabel:en:Titlefr:Titre

When using a plain string translations will be provided with gettext:

-field_name:titlelabel:Title

This field is the parent of group of repeating subfields. The value isa list of fields entered the same way as normal fields.

NOTE: CKAN needs an IPackageController plugin withbefore_index toconvert repeating subfields to formats that can be indexed by solr. Fortesting you may use the includedscheming_nerf_index plugin to encodeall repeating fields as JSON strings to prevent solr errors.

repeating_label may be used to provide a singular version of the labelfor each group.

-field_name:contactslabel:Contactsrepeating_label:Contactrepeating_subfields:  -field_name:addresslabel:Addressrequired:true  -field_name:citylabel:City  -field_name:phonelabel:Phone Number

Dataset fields may be divided into separate form pages for creationand editing.CKAN 2.9+ only. Form pages fordataset typeonly supported byCKAN 2.10+ or withckan/ckan#7032. Addingstart_form_page to a field marks this field as the start of anew page of fields.

-start_form_page:title:Detailed Metadatadescription:These fields improve search and give users important linksfield_name:addresslabel:Address

A title and description should be provided to help with navigation.These values may be strings or objects providing multiplelanguage versions of text.

required:true

Use for fields that must be included. Set tofalse ordon't include this key for fields that are optional.

Setting totrue will mark the field as required in the editing formand includenot_empty in the default validators that will be appliedwhenvalidators is not specified.

NOTE: To honor this settings with custom validators includescheming_requiredas the first validator.scheming_required will check the requiredsetting for this field and apply either thenot_empty orignore_missingvalidator.

Thechoices list may be provided forselect and multiple choice fields.List elements includelabels for human-readable text foreach element (may be multiple languages like afield label)andvalues that will be stored in the dataset or resource:

-field_name:categorypreset:selectchoices:  -value:bactrianlabel:Bactrian Camel  -value:hybridlabel:Hybrid Camel

For storing non-string values seeoutput_validators.

For requiredselect fields you may also want to add this settingso that users are forced to choose an item in the form, otherwise the firstchoice will be selected in the form by default:

form_include_blank_choice:true

To set the number of choices displayed in themultiple_selectform snippets use:

select_size:5

To sort choices alphabetically informanddisplay snippets use:

sorted_choices:true

If a choices list is not provided you must provide achoices_helperfunction that will return a list of choices in the same format asthechoices list above.

You mayregister your own helper function or use thescheming_datastore_choices helper included in ckanext-scheming:

-field_name:countrypreset:selectchoices_helper:scheming_datastore_choicesdatastore_choices_resource:countries-resource-id-or-aliasdatastore_choices_columns:value:Country Codelabel:English Country Namedatastore_additional_choices:  -value:nonelabel:None  -value:nalabel:N/A

Pre-fill new forms with thisdefault value for this field.

Pre-fill new forms with a jinja2 snippet defined bydefault_jinja2, usefulif you need to call a template helper function or use jinja2 logic to determinethe default value for this field.

Apreset specifies a set of default values for other field keys. Theyallow reuse of definitions for validation and snippets for common field types.

This extension includes the following presets inpresets.json:

preset:title

title validation and large text form snippet

preset:select

validation that choice is fromchoices, form select box and display snippet

preset:radio

validation that choice is fromchoices, form radio buttons group and display snippet

preset:multiple_checkbox

multiple choice fromchoices rendered as checkboxes in the form, stored as a list of values

preset:multiple_select

multiple choice fromchoices rendered as a multiple select box in the form, stored as a list of values

preset:multiple_text

repeating text field with add and remove buttons, stored as a list of strings

preset:date

date validation and form snippet

preset:datetime

date and time validation and form snippet

preset:dataset_slug

dataset slug validation and form snippet that autofills the value from the title field

preset:tag_string_autocomplete

tag string validation and form autocomplete

preset:dataset_organization

organization validation and form select box

preset:resource_url_upload

resource url validaton and link/upload form field

preset:resource_format_autocomplete

resource format validation and form autocomplete

preset:organization_url_upload

organization url validaton and link/upload form fieldformat guessing based on url and autocompleting form field

preset:json_object

JSON based input. Only JSON objects are supported. The input JSON will be loaded during output (eg when loading the dataset in a template or via the API

preset:markdown

markdown text area and display

You may define your own presets by adding additional files to thescheming.presetsconfiguration setting.

Theform_snippet value is the name of the snippet template touse for this field in the dataset or resource editing form.A number of snippets are provided with thisextension, but you may also provide your own by creating templatesunderscheming/form_snippets/ in a template directory in yourown extension.

This snippet is passed thefield dict containing all the keys andvalues in this field record, including any additional onesyou added to your that aren't handled by this extension.

The included form snippets may be found undertemplates/scheming/form_snippets.

Thedisplay_snippet value is the name of the snippet template touse for this field in the dataset, resource, group or organization view page.A number of snippets are provided with thisextension, but you may also provide your own by creating templatesunderscheming/display_snippets/ in a template directory in yourown extension.

This snippet is passed thefield dict containing all the keys andvalues in this field record, including any additional onesyou added to your that aren't handled by this extension.

The included display snippets may be found undertemplates/scheming/display_snippets.

Ifdisplay_snippet: null is used the field will be removed from the view page.

-field_name:authorlabel:Authordisplay_property:dc:creator

Set aproperty attribute on dataset fields displayed as "Additional Info", useful for adding RDF markup.

Thevalidators value is a space-separated string of validator and converterfunctions to use for this field when creating or updating data. When avalidator name is followed by parenthesis the function is called passing thecomma-separated values within and the result is used as thevalidator/converter.

validators:if_empty_same_as(name) unicode_safe

is the same as a plugin using the validators:

[get_validator('if_empty_same_as')("name"),unicode_safe]

If parameters can be parsed as a valid python literals, they are passed withoriginal type. If not, all parameters passed as strings. In addition, spacecharacter is not allowed in argument position. Use its HEX code instead\\x20.

validators:xxx(hello,world)# xxx("hello", "world")validators:xxx(hello,1)# xxx("hello", "1")validators:xxx("hello",1,None)# xxx("hello", 1, None)validators:xxx("hello\\x20world")# xxx("hello world")

This string does not contain arbitrary python code to be executed,you may only use registered validator functions, optionally callingthem with static string values provided.

NOTE: ckanext-scheming automatically adds calls toconvert_to_extrasfor extra fields when required.

New validators and converters may be added using theIValidators plugin interface.

Validators that need access to other values in this schema (e.g.to test values against the choices list) may be decorated withthescheming.validation.scheming_validatorfunction. This decorator will make scheming pass this field dict to thevalidator and use its return value for validation of the field.

CKAN'svalidator functions referencelists available validators ready to be used.

Internally all extra fields are stored as strings. If you are attempting to save and restore other types of data you will need to use output validators.

For example if you use a simple "yes/no" question, you will need to let ckanext-scheming know that this data needs to be storedand retrieved as a boolean. This is acheieved usingvalidators andoutput_validators keys.

  - field_name: is_camel_friendly    label: Is this camel friendly?    required: true    preset: select    choices:      - value: false        label: "No"      - value: true        label: "Yes"    validators: scheming_required boolean_validator    output_validators: boolean_validator

Theoutput_validators value is likevalidators but used whenretrieving values from the database instead of when saving them.These validators may be used to transform the data before it issent to the user.

NOTE: ckanext-scheming automatically adds calls toconvert_from_extrasfor extra fields when required.

Thecreate_validators value if present overridesvalidators duringcreate only.

help_text:License definitions and additional information

If this key is supplied, its value will be shown after the field as help text.Help text may be provided in multiple languages likelabel fields.

Help text must be plain text, no markdown or HTML are allowed unless:

help_allow_html:true

Allow HTML inside the help text if set totrue. Default isfalse.

Adjust the position ofhelp_text with:

help_inline:true

Display help text inline (next to the field) if set totrue. Default isfalse (display help text under the field).

The extension adds action endpoints which expose any configured schemas via:https://github.com/ckan/ckanext-scheming/blob/master/ckanext/scheming/logic.py

Some examples:

Callinghttp://localhost:5000/api/3/action/scheming_dataset_schema_list

Returns:

{  help: "http://localhost:5005/api/3/action/help_show?name=scheming_dataset_schema_list",  success: true,  result: [    "dataset",    "camel-photos"  ]}

Callinghttp://localhost:5000/api/3/action/scheming_dataset_schema_show?type=dataset

Returns:

{  help: "http://localhost:5005/api/3/action/help_show?name=scheming_dataset_schema_show",  success: true,  result: {    scheming_version: 2,    dataset_type: "dataset",    about: "A reimplementation of the default CKAN dataset schema",    about_url: "http://github.com/ckan/ckanext-scheming",    dataset_fields: [...],    resource_fields: [...]  }}

The full list of API actions are available inckanext/scheming/logic.py

To run the tests:

pytest --ckan-ini=test.ini ckanext/scheming/tests