Please check theerrata for any errors or issues reported since publication.
This document is also available in this non-normative format:ePub
The English version of this specification is the only normative version. Non-normativetranslations may also be available.
Copyright © 2015W3C® (MIT,ERCIM,Keio,Beihang).W3Cliability,trademark anddocument use rules apply.
Validation, conversion, display, and search of tabular data on the web requires additional metadata that describes how the data should be interpreted. This document defines a vocabulary for metadata that annotates tabular data. This can be used to provide metadata at various levels, from groups of tables and how they relate to each other down to individual cells within a table.
The metadata defined in this specification is used to provide annotations on anannotated table orgroup of tables, as defined in [tabular-data-model]. Annotated tables form the basis for all further processing, such as validating, converting, or displaying the tables.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of currentW3C publications and the latest revision of this technical report can be found in theW3C technical reports index at http://www.w3.org/TR/.
TheCSV on the Web Working Group waschartered to produce a recommendation "Access methods for CSV Metadata" as well as recommendations for "Metadata vocabulary for CSV data" and "Mapping mechanism to transforming CSV into various formats (e.g., RDF, JSON, or XML)". This document aims to primarily satisfy the "Metadata vocabulary for CSV data" recommendation.
This document was published by theCSV on the Web Working Group as a Recommendation. If you wish to make comments regarding this document, please send them topublic-csv-wg@w3.org (subscribe,archives). All comments are welcome.
Please see the Working Group'simplementation report.
This document has been reviewed byW3C Members, by software developers, and by otherW3C groups and interested parties, and is endorsed by the Director as aW3C Recommendation. It is a stable document and may be used as reference material or cited from another document.W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web.
This document was produced by a group operating under the5 February 2004W3C Patent Policy.W3C maintains apublic list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes containsEssential Claim(s) must disclose the information in accordance withsection 6 of theW3C Patent Policy.
This document is governed by the1 September 2015W3C Process Document.
Interpreting tabular data that is available on the web, particularly as CSV, usually requires additional metadata. As an example, say that the following CSV file were available athttp://example.org/tree-ops.csv
:
GID,On Street,Species,Trim Cycle,Inventory Date1,ADDISON AV,Celtis australis,Large Tree Routine Prune,10/18/20102,EMERSON ST,Liquidambar styraciflua,Large Tree Routine Prune,6/2/2010
A human consumer of this data might be able to figure out the meaning of the different columns, particularly if there were some additional human-readable documentation made available. Automated processors would have a much harder time; realistically they would be limited to displaying the information in a table. Making available machine-readable metadata helps with the interpretation of the tabular data. For example, say that the following metadata file were available athttp://example.org/tree-ops.csv-metadata.json
:
{"@context":["http://www.w3.org/ns/csvw",{"@language":"en"}],"url":"tree-ops.csv","dc:title":"Tree Operations","dcat:keyword":["tree","street","maintenance"],"dc:publisher":{"schema:name":"Example Municipality","schema:url":{"@id":"http://example.org"}},"dc:license":{"@id":"http://opendefinition.org/licenses/cc-by/"},"dc:modified":{"@value":"2010-12-31","@type":"xsd:date"},"tableSchema":{"columns":[{"name":"GID","titles":["GID","Generic Identifier"],"dc:description":"An identifier for the operation on a tree.","datatype":"string","required":true},{"name":"on_street","titles":"On Street","dc:description":"The street that the tree is on.","datatype":"string"},{"name":"species","titles":"Species","dc:description":"The species of the tree.","datatype":"string"},{"name":"trim_cycle","titles":"Trim Cycle","dc:description":"The operation performed on the tree.","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","dc:description":"The date of the operation that was performed.","datatype":{"base":"date","format":"M/d/yyyy"}}],"primaryKey":"GID","aboutUrl":"#gid-{GID}"}}
This metadata file may be referenced from aLink
header when the CSV file is retrieved, or through looking in known locations for metadata (as described in [tabular-data-model]). It provides information for different types of applications:
GID
column are all present and unique.Implementations may fulfil one or more of these functions. In particular,Converters may or may not act as aValidator (perhaps through the setting of a flag), and check the data that they are converting to ensure that it is compliant with the schema. If a Converter does not also act as aValidator it may produce invalid output.
[tabular-data-model] defines anannotated tabular data model in which groups of tables, individual tables, columns, rows, and cells can be annotated with annotations. That specification also describeshow to locate metadata about a given tabular data file.
This document defines the format and structure of metadata documents, and how these are interpreted to create anAnnotated Tabular Data Model. It also defines how to validate tabular data based on some of these annotations.
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.
The key wordsMAY,MUST,MUST NOT,SHOULD, andSHOULD NOT are to be interpreted as described in [RFC2119].
The metadata format is based on a dialect of [JSON-LD] as defined insectionA.JSON-LD Dialect. This metadata can therefore be expressed as an RDF graph. It is not necessary for conformant applications to be able to process all JSON-LD, only the dialect defined in this specification. All applications that conform to this specification (includingvalidators and applications that read or convert tabular data)MUST read the JSON-based format described in this document.
Tabular dataMUST conform to the description from [tabular-data-model]. In particular note that eachrowMUST contain the same number ofcells (although some of thesecells may be empty). Parsers might not be able to map all CSV-encoded data to such a table. As such, the metadata format described in this specification cannot be applied to all CSV files.
This specification makes use of thecompact IRI Syntax; please refer to theCompact IRIs from [JSON-LD].
This specification makes use of the following namespaces:
csvw
:http://www.w3.org/ns/csvw#
dc
:http://purl.org/dc/terms/
dcat
:http://www.w3.org/ns/dcat#
foaf
:http://xmlns.com/foaf/0.1/
rdf
:http://www.w3.org/1999/02/22-rdf-syntax-ns#
schema
:http://schema.org/
xsd
:http://www.w3.org/2001/XMLSchema#
The following typographic conventions are used in this specification:
markup
markup definition reference
markup external definition reference
Notes are in light green boxes with a green left border and with a "Note" header in green. Notes are normative or informative depending on the whether they are in a normative or informative section, respectively.
Examples are in light khaki boxes, with khaki left border, and with a numbered "Example" header in khaki. Examples are always informative. The content of the example is in monospace font and may be syntax colored.
The metadata defined in this specification is used to provide annotations on anannotated table orgroup of tables, as defined in [tabular-data-model]. Annotated tables form the basis for all further processing, such as validating, converting, or displaying the tables.
All compliant applicationsMUST createannotated tables based on the algorithm defined here. All compliant applicationsMUST generate errors and stop processing if a metadata document:
Compliant applicationsMUST ignore properties (aside fromcommon properties) which are not defined in this specification andMUST generate a warning when they are encoutered.
If a property has a value that is not permitted by this specification, then if a default value is provided for that property, compliant applicationsMUST generate a warning and use that default value. If no default value is provided for that property, compliant applicationsMUST generate a warning and behave as if the property had not been specified. Additionally, including:
Metadata documents contain descriptions of groups of tables, tables, columns, rows, and cells, which are used to create annotations on anannotated tabular data model. Adescription object is a JSON object that describes a component of theannotated tabular data model (agroup of tables, atable or acolumn) and has one or more properties that are mapped into properties on that component. There are two types of description objects:
Thedescription objects contain a number of properties. These are:
name
of a column or thedc:provenance
of a table.For example, in thecolumn description:
{"name":"inventory_date","titles":"Inventory Date","dc:description":"The date of the operation that was performed.","datatype":{"base":"date","format":"M/d/yyyy"}}
the propertiesname
,titles
, anddc:description
are used to create thename,titles,datatype anddc:description
annotations on thecolumn in the data model. Thedatatype
property is aninherited property that also affects thevalue of eachcell in that column (seesection5.7Inherited Properties for more oninherited properties).
This section defines a set of properties and permitted values for annotating tabular data, and how these properties should be interpreted by applications.
A metadata document is a JSON document which holds an object at the top level. This object is adescription object of either agroup of tables or a singletable. A metadata document may contain other referenced or embeddeddescription objects,description objects fortables andcolumns. Additional JSON objects, not part of theannotated tabular data model, are used to describeschemas,dialect descriptions,foreign key definitions andtransformation definitions.
There are different types of properties on description objects:
Array properties hold an array of one or more objects, which are usuallydescription objects.
For example, thetables
property is an array property. Atable group description might contain:
"tables": [{ "url": "https://example.org/countries.csv", "tableSchema": "https://example.org/countries.json"}, { "url": "https://example.org/country_slice.csv", "tableSchema": "https://example.org/country_slice.json"}]
in which case thetables
property has a value that is an array of twotable description objects.
Any items within an array that are not valid objects of the type expected are ignored. If the supplied value of an array property is not an array (e.g. if it is an integer), compliant applicationsMUST issue a warning and proceed as if the property had been supplied with an empty array.
Link properties hold a single reference to another resource by URL. Their value is astring — resolved as a URL against thebase URL. If the supplied value of a link property is not a string (e.g. if it is an integer), compliant applicationsMUST issue a warning and proceed as if the property had been supplied with an empty string.
For example, theurl
property is a link property. Atable description might contain:
"url":"example-2014-01-03.csv"
in which case theurl
property on the table would have a single value, a link toexample-2014-01-03.csv
, resolved against thebase URL of the metadata document in which this was located. For example if the metadata document contained:
"@context":["http://www.w3.org/ns/csvw",{"@base":"http://example.org/"}]
this is equivalent to specifying:
"url":"http://example.org/example-2014-01-03.csv"
URI template properties contain a [URI-TEMPLATE] which can be used to generate a URI. These URI templates are expanded in the context of each row by combining the template with a set of variables with values as defined in [URI-TEMPLATE]. The following variables are set:
null
,The languages ofcell values are ignored.
_column
_column
is set to thecolumn number of thecolumn from theannotated table that is currently being processed._sourceColumn
_sourceColumn
is set to thesource number of the column that is currently being processed; this usually varies from_column
byskip columns._row
_row
is set to therow number of therow from theannotated table that is currently being processed._sourceRow
_sourceRow
is set to thesource number of the row that is currently being processed; this usually varies from_row
byskip rows andheader rows._name
_name
is set to the URI decodedcolumn name annotation, as defined in [tabular-data-model], for the column that is currently being processed. (Percent-decoding is necessary asname
may have been encoded if taken fromtitles
; this prevents double percent-encoding.)Theannotation value is the result of:
If the supplied value of a URI template property is not a string (e.g. if it is an integer), compliant applicationsMUST issue a warning and proceed as if the property had been supplied with an empty string.
For example, theaboutUrl
property holds a URI template that is used to generate a URL identifier for each row, which might look like:
"aboutUrl":"http://example.org/example.csv#row.{_row}"
Theabout URL annotations that are generated and used as identifiers for the rows would then look likehttp://example.org/example.csv#row.1
,http://example.org/example.csv#row.2
and so on.
Alternatively, with the CSV and metadata in thesection1.Introduction, theaboutUrl
might look like:
"aboutUrl":"http://example.org/tree/{on_street}/{GID}"
This would generate URIs such ashttp://example.org/tree/ADDISON%20AV/1
andhttp://example.org/tree/EMERSON%20ST/2
.
If the value of theon_street
orGID
column werenull
, the URL would still be generated with the null value generating an empty string in the URL. For example ifon_street
werenull
andGID
were3
, the generated URL would behttp://example.org/tree//3
.
Once the URI has been generated, it is resolved against the url of the table (e.g. the CSV file) to create an absolute URI. For example, given aaboutUrl
within a schema such as:
"aboutUrl":"#row.{_row}"
and given a CSV file athttp://example.com/temp.csv
, the URL for the first row will behttp://example.com/temp.csv#row.1
.
ThepropertyUrl
property might be defined as"{#_name}"
, meaning that it resolves as a fragment identifier relative to the URL of the source of the table. For example, accessing it from a column with the column nameGID would look like:
"http://example.org/example.csv#GID"
A value defined within the data is also subject to expansion. For example, consider the following table:
project_name,project_type,keywordsCSVW,foaf:Project,table;data;conversion
Theproject_type
column might have avalueUrl
specified as"{project_type}"
. In the first row the cell value is"foaf:Project"
. Thefoaf
prefix is understood, as described insection5.8Common Properties, to expand tohttp://xmlns.com/foaf/0.1/Project
.
Similarly, thekeywords
column might have avalueUrl
specified as"https://duckduckgo.com/?q={keywords}"
. If the column also specifies"separator": ";"
, then the cell value of thekeywords
column would be an array of the three valuestable
,data
, andconversion
. This is set as the value of thekeywords variable within the URI template, which means the result would behttps://duckduckgo.com/?q=table,data,conversion
.
If the value in thekeywords
column were an empty sequence (created from an empty cell in the original data), the reference to that column would be expanded to an empty string, generatinghttps://duckduckgo.com/?q=
.
When a cell'svalue is not a string, the canonical representation of that value is used within the expanded URL. For example, the data may include dates such as those in:
GID,On Street,Species,Trim Cycle,Inventory Date1,ADDISON AV,Celtis australis,Large Tree Routine Prune,10/18/20102,EMERSON ST,Liquidambar styraciflua,Large Tree Routine Prune,6/2/2010
TheInventory Date
column description would indicate that these were dates with the formatM/d/yyyy
:
{"name":"inventory_date","titles":"Inventory Date","datatype":{"base":"date","format":"M/d/yyyy"}}
The string value of theinventory_date
column in the first row is parsed to create the date 18th October 2010. When theinventory_date
column is referenced within a URI template such ashttp://example.org/event/{inventory_date}
, thecanonical representation of that date, as defined in [xmlschema11-2] is used within the URL, giving the resulthttp://example.org/event/2010-10-18
.
Column reference properties hold one or more references to othercolumn description objects. The referenced description object must have aname
property. Column reference properties can then reference column description objects through values that are:
name
on acolumn description object within the metadata document.Compliant applicationsMUST issue a warning and proceed as if the column reference property had not been specified if:
For example, theprimaryKey
property is a column reference property on the schema. It has to hold references to columns defined elsewhere in the schema, and the descriptions of those columns must havename
properties. It can hold a single reference, like this:
"tableSchema":{"columns":[{"name":"GID"},...],"primaryKey":"GID"}
or it can contain an array of references, like this:
"tableSchema":{"columns":[{"name":"givenName"},{"name":"familyName"},...],"primaryKey":["givenName","familyName"]}
If theprimaryKey
property were given an invalid value, such as1
, or a column name were misspelled, the processorMUST issue a warning and ignore the value.
On the other hand, thecolumnReference
property is a required property; if it has an invalid value, such as an empty array, then the processor will issue an error as if the property were not specified at all.
Object properties hold either a single object or a reference to an object by URL. Their values may be:
If the supplied value of an object property is not a string or object (e.g. if it is an integer), compliant applicationsMUST issue a warning and proceed as if the property had been specified as an object with no properties.
Object properties are often used when the values can be or should be values within controlled vocabularies, or structured information which may be held elsewhere. For example, thedialect
of a table is an object property. It could be provided as a URL that indicates a commonly used dialect, like this:
"dialect":"http://example.org/tab-separated-values"
or a structured object, like this:
"dialect":{"delimiter":"\t","encoding":"utf-8"}
When specified as a string, the resolved URL is used to fetch the referenced object duringnormalization as described insection6.Normalization. For example, ifhttp://example.org/tab-separated-values
resolved to:
{"@context":"http://www.w3.org/ns/csvw","quoteChar":null,"header":true,"delimiter":"\t"}
Followingnormalization, the value of thedialect
property would then be:
"dialect":{"@id":"http://example.org/tab-separated-values","quoteChar":null,"header":true,"delimiter":"\t"}
Natural language properties hold natural language strings. Their values may be:
Natural language properties are used for titles. For example, thetitles
property on acolumn description provides a natural language label for a column. If it's a plain string like this:
"titles":"Project title"
then that string is assumed to be in thedefault language (or have an undefined language,und
, if there is no such property). Multiple alternative values can be given in an array:
"titles":["Project title","Project"]
It's also possible to provide multiple values in different languages, using an object structure. For example:
"titles":{"en":"Project title","fr":"Titre du projet"}
and within such an object, the values of the properties can themselves be arrays:
"titles":{"en":["Project title","Project"],"fr":"Titre du projet"}
Theannotation value of anatural language property is an object whose properties are language codes and where the values of those properties are an array of strings (seeLanguage Maps in [JSON-LD]).
When extracting aannotation value from a metadata that will have already beennormalized, anatural language property will already have this form.
If the supplied value of a natural language property is not a string, array or object (e.g. if it is an integer), compliant applicationsMUST issue a warning and proceed as if the property had been specified as an empty array. If the supplied value is an array, any items in that array that are not stringsMUST be ignored. If the supplied value is an object, any properties that are not valid language codes as defined by [BCP47]MUST be ignored, as must any properties whose value is not a string or an array, and any items that are not strings within array values of these properties.
Atomic properties holdatomic values. Their values may be:
true
orfalse
). Theannotation value of a boolean atomic property isfalse
if unset; otherwise, theannotation value of an atomic property isnormalized value of that property, or the defined default value ornull
, if unset. ProcessorsMUST issue a warning if a property is set to an invalid value type, such as a boolean atomic property being set to the number1
or a numeric atomic property being set to the string"3.1415"
, and act as if the property had not been specified (which may mean using the default value for the property, or may mean raising an error and halting processing if the property is a required property).
The top-level object of a metadata document or object referenced through anobject property (whether it is atable group description,table description,schema,dialect description ortransformation definition)MUST have a@context
property. This is anarray property, as defined inSection 8.7 of [JSON-LD]. The@context
MUST have one of the following values:
http://www.w3.org/ns/csvw
, orhttp://www.w3.org/ns/csvw
and the object represents a local context definition, which is restricted to contain either or both of the following members:@base
anatomic property that provides the base URL against which other URLs within the metadata file are resolved. If present, its valueMUST be a string that is interpreted as a URL which is resolved against the location of the metadata document to provide thebase URL for other URLs in the metadata document; if unspecified, the base URL used for interpreting relative URLs within the metadata document is the location of the metadata document itself.
Note that the@base
property of the@context
object provides the base URL used for URLs within the metadata document, not the URLs that appear as data within the group of tables or table it describes.URI template properties are not resolved against this base URL: they are resolved against the URL of the table.
@language
anatomic property that indicates thedefault language for the values ofnatural language or string-valuedcommon properties in the metadata document; if present, its valueMUST be a language code [BCP47]. The default isund
.
Note that the@language
property of the@context
object, which gives the default language used within the metadata file, is distinct from thelang
property on adescription object, which gives the language used in the data within a group of tables, table, or column.
Atable group description is a JSON object that describes agroup of tables.
tables
Anarray property oftable descriptions for the tables in the group, namely those listed in thetables annotation on thegroup of tables being described. Compliant applicationMUST raise an error if this array does not contain one or more table descriptions.
The description of a group of tablesMAY also contain:
dialect
Anobject property that provides a singledialect description. If provided,dialect
provides hints to processors about how to parse the referenced files to create tabular data models for the tables in the group. This may be provided as an embedded object or as a URL reference. Seesection5.9Dialect Descriptions for more details.
notes
Anarray property that provides an array of objects representing arbitrary annotations on the annotatedgroup of tables. The value of this property becomes the value of thenotes annotation for thegroup of tables. The properties on these objects are interpreted equivalently tocommon properties as described insection5.8Common Properties.
TheWeb Annotation Working Group is developing a vocabulary for expressing annotations. In future versions of this specification, we anticipate referencing that vocabulary.
tableDirection
Anatomic property thatMUST have a single string value that is one of"rtl"
,"ltr"
, or"auto"
. Indicates whether the tables in the group should be displayed with the first column on the right, on the left, or based on the first character in the table that has a specific direction. The value of this property becomes the value of thetable direction annotation for all the tables in the table group. SeeBidirectional Tables in [tabular-data-model] for details. The default value for this property is"auto"
.
tableSchema
Anobject property that provides a singleschema description as described insection5.5Schemas, used as the default for all the tables in the group. This may be provided as an embedded object within the JSON metadata or as a URL reference to a separate JSON object that is aschema description.
transformations
Anarray property oftransformation definitions that provide mechanisms to transform the tabular data into other formats. The value of this property becomes the value of thetransformations annotation for all the tables in the table group.
@id
@id
is alink property that identifies thegroup of tables, as defined by [tabular-data-model], described by thistable group description. ItMUST NOT start with_:
. The value of this property becomes the value of theid annotation for thegroup of tables.@type
If included,@type
is anatomic property thatMUST be set to"TableGroup"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
The descriptionMAY contain anycommon properties to provide extra metadata about the group of tables as a whole.
The descriptionMAY containinherited properties to describe cells within the tables.
Twotable group descriptions arecompatible if anytable descriptions they contain with matchingnormalizedurl
properties are themselvescompatible as defined insection5.4.3Table Description Compatibility.
Atable description is a JSON object that describes a table within a CSV file.
url
Thislink property gives the single URL of the CSV file that the table is held in, relative to the location of the metadata document. The value of this property is the value of theurl annotation for theannotated table this table description describes.
The description of a tableMAY also contain:
dialect
As defined fortable groups.
notes
Anarray property that provides an array of objects representing arbitrary annotations on theannotated tabular data model. The value of this property becomes the value of thenotes annotation for thetable. The properties on these objects are interpreted equivalently tocommon properties as described insection5.8Common Properties.
TheWeb Annotation Working Group is developing a vocabulary for expressing annotations. In future versions of this specification, we anticipate referencing that vocabulary.
suppressOutput
A booleanatomic property. Iftrue
, suppresses any output that would be generated when converting this table. The value of this property becomes the value of thesuppress output annotation for this table. The default isfalse
.
tableDirection
As defined fortable groups. The value of this property becomes the value of thetable direction annotation for this table.
tableSchema
Anobject property that provides a singleschema description as described insection5.5Schemas. This may be provided as an embedded object within the JSON metadata or as a URL reference to a separate JSON schema document. If a table description is within atable group description, thetableSchema
from that table group acts as the default for this property.
If atableSchema
is not declared intable description, it may be declared on thetable group description, which is then used as theschema for thistable description.
The@id
property of thetableSchema
, if there is one, becomes the value of theschema annotation for this table.
When a schema is referenced by URL, this URL becomes the value of the@id
property in the normalized schema description, and thus the value of the schema annotation on the table.
transformations
As defined fortable groups. The value of this property becomes the value of thetransformations annotation for this table.
@id
If included,@id
is alink property that identifies thetable, as defined in [tabular-data-model], described by thistable description. ItMUST NOT start with_:
. The value of this property becomes the value of theid annotation for this table.
@type
If included,@type
is anatomic property thatMUST be set to"Table"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
The descriptionMAY contain anycommon properties to provide extra metadata about the table as a whole.
The descriptionMAY containinherited properties to describe cells within the table.
Two table descriptions arecompatible if they have equivalentnormalizedurl
properties, and havecompatibleschemas as defined insection5.5.1Schema Compatibility.
Aschema is a definition of a tabular format that may be common to multiple tables. For example, multiple tables from different sources may have the same columns and be designed such that they can be aggregated together.
Aschema description is a JSON object that encodes the information about aschema, which describes the structure of a table. All the properties of a schema description are optional.
columns
Anarray property ofcolumn descriptions as described insection5.6Columns. These are matched to columns in tables that use the schema by position: the first column description in the array applies to the first column in the table, the second to the second and so on.
Thename
properties of the column descriptionsMUST be unique within a given table description.
foreignKeys
Anarray property offoreign key definitions that define how the values from specified columns within this table link to rows within this table or other tables. Aforeign key definition is a JSON object thatMUST contain only the following properties:
columnReference
Acolumn reference property that holds either a single reference to acolumn description object within this schema, or an array of references. These form thereferencing columns for theforeign key definition.
reference
Anobject property that identifies areferenced table and a set ofreferenced columns within that table. Its properties are:
resource
Alink property holding a URL that is the identifier for a specific table that is being referenced. If this property is present thenschemaReference
MUST NOT be present. Thetable groupMUST contain atable whoseurl annotation is identical to the expanded value of this property. That table is thereferenced table.
schemaReference
Alink property holding a URL that is the identifier for aschema that is being referenced. If this property is present thenresource
MUST NOT be present. Thetable groupMUST contain atable with atableSchema
having a@id
that is identical to the expanded value of this property, and thereMUST NOT be more than one such table. That table is thereferenced table.
columnReference
Acolumn reference property that holds either a single reference (by name) to acolumn description object within thetableSchema
of thereferenced table, or an array of such references.
The value of this property becomes theforeign keys annotation on thetable using this schema by creating a list of foreign keys comprising a list of columns in the table and a list of columns in the referenced table. The value of this property is also used to create the value of thereferenced rows annotation on each of the rows in the table that uses this schema, which is a pair of the relevant foreign key and the referenced row in the referenced table.
As defined in [tabular-data-model],validatorsMUST check that, for each row, the combination of cells in thereferencing columns references a unique row within thereferenced table through a combination of cells in thereferenced columns. For examples, seesection5.5.2.1Foreign Key Reference Between Tables andsection5.5.2.2Foreign Key Reference Between Schemas.
It is not required for the table or schema referenced from aforeignKeys
property to have a similarly definedprimaryKey
, though frequently it will.
primaryKey
Acolumn reference property that holds either a single reference to acolumn description object or an array of references. The value of this property becomes theprimary key annotation for eachrow within atable that uses this schema by creating a list of the cells in that row that are in the referenced columns.
As defined in [tabular-data-model],validatorsMUST check that each row has a unique combination of values of cells in the indicated columns. For example, ifprimaryKey
is set to["familyName", "givenName"]
then every row must have a unique value for the combination of values of cells in thefamilyName
andgivenName
columns.
rowTitles
Acolumn reference property that holds either a single reference to acolumn description object or an array of references. The value of this property determines thetitles annotation for eachrow within atable that uses this schema. Thetitles annotation holds the list of thevalues of the cells in that row that are in the referenced columns; if the value is not a string or has no associated language, it is interpreted as a string with an undefined language (und
).
@id
If included,@id
is alink property that identifies theschema described by thisschema description. ItMUST NOT start with_:
.
@type
If included,@type
is anatomic property thatMUST be set to"Schema"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
The descriptionMAY contain anycommon properties to provide extra metadata about the schema as a whole.
The descriptionMAY containinherited properties to describe cells within tables that use this schema.
Two schemas arecompatible if they have the same number of non-virtual
column descriptions, and the non-virtual
column descriptions at the same index within each are compatible with each other.Column descriptions are compatible under the following conditions:
name
nortitles
properties.name
properties of the columns.titles
values, where matches must have a matching language;und
matches any language, and languages match if they are equal whentruncated, as defined in [BCP47], to the length of the shortest language tag.name
property but not atitles
property, and the other has atitles
property but not aname
property.Acolumn description within embedded metadata where theheader
dialect property isfalse
will have neithername
nortitles
properties.
This section is non-normative.
A list of countries is published athttp://example.org/countries.csv
with the structure:
countryCode,latitude,longitude,nameAD,42.5,1.6,AndorraAE,23.4,53.8,"United Arab Emirates"AF,33.9,67.7,Afghanistan
Another file contains information about the population in some countries each year, athttp://example.com/country_slice.csv
with the structure:
countryRef,year,populationAF,1960,9616353AF,1961,9799379AF,1962,9989846
The following metadata for thegroup of tables links the two together by defining aforeignKeys
property:
{"@context":"http://www.w3.org/ns/csvw","tables":[{"url":"http://example.org/countries.csv","tableSchema":{"columns":[{"name":"countryCode","datatype":"string","propertyUrl":"http://www.geonames.org/ontology{#_name}"},{"name":"latitude","datatype":"number"},{"name":"longitude","datatype":"number"},{"name":"name","datatype":"string"}],"aboutUrl":"http://example.org/countries.csv{#countryCode}","propertyUrl":"http://schema.org/{_name}","primaryKey":"countryCode"}},{"url":"http://example.org/country_slice.csv","tableSchema":{"columns":[{"name":"countryRef","valueUrl":"http://example.org/countries.csv{#countryRef}"},{"name":"year","datatype":"gYear"},{"name":"population","datatype":"integer"}],"foreignKeys":[{"columnReference":"countryRef","reference":{"resource":"http://example.org/countries.csv","columnReference":"countryCode"}}]}}]}
Within theannotated table generated forcountries.csv
, eachrow will have aprimary key annotation whose value is a list containing the cell from the first column of that row (countryCode
).
Theannotated table generated forcountry_slice.csv
will have aforeign keys annotation whose value is a list containing a single foreign key referencing the first column from the table generated fromcountry_slice.csv
(countryRef
) and the first column from the table generated fromcountries.csv
(countryCode
). Eachrow within that table will have areferenced row annotation referencing this foreign key and the third row in the table generated fromcountries.csv
.
When the population data incountry_slice.csv
is validated, thevalidator must check that everycountryRef
withincountry_slice.csv
has a matchingcountryCode
withincountries.csv
.
When publishing information about public sector roles and salaries, as inUse Case 4, the UK government requires departments to publish two files which are interlinked. The first lists senior grades (simplified here) e.g., atHEFCE_organogram_senior_data_31032011.csv
:
Post Unique Reference, Name,Grade, Job Title,Reports to Senior Post 90115, Steve Egan,SCS1A,Deputy Chief Executive, 90334 90250, David Sweeney,SCS1A, Director, 90334 90284, Heather Fry,SCS1A, Director, 90334 90334,Sir Alan Langlands, SCS4, Chief Executive, xx
The second provides information about the number of junior positions that report to those individuals (simplified here) e.g., atHEFCE_organogram_junior_data_31032011.csv
:
Reporting Senior Post,Grade,Payscale Minimum (£),Payscale Maximum (£),Generic Job Title,Number of Posts in FTE, Profession 90284, 4, 17426, 20002, Administrator, 2,Operational Delivery 90284, 5, 19546, 22478, Administrator, 1,Operational Delivery 90115, 4, 17426, 20002, Administrator, 8.67,Operational Delivery 90115, 5, 19546, 22478, Administrator, 0.5,Operational Delivery
The schemas are reused by multiple departments and for multiple pairs of files. The schemas are therefore defined in separate files, and they need to define links between the schemas which are then picked up as applying between tables that use those schemas.
The metadata file for the particular publication of the files above is:
{"@context":"http://www.w3.org/ns/csvw","tables":[{"url":"HEFCE_organogram_senior_data_31032011.csv","tableSchema":"http://example.org/schema/senior-roles.json"},{"url":"HEFCE_organogram_junior_data_31032011.csv","tableSchema":"http://example.org/schema/junior-roles.json"}]}
The schema for the senior role CSV (athttp://example.org/schema/senior-roles.json
) is as follows:
{"@id":"http://example.org/schema/senior-roles.json","@context":"http://www.w3.org/ns/csvw","columns":[{"name":"ref","titles":"Post Unique Reference"},{"name":"name","titles":"Name"},{"name":"grade","titles":"Grade"},{"name":"job","titles":"Job Title"},{"name":"reportsTo","titles":"Reports to Senior Post"}],"primaryKey":"ref"}
The schema for the junior role CSV (athttp://example.org/schema/junior-roles.json
) is as follows; it includes a foreign key reference to the senior roles schema:
{"@id":"http://example.org/schema/junior-roles.json","@context":"http://www.w3.org/ns/csvw","columns":[{"name":"reportsTo","titles":"Reporting Senior Post"},...],"foreignKeys":[{"columnReference":"reportsTo","reference":{"schemaReference":"http://example.org/schema/senior-roles.json","columnReference":"ref"}}]}
The foreign key definition here contains aschemaReference
tosenior-roles.json
. Implementations will look for the table referenced within the original metadata file whosetableSchema
issenior-roles.json
, which isHEFCE_organogram_senior_data_31032011.csv
. The implementation will therefore look for a relationship between thereportsTo
column inHEFCE_organogram_junior_data_31032011.csv
and theref
column inHEFCE_organogram_senior_data_31032011.csv
.
For example, in the first line ofHEFCE_organogram_junior_data_31032011.csv
, thereportsTo
(Reporting Senior Post
) column contains the value90284
. When validating that file,validators will check that there is a single row within the table generated fromHEFCE_organogram_senior_data_31032011.csv
whoseref
column contains the value90284
.
Foreign key definitions provide for strong linking between tables that guarantees (through validation) the existance of a referenced row. It is also possible to provide weak linking between tables that are not tested by validations, but which may be useful when converting tabular data into other formats, usingaboutUrl
andvalueUrl
.
Taking the example above as a starting point, the schema forHEFCE_organogram_senior_data_31032011.csv
could useaboutUrl
to provide a URL for each row, which can similarly be created as avalueUrl
for thereportsTo
column:
{"@id":"http://example.org/schema/senior-roles.json","@context":"http://www.w3.org/ns/csvw","aboutUrl":"#role-{ref}","columns":[{"name":"ref","titles":"Post Unique Reference"},{"name":"name","titles":"Name"},{"name":"grade","titles":"Grade"},{"name":"job","titles":"Job Title"},{"name":"reportsTo","titles":"Reports to Senior Post","valueUrl":"#role-{reportsTo}"}],"primaryKey":"ref"}
The URLs generated for the values of thereportsTo
will (if the data is correct) match the URLs generated for each row within the table. There will be no validation error, however, if there is a value in thereportsTo
column that does not match a value in theref
column. In contrast, if a foreign key had been specified with:
"foreignKeys":[{"columnReference":"reportsTo","reference":{"schemaReference":"http://example.org/schema/senior-roles.json","columnReference":"ref"}}]
then validators would raise an error if a value in thereportsTo
column did not match any value in theref
column.
Acolumn description is a JSON object that describes a single column. The description provides additional human-readable documentation for a column, as well as additional information that may be used to validate the cells within the column, create a user interface for data entry, or inform conversion into other formats. All properties are optional.
name
Anatomic property that gives a single canonical name for the column. The value of this property becomes thename annotation for the describedcolumn. ThisMUST be a string and this property has no default value, which means itMUST be ignored if the supplied value is not a string.
For ease of reference withinURI template properties, column names are restricted as defined inVariables in [URI-TEMPLATE] with the additional provision that names beginning with"_"
are reserved by this specification andMUST NOT be used within metadata documents.
suppressOutput
A booleanatomic property. Iftrue
, suppresses any output that would be generated when converting cells in this column. The value of this property becomes thesuppress output annotation for the describedcolumn. The default isfalse
.
titles
Anatural language property that provides possible alternative names for the column. The string values of this property, along with their associated language tags, become thetitles annotation for the describedcolumn.
If there is noname
property defined on this column, the firsttitles
value having the same language tag asdefault language, orund
or if no default language is specified, becomes thename annotation for the describedcolumn. This annotationMUST bepercent-encoded as necessary to conform to the syntactic requirements defined in [RFC3986].
virtual
A booleanatomic property taking a single value which indicates whether the column is avirtual column not present in the original source. The default value isfalse
. Thenormalized value of this property becomes thevirtual annotation for the describedcolumn. If present, a virtual columnMUST appear after all other non-virtual column definitions.
Virtual columns are useful for inserting cells with default values into anannotated table to control the results of conversions.
@id
If included,@id
is alink property that identifies thecolumns, as defined in [tabular-data-model], and potentially appearing across separate tables, described by thiscolumn description. ItMUST NOT start with_:
.
@type
If included,@type
is anatomic property thatMUST be set to"Column"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
If the column description has neithername
nortitles
properties, the string"_col.[N]"
where[N]
is thecolumn number, becomes thename annotation for the describedcolumn.
The descriptionMAY contain anycommon properties to provide extra metadata about the column as a whole, such as a full description.
The descriptionMAY containinherited properties to describe cells within the column.
This section is non-normative.
virtual
columns Virtual columns are useful when data needs to be added as part of an output transformation that doesn't exist in the source file. This may be to add type information to a column, or to relate different columns having differentaboutUrl
. For example, thehttp://example.org/tree-ops.csv
example used in the introduction can be used with the following metadata:
{"url":"tree-ops.csv","@context":["http://www.w3.org/ns/csvw",{"@language":"en"}],"tableSchema":{"columns":[{"name":"GID","titles":"GID","datatype":"string","propertyUrl":"schema:url","valueUrl":"#gid-{GID}"},{"name":"on_street","titles":"On Street","datatype":"string","aboutUrl":"#location-{GID}","propertyUrl":"schema:streetAddress"},{"name":"species","titles":"Species","datatype":"string","propertyUrl":"schema:name"},{"name":"trim_cycle","titles":"Trim Cycle","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","datatype":{"base":"date","format":"M/d/yyyy"},"aboutUrl":"#event-{inventory_date}","propertyUrl":"schema:startDate"},{"propertyUrl":"schema:event","valueUrl":"#event-{inventory_date}","virtual":true},{"propertyUrl":"schema:location","valueUrl":"#location-{GID}","virtual":true},{"aboutUrl":"#location-{GID}","propertyUrl":"rdf:type","valueUrl":"schema:PostalAddress","virtual":true}],"aboutUrl":"#gid-{GID}"}}
This metadata creates a relationship model between data in each column by different combinations ofaboutUrl
,propertyUrl
, andvalueUrl
on existing columns, and defining newvirtual columns to supply additional information. In this case, theon_street
andinventory_date
values are split into separate entities, each having their ownaboutUrl
. Newvirtual columns are defined to provide a location type, and to relate the main row entity to the event and location associated with it. The result of converting the table to RDF would include the following, for the first row, with the contributions from thevirtual columns highlighted:
<#gid-1> schema:url <#gid-1> ; schema:name "Celtis australis" ; :trim_cycle "Large Tree Routine Prune" ;schema:event <#event-2010-10-18> ;schema:location <#location-1> ; .<#event-1> a schema:Event ; schema:startDate "2010-10-18"^^xsd:date ; .<#location-1> a schema:PostalAddress ; schema:streetAddress "ADDISON AV" ; .
The JSON would similarly include, again with the contributions from thevirtual columns highlighted:
{"@id":"#gid-1","schema:url":"#gid-1","schema:name":"Celtis australis","trim_cycle":"Large Tree Routine Prune","schema:event":{"@id":"#event-1","@type":"schema:Event","schema:startDate":"2010-10-18"},"schema:location":{"@id":"#location-1","@type":"schema:PostalAddress","schema:streetAddress":"ADDISON AV"}}
Columns andcells may be assigned annotations based on properties on the description objects forgroups of tables,tables,schemas, orcolumns. These properties are known asinherited properties and are listed below.
If aninherited property is not defined on acolumn description, it defaults to the first value, if any, found by looking through all of its containing objects: aninherited property defined in its containingschema description takes precedence over one defined in its containingtable description, which in turn takes precedence of one defined in its containingtable group description. This value is used to determine the value of the relevant annotation on the describedcolumn, which is then used to determine the value of the relevant annotation on thecells in that column.
aboutUrl
AURI template property thatMAY be used to indicate what a cell contains information about. The value of this property becomes theabout URL annotation for the describedcolumn and is used to create the value of theabout URL annotation for the cells within that column as described insection5.1.3URI Template Properties.
aboutUrl
is typically defined on aschema description ortable description to indicate what each row is about. If defined on individualcolumn descriptions, care must be taken to ensure that transformed cell values maintain a semantic relationship.
datatype
Anatomic property that contains either a single string that is the main datatype of the values of the cell or adatatype description object. If the value of this property is a string, itMUST be the name of one of the built-in datatypes defined insection5.11.1Built-in Datatypes and this value isnormalized to an object whosebase
property is the original string value. If it is an object then it describes a more specialized datatype. If a cell contains a sequence (i.e. theseparator
property is specified and notnull
) then this property specifies the datatype of each value within that sequence. See5.11Datatypes andParsing Cells in [tabular-data-model] for more details.
Thenormalized value of this property becomes thedatatype annotation for the describedcolumn.
default
Anatomic property holding a single string that is used to create a defaultvalue for the cell in cases where the originalstring value is an empty string. SeeParsing Cells in [tabular-data-model] for more details. If not specified, the default for thedefault
property is the empty string,""
. The value of this property becomes thedefault annotation for the describedcolumn.
lang
Anatomic property giving a single string language code as defined by [BCP47]. Indicates the language of the value within the cell. SeeParsing Cells in [tabular-data-model] for more details. The value of this property becomes thelang annotation for the describedcolumn. The default isund
.
null
Anatomic property giving the string or strings used for null values within the data. If thestring value of the cell is equal to any one of these values, thecell value isnull
. SeeParsing Cells in [tabular-data-model] for more details. If not specified, the default for thenull
property is the empty string""
. The value of this property becomes thenull annotation for the describedcolumn.
ordered
A booleanatomic property taking a single value which indicates whether a list that is the value of the cell is ordered (iftrue
) or unordered (iffalse
). The default isfalse
. This property is irrelevant if theseparator
isnull
or undefined, but this is not an error. The value of this property becomes theordered annotation for the describedcolumn, and theordered annotation for thecells within that column.
propertyUrl
AURI template property thatMAY be used to create a URI for a property if the table is mapped to another format. The value of this property becomes theproperty URL annotation for the describedcolumn and is used to create the value of theproperty URL annotation for the cells within that column as described insection5.1.3URI Template Properties.
propertyUrl
is typically defined on acolumn description. If defined on aschema description,table description ortable group description, care must be taken to ensure that transformed cell values maintain an appropriate semantic relationship, for example by including thename of the column in the generated URL by using_name
in the template.
required
A booleanatomic property taking a single value which indicates whether thecell value can benull
. SeeParsing Cells in [tabular-data-model] for more details. The default isfalse
, which means cells can have null values. The value of this property becomes therequired annotation for the describedcolumn.
separator
Anatomic property thatMUST have a single string value that is the string used to separate items in the string value of the cell. Ifnull
(the default) or unspecified, the cell does not contain a list. Otherwise, applicationMUST split the string value of the cell on the specified separator and parse each of the resulting strings separately. Thecell's value will then be a list. SeeParsing Cells in [tabular-data-model] for more details. The value of this property becomes theseparator annotation for the describedcolumn.
textDirection
Anatomic property thatMUST have a single string value that is one of"ltr"
,"rtl"
,"auto"
or"inherit"
(the default). Indicates whether the text within cells should be displayed as left-to-right text (ltr
), as right-to-left text (rtl
), according to the content of the cell (auto
) or in the direction inherited from thetable direction annotation of thetable. The value of this property determines thetext direction annotation for thecolumn, and thetext direction annotation for thecells within that column: if the value isinherit
then the value of thetext direction annotation is the value of thetable direction annotation on the table, otherwise it is the value of this property. SeeBidirectional Tables in [tabular-data-model] for details.
valueUrl
AURI template property that is used to map the values of cells into URLs. The value of this property becomes thevalue URL annotation for the describedcolumn and is used to create the value of thevalue URL annotation for the cells within that column as described insection5.1.3URI Template Properties.
This allows processors to build URLs fromcell values, for example to reference RDFresources, as defined in [rdf-concepts]. For example, if the value URL were"{#reference}"
, each cell value of a column namedreference would be used to create a URI such ashttp://example.com/#1234
, if1234
were a cell value of that column.
valueUrl
is typically defined on acolumn description. If defined on aschema description,table description ortable group description, care must be taken to ensure that transformed cell values maintain an appropriate semantic relationship.
This section is non-normative.
In the following example,aboutUrl
property is defined on thetableSchema
, and therefore affects all cells for that table.
{"@context":["http://www.w3.org/ns/csvw",{"@language":"en"}],"url":"tree-ops.csv","dc:title":"Tree Operations","dcat:keyword":["tree","street","maintenance"],"dc:publisher":{"schema:name":"Example Municipality","schema:url":{"@id":"http://example.org"}},"dc:license":{"@id":"http://opendefinition.org/licenses/cc-by/"},"dc:modified":{"@value":"2010-12-31","@type":"xsd:date"},"tableSchema":{"columns":[{"name":"GID","titles":["GID","Generic Identifier"],"dc:description":"An identifier for the operation on a tree.","datatype":"string","required":true},{"name":"on_street","titles":"On Street","dc:description":"The street that the tree is on.","datatype":"string"},{"name":"species","titles":"Species","dc:description":"The species of the tree.","datatype":"string"},{"name":"trim_cycle","titles":"Trim Cycle","dc:description":"The operation performed on the tree.","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","dc:description":"The date of the operation that was performed.","datatype":{"base":"date","format":"M/d/yyyy"}}],"primaryKey":"GID","aboutUrl":"#gid-{GID}"}}
The equivalent effect could be achieved by using theaboutUrl
property on each column:
{"@context":["http://www.w3.org/ns/csvw",{"@language":"en"}],"url":"tree-ops.csv","dc:title":"Tree Operations","dcat:keyword":["tree","street","maintenance"],"dc:publisher":{"schema:name":"Example Municipality","schema:url":{"@id":"http://example.org"}},"dc:license":{"@id":"http://opendefinition.org/licenses/cc-by/"},"dc:modified":{"@value":"2010-12-31","@type":"xsd:date"},"tableSchema":{"columns":[{"name":"GID","titles":["GID","Generic Identifier"],"aboutUrl":"#gid-{GID}","dc:description":"An identifier for the operation on a tree.","datatype":"string","required":true},{"name":"on_street","titles":"On Street","aboutUrl":"#gid-{GID}","dc:description":"The street that the tree is on.","datatype":"string"},{"name":"species","titles":"Species","aboutUrl":"#gid-{GID}","dc:description":"The species of the tree.","datatype":"string"},{"name":"trim_cycle","titles":"Trim Cycle","aboutUrl":"#gid-{GID}","dc:description":"The operation performed on the tree.","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","aboutUrl":"#gid-{GID}","dc:description":"The date of the operation that was performed.","datatype":{"base":"date","format":"M/d/yyyy"}}],"primaryKey":"GID"}}
Descriptions of groups of tables, tables, schemas and columnsMAY contain anycommon properties whose names are either absolute URLs orprefixed names. For example, a table description may containdc:description
,dcat:keyword
, orschema:copyrightHolder
properties to provide a description, keywords, or the name of the copyright holder, as defined inDublin Core Terms,DCAT, orschema.org.
The names of common properties areprefixed names, in the syntaxprefix:name
.
Prefixed names can beexpanded to provide a URI, by replacing the prefix and following colon with the URI that the prefix is associated with. Expansion is intended to be entirely consistent withSection 6.3 IRI Expansion in [JSON-LD-API] and implementationsMAY use a JSON-LD processor for performing prefixed name and IRI expansion.
The prefixes that are recognized are those defined for [rdfa-core] within theRDFa 1.1 Initial Context and other prefixes defined within [csvw-context] and theseMUST NOT be overridden. These prefixes are periodically extended; refer to [csvw-context] for details. Properties from other vocabulariesMUST be named using absolute URLs.
Forbidding the declaration of new prefixes ensures consistent processing between JSON-LD-aware and non-JSON-LD-aware processors.
This specification does not define how common properties are interpreted by implementations. ImplementationsSHOULD treat theprefixed names for common properties and the URLs that they expand into in the same way. For example, if an implementation recognises and displays the value of thedc:description
property, it should also recognise and display the value of thehttp://purl.org/dc/terms/description
property in the same way.
Common properties can take any JSON value, so long as any objects within the value (for example as items of an array or values of properties on other objects) adhere to the following restrictions, which are designed to ensure compatibility between JSON-LD-aware and non-JSON-LD-aware processors:
If a@value
property is used on an object, that objectMUST NOT have any other properties aside from either@type
or@language
, andMUST NOT have both@type
and@language
as properties. The value of the@value
propertyMUST be a string, number, or boolean value.
If@type
is also used, its valueMUST be one of:
If a@language
property is used, itMUST have a string value that adheres to the syntax defined in [BCP47], or benull
.
If a@type
property is used on an object without a@value
property, its valueMUST be one of:
@type
as defined for any of the description objects in this specification. A@type
property can also have a value that is an array of such values.
The values of@id
properties arelink properties and are treated as URLs. Duringnormalization, as described insection6.Normalization, they will have any prefixexpanded and the result resolved against thebase URL. Therefore, if an@id
property is used on an object, itMUST have a value that is a string and that stringMUST NOT start with_:
.
A@language
propertyMUST NOT be used on an object unless it also has a@value
property.
@value
,@type
,@language
, and@id
, the properties used on an objectMUST NOT start with@
.These restrictions are also described insectionA.JSON-LD Dialect, from the perspective of a processor that otherwise supports JSON-LD. Examples of common property values and the impact ofnormalization are given insection6.1Examples.
Much of the tabular data that is published on the web is messy, and CSV parsers frequently need to be configured in order to correctly read in CSV. Adialect description provides hints to parsers about how to parse the file linked to from theurl
property in atable description. It can have any of the following properties, which relate to the flags described inSection 5 Parsing Tabular Data within the [tabular-data-model]:
commentPrefix
Anatomic property that sets thecomment prefix flag to the single provided value, whichMUST be a string. The default is"#"
.
delimiter
Anatomic property that sets thedelimiter flag to the single provided value, whichMUST be a string. The default is","
.
doubleQuote
A booleanatomic property that, iftrue
, sets theescape character flag to"
. Iffalse
, to\
. The default istrue
.
encoding
Anatomic property that sets theencoding flag to the single provided string value, whichMUST be a defined in [encoding]. The default is"utf-8"
.
header
A booleanatomic property that, iftrue
, sets theheader row count flag to1
, and iffalse
to0
, unlessheaderRowCount
is provided, in which case the value provided for theheader
property is ignored. The default istrue
.
headerRowCount
A numericatomic property that sets theheader row count flag to the single provided value, whichMUST be a non-negative integer. The default is1
.
lineTerminators
Anatomic property that sets theline terminators flag to either an array containing the single provided string value, or the provided array. The default is["\r\n", "\n"]
.
quoteChar
Anatomic property that sets thequote character flag to the single provided value, whichMUST be a string ornull
. If the value isnull
, theescape character flag is also set tonull
. The default is'"'
.
skipBlankRows
A booleanatomic property that sets theskip blank rows flag to the single provided boolean value. The default isfalse
.
skipColumns
A numericatomic property that sets theskip columns flag to the single provided numeric value, whichMUST be a non-negative integer. The default is0
.
skipInitialSpace
A booleanatomic property that, iftrue
, sets thetrim flag to"start"
and iffalse
, tofalse
. If thetrim
property is provided, theskipInitialSpace
property is ignored. The default isfalse
.
skipRows
A numericatomic property that sets theskip rows flag to the single provided numeric value, whichMUST be a non-negative integer. The default is0
.
trim
Anatomic property that, if the booleantrue
, sets thetrim flag totrue
and if the booleanfalse
tofalse
. If the value provided is a string, sets thetrim flag to the provided value, whichMUST be one of"true"
,"false"
,"start"
, or"end"
. The default istrue
.
@id
If included,@id
is alink property that identifies the dialect described by thisdialect description. ItMUST NOT start with_:
.
@type
If included,@type
is anatomic property thatMUST be set to"Dialect"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
Dialect descriptions do not provide a mechanism for handling CSV files in which there are multiple tables within a single file (e.g. separated by empty lines).
The default dialect description for CSV files is:
{ "encoding": "utf-8", "lineTerminators": ["\r\n", "\n"], "quoteChar": "\"", "doubleQuote": true, "skipRows": 0, "commentPrefix": "#", "header": true, "headerRowCount": 1, "delimiter": ",", "skipColumns": 0, "skipBlankRows": false, "skipInitialSpace": false, "trim": false}
Atransformation definition is a definition of how tabular data can be transformed into another format using a script or template.
For example, the following transformation definition will enable a processor that supports it to generate an iCalendar document using aMustache template based on the JSON created from the simple mapping to JSON.
{"url": "templates/ical.txt","titles": "iCalendar","targetFormat": "http://www.iana.org/assignments/media-types/text/calendar","scriptFormat": "https://mustache.github.io/","source": "json"}
A processor that recognises templates in the Mustache format indicated by"https://mustache.github.io/"
and that could convert tables into JSON based on [csv2json] would retrieve the template from"templates/ical.txt"
and apply this to the resulting JSON.
Transformation definitions have the following properties:
Transformation definitionsMUST have the following properties:
url
Alink property giving the single URL of the file that the script or template is held in, relative to the location of the metadata document.
scriptFormat
Alink property giving the single URL for the format that is used by the script or template. If one has been defined, this should be a URL for a media type, in the formhttp://www.iana.org/assignments/media-types/media-type
such ashttp://www.iana.org/assignments/media-types/application/javascript
. Otherwise, it can be any URL that describes the script or template format.
ThescriptFormat
URL is intended as an informative identifier for the template format, and applicationsSHOULD NOT access the URL. The template formats that an application supports are implementation defined.
targetFormat
Alink property giving the single URL for the format that will be created through the transformation. If one has been defined, this should be a URL for a media type, in the formhttp://www.iana.org/assignments/media-types/media-type
such ashttp://www.iana.org/assignments/media-types/text/calendar
. Otherwise, it can be any URL that describes the target format.
ThetargetFormat
URL is intended as an informative identifier for the target format, and applicationsSHOULD NOT access the URL.
Transformation definitionsMAY have the following properties:
source
A single stringatomic property that provides, if specified, the format to which the tabular data should be transformed prior to the transformation using the script or template. If the value isjson
, the tabular dataMUST first be transformed to JSON as defined by [csv2json] usingstandard mode. If the value isrdf
, the tabular dataMUST first be transformed to an RDF graph as defined by [csv2rdf] usingstandard mode. If thesource
property is missing ornull
(the default) then the source of the transformation is theannotated tabular data model. No other values are valid; applicationsMUST generate a warning and behave as if the property had not been specified.
titles
Anatural language property that describes the format that will be generated from the transformation. This is useful if the target format is a generic format (such asapplication/json
) and the transformation is creating a specific profile of that format.
@id
If included,@id
is alink property that identifies the transformation described by thistransformation definition. ItMUST NOT start with_:
.
@type
If included,@type
is anatomic property thatMUST be set to"Template"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
The transformation definitionMAY contain anycommon properties to provide extra metadata about the transformation.
ImplementationsMAY present users with options for transformations based on the availabletransformation definitions and their properties. ImplementationsSHOULD filter this list to only include those transformations whosescriptFormat
they understand and can apply, and whosesource
property, if present, specifies a format that the implementation can convert to. Users may find thetargetFormat
andtitles
properties useful in deciding which transformation to apply.
When directed by a user to transform a table using atransformation definition, implementationsMUST:
source
property, if this is specified and notnull
.url
property and raise an error if this does not exist.scriptFormat
property to determine how to interpret that script or template, and apply it to the table (or the result of converting the table). Cells within tables may be annotated with adatatype
which indicates the type of the values obtained by parsing thestring value of the cell. See [tabular-data-model] for a description of annotations on adatatype.
The possible built-in datatypes, as shown onthe diagram, are:
anyAtomicType
, whose identifier URLs are generated by prefixing the name withhttp://www.w3.org/2001/XMLSchema#
.number
whose identifier URL ishttp://www.w3.org/2001/XMLSchema#double
.binary
whose identifier URL ishttp://www.w3.org/2001/XMLSchema#base64Binary
.datetime
whose identifier URL ishttp://www.w3.org/2001/XMLSchema#dateTime
.any
whose identifier URL ishttp://www.w3.org/2001/XMLSchema#anyAtomicType
.xml
, a sub-type ofstring
, whose identifier URL ishttp://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral
and which indicates the value is an XML fragment.html
, a sub-type ofstring
, whose identifier URL ishttp://www.w3.org/1999/02/22-rdf-syntax-ns#HTML
and which indicates the value is an HTML fragment.json
, a sub-type ofstring
, whose identifier URL ishttp://www.w3.org/ns/csvw#JSON
and which indicates the value is serialized JSON.More specialized datatypes can be defined through adatatype description. A datatype description may have any of the following properties, all of which are optional.
base
Anatomic property that contains a single string: the name of one of the built-in datatypes, as listed above (and which are defined asterms in the default context). Its default isstring
. All values of the datatypeMUST be valid values of the base datatype. The value of this property becomes thebase annotation for the describeddatatype.
format
Anatomic property that contains either a single string or an object that defines the format of a value of this type, used when parsing a string value as described inParsing Cells in [tabular-data-model]. The value of this property becomes theformat annotation for the describeddatatype.
length
A numericatomic property that contains a single integer that is the exact length of the value. The value of this property becomes thelength annotation for the describeddatatype. SeeLength Constraints in [tabular-data-model] for details.
minLength
Anatomic property that contains a single integer that is the minimum length of the value. The value of this property becomes theminimum length annotation for the describeddatatype. SeeLength Constraints in [tabular-data-model] for details.
maxLength
A numericatomic property that contains a single integer that is the maximum length of the value. The value of this property becomes themaximum length annotation for the describeddatatype. SeeLength Constraints in [tabular-data-model] for details.
minimum
Anatomic property that contains a single number or string that is the minimum valid value (inclusive); equivalent tominInclusive
. The value of this property becomes theminimum annotation for the describeddatatype. SeeValue Constraints in [tabular-data-model] for details.
maximum
Anatomic property that contains a single number or string that is the maximum valid value (inclusive); equivalent tomaxInclusive
. The value of this property becomes themaximum annotation for the describeddatatype. SeeValue Constraints in [tabular-data-model] for details.
minInclusive
Anatomic property that contains a single number or string that is the minimum valid value (inclusive). The value of this property becomes theminimum annotation for the describeddatatype. SeeValue Constraints in [tabular-data-model] for details.
maxInclusive
Anatomic property that contains a single number or string that is the maximum valid value (inclusive). The value of this property becomes themaximum annotation for the describeddatatype. SeeValue Constraints in [tabular-data-model] for details.
minExclusive
Anatomic property that contains a single number or string that is the minimum valid value (exclusive). The value of this property becomes theminimum exclusive annotation for the describeddatatype. SeeValue Constraints in [tabular-data-model] for details.
maxExclusive
Anatomic property that contains a single number or string that is the maximum valid value (exclusive). The value of this property becomes themaximum exclusive annotation for the describeddatatype. SeeValue Constraints in [tabular-data-model] for details.
@id
If included,@id
is alink property that identifies the datatype described by thisdatatype description. The value of this property becomes theid annotation for the describeddatatype. ItMUST NOT start with_:
and itMUST NOT be the URL of a built-in datatype.
@type
If included,@type
is anatomic property thatMUST be set to"Datatype"
. PublishersMAY include this to provide additional information to JSON-LD based toolchains.
Thedatatype descriptionMAY contain anycommon properties to provide extra metadata about the datatype, such as a title or description.
ApplicationsMUST raise an error if the@id
property has the value of a built-in datatype and any other property is specified. In these cases, the other properties are ignored.
ApplicationsMUST raise an error if bothlength
andminLength
are specified andlength
is less thanminLength
. Similarly, applicationsMUST raise an error if bothlength
andmaxLength
are specified andlength
is greater thanmaxLength
. ApplicationsMUST raise an error ifminLength
andmaxLength
are both specified andminLength
is greater thanmaxLength
. ApplicationsMUST raise an error iflength
,maxLength
, orminLength
are specified and thebase
datatype is neitherstring
, a subtype ofstring
, nor a binary type.
In all ways, including the errors described below, theminimum
property is equivalent to theminInclusive
property and themaximum
property is equivalent to themaxInclusive
property. ApplicationsMUST raise an error if bothminimum
andminInclusive
are specified and they do not have the same value. Similarly, applicationsMUST raise an error if bothmaximum
andmaxInclusive
are specified and they do not have the same value.
ApplicationsMUST raise an error if bothminInclusive
andminExclusive
are specified, or if bothmaxInclusive
andmaxExclusive
are specified. ApplicationsMUST raise an error if bothminInclusive
andmaxInclusive
are specified andmaxInclusive
is less thanminInclusive
, or if bothminInclusive
andmaxExclusive
are specified andmaxExclusive
is less than or equal tominInclusive
. Similarly, applicationsMUST raise an error if bothminExclusive
andmaxExclusive
are specified andmaxExclusive
is less thanminExclusive
, or if bothminExclusive
andmaxInclusive
are specified andmaxInclusive
is less than or equal tominExclusive
.
ApplicationsMUST raise an error ifminimum
,minInclusive
,maximum
,maxInclusive
,minExclusive
, ormaxExclusive
are specified and thebase
datatype is not a numeric, date/time, or duration type.
Validation against these properties is as defined in [xmlschema11-2].
Theformat
property is not used when interpreting values for theminimum
,maximum
,minInclusive
,maxInclusive
,minExclusive
, ormaxExclusive
values. If these values are strings, they must be specified using a valid representation appropriate for the datatypebase
as defined in [xmlschema11-2]. For example:
{ "datatype": "date", "format": "dd/MM/yyyy", "minimum": "2000-01-01"}
Adescription object isnormalized as follows:
notes
the valueMUST be normalized as follows:@value
property whose value is that string. If adefault language is specified, add a@language
property whose value is that default language.@value
property, it remains as is.@id
,expand anyprefixed names and resolve its value against thebase URL.@type
, then its value remains as is.@context
. Normalize each property in the resulting object recursively using this algorithm and with its local@context
then remove the local@context
property. If the resulting object does not have an@id
property, add an@id
whose value is the original URL. This object becomes the value of the original object property.und
MUST be used. Following this normalization process, the@base
and@language
properties within the@context
are no longer relevant; the normalized metadata can have its@context
set tohttp://www.w3.org/ns/csvw
.
This section is non-normative.
The following are examples of how common properties are normalized.
In this example, a simple string is used as the title for atable using thedc:title
common property:
{"@context":{"http://www.w3.org/ns/csvw",{"@language":"en"}},"@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"dc:title":"The title of this Table"}
Since there is adefault language, this is equivalent to explicitly specifying the language of that title; the original string value becomes the value of the@value
property within avalue object:
{"@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"dc:title":{"@value":"The title of this Table","@language":"en"}}
It is also possible to use a simplevalue object to give a title. However, in this case thedefault language is not applied to the title:
{"@context":{"http://www.w3.org/ns/csvw",{"@language":"en"}},"@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"dc:title":{"@value":"The title of this Table"}}
The next example uses an array of a string and avalue object to give two titles with different languages:
{"@context":{"http://www.w3.org/ns/csvw",{"@language":"en"}},"@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"dc:title":["The title of this Table",{"@value":"Der Titel dieser Tabelle","@language":"de"}]}
The normalized version of this is:
{"@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"dc:title":[{"@value":"The title of this Table","@language":"en"}{"@value":"Der Titel dieser Tabelle","@language":"de"}]}
The next example demonstrates anode object, in which the value of theschema:url
property is a reference to another resource:
{"@context":["http://www.w3.org/ns/csvw",{"@base":"http://example.com/"}],"@type":"Table","url":"table.csv","tableSchema":[...],"schema:url":{"@id":"table.csv"}}
The value of the@id
property is normalized as described insection6.Normalization against thebase URL provided through the@base
property, which means the above example is equivalent to:
{"@context":"http://www.w3.org/ns/csvw","@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"schema:url":{"@id":"http://example.com/table.csv"}}
The following example shows thedc:publisher
property as an array that contains a singlenode object:
{"@context":"http://www.w3.org/ns/csvw","@type":"Table","url":"http://example.com/table.csv","tableSchema":[...],"dc:publisher":[{"schema:name":"Example Municipality","schema:url":{"@id":"http://example.org"}}],}
Following normalization, theschema:name
property of thedc:publisher
is expanded as shown:
"dc:publisher":[{"schema:name":{"@value":"Example Municipality"},"schema:url":{"@id":"http://example.org"}}]
Applications that process tabular data may use that data to drive other actions, which may have security implications. These behaviors are outside the scope of this specification.
Third party metadata provided about a tabular data file (such as a CSV file) may rename or ignore headers, or exclude rows or columns, which may lead to data being misinterpreted by applications that process it.
Transformation definitions are a possible security risk as they enable the creators of metadata to reference arbitrary code that may be executed to convert tabular data into other formats. Implementations should run this arbitrary code in a sandboxed environment to reduce the security risk.
The Metadata Vocabulary for Tabular Data uses a format based on JSON-LD [JSON-LD] with some restrictions.
The value of any@id
or@type
contained within a metadata documentMUST NOT be ablank node.
A metadata documentMUST NOT add a new context (i.e. include a@context
property except at the top level), or extend the top-level context in anyway other than as specifically allowed insection5.2Top-Level Properties.
Common properties and notes may contain arbitrary JSON-LD with the following restrictions:
The value of any member of@type
MUST be either aterm defined in [csvw-context], aprefixed name where the prefix is aterm defined in [csvw-context], or an absolute URL.
ValuesMAY be astring, native JSON type (such asnumber,true
, orfalse
),value object,node object or an array of zero or more of any of these.
ValuesMUST NOT uselist objects orset objects.
Keys ofnode objectsMUST NOT include@graph
,@context
,terms, orblank nodes.
When normalizing metadata,prefixed names used incommon properties andnotes
are expanded to absolute URLs. For some serializations, these are more appropriately presented usingprefixed names orterms. This algorithm compacts an absolute URL to aprefixed name orterm.
:
(U+0040
) to create aprefixed name. If the resultingprefixed name isrdf:type
, replace with@type
.This document is influenced byData Package specification and theJSON Table Schema, which are maintained as part ofData Protocols. Particular contributors to that work are Rufus Pollock, Paul Fitzpatrick, Andrew Berkeley, Francis Irving, Benoit Chesneau, Leigh Dodds, Martin Keegan, and Gunnlaugur Thor Briem.
text/csv
andtext/tab-delimited-values
media types, but a JSON-based format used to annotate such documents. Also see [RFC7159] for interoperability information for JSON. Although no byte sequence can be guaranteed at a specific location, a validapplication/csvm+json
documentMUST somewhere contain the string"@context"
(including quotation characters), followed by one or more whitespace, colon or open-square-bracket characters, followed by the string"http://www.w3.org/ns/csvw"
(including quotation characters).
org.w3c.csvm
conforms topublic.json
The JSON-LD context, located athttp://www.w3.org/ns/csvw.jsonld
is used with metadata documents. When used within a metadata document, the context can be referenced ashttp://www.w3.org/ns/csvw
. See [csvw-context] for a full description of definedterms andprefixes. This context may be updated from time-to-time to define new terms and prefixes.
A JSON-LD processor retrieving the context will usecontent negotiation to request the resource athttp://www.w3.org/ns/csvw
with anAccept: application/ld+json
HTTP header as defined inRemote Document and Context Retrieval in [JSON-LD-API]. Without this header, this resource will return an HTML representation of thecsvw
namespace. All serializations of this resource also define the metadata vocabulary usingRDFS.
application/csvm+json
has been registered at IANArowTitles
property has been added to help screen readers provide cues about focus within a table.@id
to reference an external datatype definition in XSD, OWL, or some other format.trim
changed fromfalse
totrue
.The document has undergone substantial changes since the last working draft. Below are some of the changes made:
notes
andcommon properties defined.resources
property was changed totables
.foreignKeys
.templates
property was changed totransformations
.urlTemplate
property was changed from aschema property to theaboutUrl
common property, andpropertyUrl
andvalueUrl
were added as common properties.virtual
columns to allow data to be inserted into a row.