Movatterモバイル変換

4.1Table groups

Agroup of tables comprises a set ofannotated tables and a set of annotations that relate to that group of tables. Thecore annotations of a group of tables are:

• id — an identifier for this group of tables, ornull if this is undefined.
• notes — any number of additional annotations on the group of tables. This annotation may be empty.
• tables — the list oftables in the group of tables. A group of tablesMUST have one or more tables.

Groups of tablesMAY in addition have any number ofannotations which provide information about the group of tables. Annotations on a group of tables may include:

• titles or descriptions of the group of tables.
• information about the source or provenance of the group of tables.
• links to other groups of tables (e.g. to those that provide similar data from a different time period).

When originating from [tabular-metadata], these annotations arise fromcommon properties defined ontable group descriptions within metadata documents.

4.2Tables

Anannotated table is atable that is annotated with additional metadata. Thecore annotations of a table are:

• columns — the list ofcolumns in the table. A tableMUST have one or more columns and the order of the columns within the list is significant andMUST be preserved by applications.
• table direction — the direction in which the columns in the table should be displayed, as described insection6.5.1Bidirectional Tables; the value of this annotation may also become the value of thetext direction annotation oncolumns andcells within the table, if thetextDirection property is set toinherit (the default).
• foreign keys — a list of foreign keys on the table, as defined in [tabular-metadata], which may be an empty list.
• id — an identifier for this table, ornull if this is undefined.
• notes — any number of additional annotations on the table. This annotation may be empty.
• rows — the list ofrows in the table. A tableMUST have one or more rows and the order of the rows within the list is significant andMUST be preserved by applications.
• schema — a URL referencing a schema applied to this table, ornull.
• suppress output — a boolean that indicates whether or not this table should be suppressed in any output generated from converting thegroup of tables, that this table belongs to, into another format, as described insection6.7Converting Tables.
• transformations — a (possibly empty) list of specifications for converting this table into other formats, as defined in [tabular-metadata].
• url — the URL of the source of the data in the table, ornull if this is undefined.

The tableMAY in addition have any number of otherannotations. Annotations on a table may include:

• titles or descriptions of the table,
• information about the source or provenance of the data in the table, or
• links to other tables (e.g. to indicate tables that include related information).

When originating from [tabular-metadata], these annotations arise fromcommon properties defined ontable descriptions within metadata documents.

4.3Columns

Acolumn represents a vertical arrangement ofcells within atable. Thecore annotations of a column are:

• about URL — theabout URLURI template used to create a URL identifier for eachvalue ofcell in this column relative to therow in which it is contained, as defined in [tabular-metadata].
• cells — the list ofcells in the column. A columnMUST contain one cell from eachrow in the table. The order of the cells in the listMUST match the order of the rows in which they appear within therows for the associatedtable.
• datatype — the expecteddatatype for thevalues ofcells in this column, as defined in [tabular-metadata].
• default — the defaultvalue forcells whosestring value is an empty string.
• lang — the code for the expected language for thevalues ofcells in this column, expressed in the format defined by [BCP47].
• name — the name of the column.
• null — the string or strings which cause thevalue ofcells havingstring value matching any of these values to benull.
• number — the position of the column amongst thecolumns for the associatedtable, starting from 1.
• ordered — a boolean that indicates whether the order of values of a cell should be preserved or not.
• property URL — the expectedproperty URLURI template used to create a URL identifier for the property of eachvalue ofcell in this column relative to therow in which it is contained, as defined in [tabular-metadata].
• required — a boolean that indicates thatvalues ofcells in this columnMUST NOT be empty.
• separator — a string value used to create multiplevalues ofcells in this column by splitting thestring value on the separator.
• source number — the position of the column in the file at theurl of the table, starting from 1, ornull.
• suppress output — a boolean that indicates whether or not this column should be suppressed in any output generated from converting the table, as described insection6.7Converting Tables.
• table — thetable in which the column appears.
• text direction — the indicator of thetext directionvalues ofcells in this column, as described insection6.5.1Bidirectional Tables; the value of this annotation may be derived from thetable direction annotation on thetable, if thetextDirection property is set toinherit (the default).
• titles — any number of human-readable titles for the column, each of whichMAY have an associated language code as defined by [BCP47].
• value URL — the expectedvalue URLURI template used to create the URL identifier for thevalue of eachcell in this, as defined in [tabular-metadata].
• virtual — a boolean that indicates whether the column is avirtual column.Virtual columns are used to extend the source data with additional empty columns to support more advanced conversions; when this annotation isfalse, the column is areal column, which exists in the source data for the table.

ColumnsMAY in addition have any number of otherannotations, such as a description. When originating from [tabular-metadata], these annotations arise fromcommon properties defined oncolumn descriptions within metadata documents.

4.4Rows

Arow represents a horizontal arrangement ofcells within atable. Thecore annotations of a row are:

• cells — the list ofcells in the row. A rowMUST contain one cell from eachcolumn in the table. The order of the cells in the listMUST match the order of the columns in which they appear within thetable columns for the row's table.
• number — the position of the row amongst therows for thetable, starting from 1.
• primary key — a possibly empty list ofcells whose values together provide a unique identifier for this row. This is similar to thename of acolumn.
• titles — any number of human-readable titles for the row, each of whichMAY have an associated language code as defined by [BCP47].
• referenced rows — a possibly empty list of pairs of aforeign key and a row in a table within the samegroup of tables (which may be another row in the table in which this row appears).
• source number — the position of the row in the originalurl of the table, starting from 1, ornull.
• table — thetable in which the row appears.

RowsMAY have any number of additionalannotations. The annotations on a row provide additional metadata about the information held in the row, such as:

• the certainty of the information in that row.
• information about the source or provenance of the data in that row.

Neither this specification nor [tabular-metadata] defines a method to specify such annotations. ImplementationsMAY define a method for adding annotations to rows by interpretingnotes on the table.

4.5Cells

Acell represents a cell at the intersection of arow and acolumn within atable. Thecore annotations of a cell are:

• about URL — an absolute URL for the entity about which this cell provides information, ornull.
• column — thecolumn in which the cell appears; the cellMUST be in thecells for that column.
• errors — a (possibly empty) list of validation errors generated while parsing the value of the cell.
• ordered — a boolean that, if the value of this cell is a list, indicates whether the order of that list should be preserved or not.
• property URL — an absolute URL for the property associated with this cell, ornull.
• row — therow in which the cell appears; the cellMUST be in thecells for that row.
• string value — a string that is the original syntactic representation of the value of the cell, e.g. how the cell appears within a CSV file; this may be an empty string.
• table — thetable in which the cell appears.
• text direction — which direction the text within the cell should be displayed, as described insection6.5.1Bidirectional Tables; the value of this annotation may be derived from thetable direction annotation on thetable, if thetextDirection property is set toinherit (the default).
• value — the semantic value of the cell; thisMAY be alist of values, each of whichMAY have adatatype other than a string,MAY have alanguage andMAY benull. For example, annotations might enable a processor to understand thestring value of the cell as representing a number or a date. By default, if thestring value is an empty string, the semantic value of the cell isnull.
• value URL — an absolute URL for this cell's value, ornull.

CellsMAY have any number of additionalannotations. The annotations on a cell provide metadata about the value held in the cell, particularly when this overrides the information provided for thecolumn androw that the cell falls within. Annotations on a cell might be:

• notes to aid the interpretation of the value.
• information about the source or provenance of the data in that cell.
• indication of the units of measure used within a cell.

Neither this specification nor [tabular-metadata] defines a method to specify such annotations. ImplementationsMAY define a method for adding annotations to cells by interpretingnotes on the table.

4.6Datatypes

Columns andcell values within tables may be annotated with adatatype which indicates the type of the values obtained by parsing thestring value of the cell.

Datatypes are based on a subset of those defined in [xmlschema11-2]. Theannotated tabular data model limitscell values to have datatypes as shown onthe diagram:

• the datatypes defined in [xmlschema11-2] as derived from and includingxsd:anyAtomicType.
• the datatyperdf:XMLLiteral, a sub-type ofxsd:string, which indicates the value is an XML fragment.
• the datatyperdf:HTML, a sub-type ofxsd:string, which indicates the value is an HTML fragment.
• the datatypecsvw:JSON, a sub-type ofxsd:string, which indicates the value is serialized JSON.
• datatypes derived from any of these datatypes.

Thecore annotations of a datatype are:

• id — the absolute URL that identifies the datatype, ornull if undefined;convertersSHOULD use this URL when serializing values of this datatype. ProcessorsMAY use this annotation to perform additional validation on column values using this datatype.
• base — the absolute URL that identifies the datatype from which this datatype is derived. ThisMUST be the URL of a built-in datatype as listed above, ornull if the datatype isxsd:anyAtomicType. Allvalues of the datatypeMUST be valid values of the base datatype.
• format — a string or object that defines the format of a value of this type, used when parsing acell string value as described in6.4Parsing Cells.
• length — a number that the exact length of acell value as described insection4.6.1Length Constraints.
• minimum length — a number that the minimum length of acell value as described insection4.6.1Length Constraints.
• maximum length — a number that the maximum length of acell value as described insection4.6.1Length Constraints.
• minimum — a number that the minimum valid value (inclusive) of acell value as described insection4.6.2Value Constraints.
• maximum — a number that the maximum valid value (inclusive) of acell value as described insection4.6.2Value Constraints.
• minimum exclusive — a number that the minimum valid value (exclusive) of acell value as described insection4.6.2Value Constraints.
• maximum exclusive — a number that the maximum valid value (exclusive) of acell value as described insection4.6.2Value Constraints.

If theid of a datatype is that of a built-in datatype, the values of the other core annotations listed aboveMUST be consistent with the values defined in [xmlschema11-2] or above. For example, if theid isxsd:integer then thebase must bexsd:decimal.

DatatypesMAY have any number of additionalannotations. The annotations on a datatype provide metadata about the datatype such as title or description. These arise fromcommon properties defined on datatype descriptions within metadata documents, as defined in [tabular-metadata].

5.1Overriding Metadata

ProcessorsSHOULD provide users with the facility to provide their own metadata fortabular data files that they process. This might be provided:

• through processor options, such as command-line options for a command-line implementation or checkboxes in a GUI.
• by enabling the user to select an existing metadata file, which may be local or remote.
• by enabling the user to specify a series of metadata files, which are merged by the processor and handled as if they were a single file.

For example, a processor might be invoked with:

$ csvlint data.csv --datatypes:string,float,string,string

to enable the testing of the types of values in the columns of a CSV file, or with:

$ csvlint data.csv --schema:schema.json

to supply a schema that describes the contents of the file, against which it can be validated.

Metadata supplied in this way is called overriding, or user-supplied, metadata. ImplementationsSHOULD define how any options they define are mapped into the vocabulary defined in [tabular-metadata]. If the user selects existing metadata files, implementationsMUST NOT use metadata located through the Link header (as described insection5.2Link Header) or site-wide location configuration (as described insection5.3Default Locations and Site-wide Location Configuration).

5.2Link Header

If the user has not supplied a metadata file as overriding metadata, described insection5.1Overriding Metadata, then when retrieving atabular data file via HTTP, processorsMUST retrieve the metadata file referenced by anyLink header with:

• rel="describedby", and
• type="application/csvm+json",type="application/ld+json" ortype="application/json".

so long as this referenced metadata file describes the retrievedtabular data file (ie, contains atable description whoseurl matches the request URL).

If there is more than one valid metadata file linked to through multipleLink headers, then implementationsMUST use the metadata file referenced by the lastLink header.

For example, when the response to requesting a tab-separated file looks like:

HTTP/1.1200 OKContent-Type: text/tab-separated-values...Link:<metadata.json>; rel="describedBy"; type="application/csvm+json"

an implementation must use the referencedmetadata.json to supply metadata for processing the file.

If the metadata file found at this location does not explicitly include a reference to the requestedtabular data file then itMUST be ignored. URLsMUST be normalized as described insection6.3URL Normalization.

Link: <http://example.org/countries.csv>; rel="describes"; type="text/csv"Link: <http://example.org/country_slice.csv>; rel="describes"; type="text/csv"

5.3Default Locations and Site-wide Location Configuration

If the user has not supplied a metadata file as overriding metadata, described insection5.1Overriding Metadata, and no applicable metadata file has been discovered through aLink header, described insection5.2Link Header, processorsMUST attempt to locate a metadata documents through site-wide configuration.

In this case, processorsMUST retrieve the file from the well-known URI/.well-known/csvm. (Well-known URIs are defined by [RFC5785].) If no such file is located (i.e. the response results in a client error4xx status code or a server error5xx status code), processorsMUST proceed as if this file were found with the following content which defines default locations:

{+url}-metadata.jsoncsv-metadata.json

The response to retrieving/.well-known/csvmMAY be cached, subject to cache control directives. This includes caching an unsuccessful response such as a404 Not Found.

This fileMUST contain a URI template, as defined by [URI-TEMPLATE], on each line. Starting with the first such URI template, processorsMUST:

1. Expand the URI template, with the variableurl being set to the URL of the requestedtabular data file (with any fragment component of that URL removed).
2. Resolve the resulting URL against the URL of the requestedtabular data file.
3. Attempt to retrieve a metadata document at that URL.
4. If no metadata document is found at that location, or if the metadata file found at the location does not explicitly include a reference to the relevanttabular data file, perform these same steps on the next URI template, otherwise use that metadata document.

For example, if thetabular data file is athttp://example.org/south-west/devon.csv then processors must attempt to locate a well-known file athttp://example.org/.well-known/csvm. If that file contains:

{+url}.jsoncsvm.json/csvm?file={url}

the processor will first look forhttp://example.org/south-west/devon.csv.json. If there is no metadata file in that location, it will then look forhttp://example.org/south-west/csvm.json. Finally, if that also fails, it will look forhttp://example.org/csvm?file=http://example.org/south-west/devon.csv.json.

If no file were found athttp://example.org/.well-known/csvm, the processor will use the default locations and try to retrieve metadata fromhttp://example.org/south-west/devon.csv-metadata.json and, if unsuccessful,http://example.org/south-west/csv-metadata.json.

5.4Embedded Metadata

Most syntaxes for tabular data provide a facility forembedding metadata within thetabular data file itself. The definition of a syntax for tabular dataSHOULD include a description of how the syntax maps to an annotated data model, and in particular how any embedded metadata is mapped into the vocabulary defined in [tabular-metadata]. Parsing based on the default dialect for CSV, as described in8.Parsing Tabular Data, will extractcolumn titles from the first row of a CSV file.

GID,On Street,Species,Trim Cycle,Inventory Date1,ADDISON AV,Celtis australis,Large Tree Routine Prune,10/18/20102,EMERSON ST,Liquidambar styraciflua,Large Tree Routine Prune,6/2/2010

The results of this can be found insection8.2.1Simple Example.

For another example, the following tab-delimited file contains embedded metadata where it is assumed that comments may be added using a#, and that the column types may be indicated using a#datatype annotation:

# publisher City of Palo Alto# updated 12/31/2010#name GID on_street species trim_cycle  inventory_date#datatype string  string  string  string  date:M/D/YYYY  GID On Street Species Trim Cycle  Inventory Date  1 ADDISON AV  Celtis australis  Large Tree Routine Prune  10/18/2010  2 EMERSON ST  Liquidambar styraciflua Large Tree Routine Prune  6/2/2010

A processor that recognises this format may be able to extract and make sense of this embedded metadata.

6.1Creating Annotated Tables

After locating metadata, metadata isnormalized and coerced into a singletable group description. When starting with a metadata file, this involves normalizing the provided metadata file and verifying that theembedded metadata for eachtabular data file referenced from the metadata iscompatible with the metadata. When starting with atabular data file, this involves locating the first metadata file as described insection5.Locating Metadata and normalizing into a single descriptor.

If processing starts with atabular data file, implementations:

1. Retrieve the tabular data file.
2. Retrieve the first metadata file (FM) as described insection5.Locating Metadata:
   1. metadata supplied by the user (seesection5.1Overriding Metadata).
   2. metadata referenced from aLink Header that may be returned when retrieving the tabular data file (seesection5.2Link Header).
   3. metadata retrieved through a site-wide location configuration (seesection5.3Default Locations and Site-wide Location Configuration).
   4. embedded metadata as defined insection5.4Embedded Metadata with a singletables entry where theurl property is set from that of thetabular data file.
3. Proceed as if the process starts withFM.

If the process starts with a metadata file:

1. Retrieve the metadata file yielding the metadataUM (which is treated as overriding metadata, seesection5.1Overriding Metadata).
2. NormalizeUM using the process defined inNormalization in [tabular-metadata], coercingUM into atable group description, if necessary.
3. For eachtable (TM) inUM in order, create one or moreannotated tables:
   1. Extract thedialect description (DD) fromUM for thetable associated with the tabular data file. If there is no such dialect description, extract the first available dialect description from agroup of tables in which the tabular data file is described. Otherwise use the default dialect description.
   2. If using the defaultdialect description, override default values inDD based on HTTP headers found when retrieving the tabular data file:
      • If the media type from theContent-Type header istext/tab-separated-values, setdelimiter toTAB inDD.
      • If theContent-Type header includes theheader parameter with a value ofabsent, setheader tofalse inDD.
      • If theContent-Type header includes thecharset parameter, setencoding to this value inDD.

   3. Parse the tabular data file, usingDD as a guide, to create a basic tabular data model (T) and extractembedded metadata (EM), for example from theheader line.

      Note

      This specification provides a non-normative definition for parsing CSV-based files, including the extraction ofembedded metadata, insection8.Parsing Tabular Data. This specification does not define any syntax for embedded metadata beyond this; whatever syntax is used, it's assumed that metadata can be mapped to the vocabulary defined in [tabular-metadata].

   4. If aContent-Language HTTP header was found when retrieving the tabular data file, and the value provides a single language, set thelang inherited property to this value inTM, unlessTM already has alang inherited property.
   5. Verify thatTM is compatible withEM using the procedure defined inTable Description Compatibility in [tabular-metadata]; ifTM is not compatible withEM validatorsMUST raise an error, other processorsMUST generate a warning and continue processing.
   6. Use the metadataTM to add annotations to the tabular data modelT as described inSection 2 Annotating Tables in [tabular-metadata].

6.2Metadata Compatibility

When processing atabular data file using metadata as discovered usingsection5.Locating Metadata, processorsMUST ensure that the metadata andtabular data file are compatible, this is typically done by extractingembedded metadata from thetabular data file and determining that the provided or discovered metadata is compatible with theembedded metadata using the procedure defined inTable Compatibility in [tabular-metadata].

6.3URL Normalization

Metadata Discovery and Compatibility involve comparing URLs. When comparing URLs, processorsMUST useSyntax-Based Normalization as defined in [RFC3968]. ProcessorsMUST performScheme-Based Normalization for HTTP (80) and HTTPS (443) andSHOULD performScheme-Based Normalization for other well-known schemes.

6.4Parsing Cells

Unlike many other data formats, tabular data is designed to be read by humans. For that reason, it's common for data to be represented within tabular data in a human-readable way. Thedatatype,default,lang,null,required, andseparator annotations provide the information needed to parse thestring value of a cell into its (semantic)value annotation. This is used:

• byvalidators to check that the data in the table is in the expected format,
• byconverters to parse the values before mapping them into values in the target of the conversion,
• whendisplaying data, to map it into formats that are meaningful for those viewing the data (as opposed to those publishing it), and
• wheninputting data, to turn entered values into representations in a consistent format.

The process of parsing a cell creates acell with annotations based on the original string value, parsed value and othercolumn annotations and adds the cell to the list ofcells in arow andcells in acolumn:

• The raw string value becomes thestring value annotation on thecell.
• Theordered annotation on thecolumn becomes theordered annotation on thecell.
• Thetext direction annotation on thecolumn becomes thetext direction annotation on thecell.
• Therow becomes therow annotation on thecell.
• Thecolumn becomes thecolumn annotation on thecell.

After parsing, thecell value can be:

• null,
• a single value with an associated optional datatype or language, or
• a sequence of such values.

The process of parsing the string value into a single value or a list of values is as follows:

1. unless thedatatype base isstring,json,xml,html oranyAtomicType, replace all carriage return (#xD), line feed (#xA), and tab (#x9) characters with space characters.
2. unless thedatatype base isstring,json,xml,html,anyAtomicType, ornormalizedString, strip leading and trailing whitespace from thestring value and replace all instances of two or more whitespace characters with a single space character.
3. if the normalized string is an empty string, apply the remaining steps to the string given by thecolumn default annotation.
4. if thecolumn separator annotation is notnull and the normalized string is an empty string, thecell value is an empty list. If thecolumn required annotation istrue, add an error to the list oferrors for the cell.
5. if thecolumn separator annotation is notnull, thecell value is a list of values; set thelist annotation on thecell totrue, and create thecell value created by:
   1. if the normalized string is the same as any one of the values of thecolumn null annotation, then the resulting value isnull.
   2. split the normalized string at the character specified by thecolumn separator annotation.
   3. unless thedatatype base isstring oranyAtomicType, strip leading and trailing whitespace from these strings.
   4. applying the remaining steps to each of the strings in turn.
6. if the string is an empty string, apply the remaining steps to the string given by thecolumn default annotation.
7. if the string is the same as any one of the values of thecolumn null annotation, then the resulting value isnull. If thecolumn separator annotation isnull and thecolumn required annotation istrue, add an error to the list oferrors for the cell.
8. parse the string using thedatatype format if one is specified, as described below to give a value with an associateddatatype. If thedatatype base isstring, or there is nodatatype, the value has an associatedlanguage from thecolumn lang annotation. If there are any errors, add them to the list oferrors for the cell; in this case the value has adatatype ofstring; if thedatatype base isstring, or there is nodatatype, the value has an associatedlanguage from thecolumn lang annotation.
9. validate the value based on the length constraints described insection4.6.1Length Constraints, the value constraints described insection4.6.2Value Constraints and thedatatype format annotation if one is specified, as described below. If there are any errors, add them to the list oferrors for the cell.

The final value (or values) become thevalue annotation on thecell.

If there is aabout URL annotation on thecolumn, it becomes theabout URL annotation on thecell, after being transformed into an absolute URL as described inURI Template Properties of [tabular-metadata].

If there is aproperty URL annotation on thecolumn, it becomes theproperty URL annotation on thecell, after being transformed into an absolute URL as described inURI Template Properties of [tabular-metadata].

If there is avalue URL annotation on thecolumn, it becomes thevalue URL annotation on thecell, after being transformed into an absolute URL as described inURI Template Properties of [tabular-metadata]. Thevalue URL annotation isnull if thecell value isnull and thecolumn virtual annotation isfalse.

"datatype":"integer"

"datatype":{"base":"integer","minimum":1,"maximum":10},"null":"99"

"datatype":{"base":"integer","minimum":1,"maximum":10},"default":"5"

"datatype":{"base":"integer","minimum":1,"maximum":10},"default":"5","separator":" "

"datatype":"decimal","separator":" ","valueUrl":"{?values}"

6.5Presenting Tables

This section is non-normative.

When presenting tables, implementations should:

• use thetable direction annotation on eachtable, and thetext direction annotation on each cell, to determine the ordering of columns and characters within cells, as described insection6.5.1Bidirectional Tables
• use thetitles annotation on eachcolumn to provide a header for the column, selecting the first title in a language based on the user's locale and preferences, as described insection6.5.2Column and row labelling
• add links to headers based on theproperty URLs of the cells in the first row of the table
• presentcell values, particularly boolean, numeric and date/time values, in a lexical form based on the user's locale and preferences
• add links to the presentation of rows and cells based on theabout URL andvalue URL annotations on cells
• highlight or otherwise indicate cells witherrors
• provide a way of viewing non-coreannotations on table groups, tables, columns, rows and cells
• provide links to download the rawtabular data file that is being displayed

6.6Validating Tables

Validators test whether giventabular data files adhere to the structure defined within aschema. ValidatorsMUST raise errors (and halt processing) and issue warnings (and continue processing) as defined in [tabular-metadata]. In addition, validatorsMUST raise errors butMAY continue validating in the following situations:

• if thetable description is not compatible with theembedded metadata extracted from thetabular data file, as defined inTable Compatibility in [tabular-metadata].
• if there is more than onerow with the sameprimary key, that is where thecells listed for the primary key for the row have the samevalues as the cells listed for the primary key for another row,
• for eachrow that does not have a uniquereferenced row for each of theforeign keys on thetable in which the row appears, or
• for eacherror on eachcell.

6.7Converting Tables

Conversions of tabular data to other formats operate over aannotated table constructed as defined inAnnotating Tables in [tabular-metadata]. The mechanics of these conversions to other formats are defined in other specifications such as [csv2json] and [csv2rdf].

Conversion specificationsMUST define a default mapping from an annotated table that lacks any annotations (i.e., that is equivalent to an un-annotated table).

Conversion specificationsMUST use theproperty value of thepropertyUrl of a column as the basis for naming machine-readable fields in the target format, such as the name of the equivalent element or attribute in XML, property in JSON or property URI in RDF.

Conversion specificationsMAY use any of theannotations found on an annotated table group, table, column, row or cell, including non-core annotations, to adjust the mapping into another format.

Conversion specificationsMAY define additional annotations, not defined in this specification, which are specifically used when converting to the target format of the conversion. For example, a conversion to XML might specify ahttp://example.org/conversion/xml/element-or-attribute property on columns that determines whether a particular column is represented through an element or an attribute in the data.

7.1Content Type

The appropriate content type for a CSV file istext/csv. For example, when a CSV file is transmitted via HTTP, the HTTP response should include aContent-Type header with the valuetext/csv:

Content-Type: text/csv

7.2Encoding

CSV files should be encoded using UTF-8, and should be in Unicode Normal Form C as defined in [UAX15]. If a CSV file is not encoded using UTF-8, the encoding should be specified through thecharset parameter in theContent-Type header:

Content-Type: text/csv;charset=ISO-8859-1

7.3Line Endings

The ends of rows in a CSV file should beCRLF (U+000D U+000A) but may beLF (U+000A). Line endings within escapedcells are not normalised.

7.4Lines

Each line of a CSV file should contain the same number of comma-separated values.

Values that contain commas, line endings, or double quotes should be escaped by having the entire value wrapped in double quotes. There should not be whitespace before or after the double quotes. Within these escaped cells, any double quotes should be escaped with two double quotes ("").

Content-Type: text/csv;header=absent

7.5Grammar

This grammar is a generalization of that defined in [RFC4180] and is included for reference only.

TheEBNF used here is defined in XML 1.0 [EBNF-NOTATION].

8.1Bidirectionality in CSV Files

This section is non-normative.

Bidirectional content does not alter the definition of rows or the assignment of cells to columns. Whether or not a CSV file contains right-to-left characters, the first column's content is the first cell of each row, which is the text prior to the first occurrence of a comma within that row.

‌ا‌ل‌م‌ح‌ا‌ف‌ظ‌ة‌,‌ن‌س‌ب‌ة‌ ‌م‌و‌ا‌ف‌ق‌,‌ن‌س‌ب‌ة‌ ‌غ‌ي‌ر‌ ‌م‌و‌ا‌ف‌ق‌,‌ع‌د‌د‌ ‌ا‌ل‌ن‌ا‌خ‌ب‌ي‌ن‌,‌ا‌ل‌أ‌ص‌و‌ا‌ت‌ ‌ا‌ل‌ص‌ح‌ي‌ح‌ة‌,‌ا‌ل‌أ‌ص‌و‌ا‌ت‌ ‌ا‌ل‌ب‌ا‌ط‌ل‌ة‌,‌ن‌س‌ب‌ة‌ ‌ا‌ل‌م‌ش‌ا‌ر‌ك‌ة‌,‌م‌و‌ا‌ف‌ق‌,‌غ‌ي‌ر‌ ‌م‌و‌ا‌ف‌ق‌‌ا‌ل‌ق‌ل‌ي‌و‌ب‌ي‌ة‌,60.0,40.0,"2,639,808","853,125","15,224",32.9,"512,055","341,070"‌ا‌ل‌ج‌ي‌ز‌ة‌,66.7,33.3,"4,383,701","1,493,092","24,105",34.6,"995,417","497,675"‌ا‌ل‌ق‌ا‌ه‌ر‌ة‌,43.2,56.8,"6,580,478","2,254,698","36,342",34.8,"974,371","1,280,327"‌ق‌ن‌ا‌,84.5,15.5,"1,629,713","364,509","6,743",22.8,"307,839","56,670"...

8.2Examples

GID,On Street,Species,Trim Cycle,Inventory Date1,ADDISON AV,Celtis australis,Large Tree Routine Prune,10/18/20102,EMERSON ST,Liquidambar styraciflua,Large Tree Routine Prune,6/2/2010

{"@type":"Table","url":"http://example.org/tree-ops.csv","tableSchema":{"columns":[{"titles":["GID"]},{"titles":["On Street"]},{"titles":["Species"]},{"titles":["Trim Cycle"]},{"titles":["Inventory Date"]}]}}

data.parse({"column-names":["GID","on_street","species","trim_cycle","inventory_date"],"datatypes":["string","string","string","string","date"],"formats":[null,null,null,null,"M/D/YYYY"]});

{"@type":"Table","url":"http://example.org/tree-ops.csv","tableSchema":{"columns":[{"name":"GID","datatype":"string"},{"name":"on_street","datatype":"string"},{"name":"species","datatype":"string"},{"name":"trim_cycle","datatype":"string"},{"name":"inventory_date","datatype":{"base":"date","format":"M/d/yyyy"}}]}}

{"@type":"Table","url":"http://example.org/tree-ops.csv","tableSchema":{"columns":[{"name":"GID","titles":"GID","datatype":"string"},{"name":"on_street","titles":"On Street","datatype":"string"},{"name":"species","titles":"Species","datatype":"string"},{"name":"trim_cycle","titles":"Trim Cycle","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","datatype":{"base":"date","format":"M/d/yyyy"}}]}}

{"@context":["http://www.w3.org/ns/csvw",{"@language":"en"}],"url":"tree-ops.csv","dc:title":"Tree Operations","dcat:keyword":["tree","street","maintenance"],"dc:publisher":{"schema:name":"Example Municipality","schema:url":{"@id":"http://example.org"}},"dc:license":{"@id":"http://opendefinition.org/licenses/cc-by/"},"dc:modified":{"@value":"2010-12-31","@type":"xsd:date"},"tableSchema":{"columns":[{"name":"GID","titles":["GID","Generic Identifier"],"dc:description":"An identifier for the operation on a tree.","datatype":"string","required":true},{"name":"on_street","titles":"On Street","dc:description":"The street that the tree is on.","datatype":"string"},{"name":"species","titles":"Species","dc:description":"The species of the tree.","datatype":"string"},{"name":"trim_cycle","titles":"Trim Cycle","dc:description":"The operation performed on the tree.","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","dc:description":"The date of the operation that was performed.","datatype":{"base":"date","format":"M/d/yyyy"}}],"primaryKey":"GID","aboutUrl":"#gid-{GID}"}}

GID,On Street,Species,Trim Cycle,Inventory Date1,ADDISON AV,"Celtis australis","Large Tree Routine Prune",10/18/20102,,"Liquidambar styraciflua","Large Tree Routine Prune",

#publisherCity of Palo Alto#updated12/31/2010#nameGIDon_streetspeciestrim_cycleinventory_date#datatypestringstringstringstringdate:M/D/YYYYGIDOn StreetSpeciesTrim CycleInventory Date1ADDISON AVCeltis australisLarge Tree Routine Prune10/18/20102EMERSON STLiquidambar styracifluaLarge Tree Routine Prune6/2/2010

{"delimiter":"\t","skipRows":4,"skipColumns":1,"commentPrefix":"#"}

{"@context":"http://www.w3.org/ns/csvw","url":"tree-ops.csv","dc:publisher":"City of Palo Alto","dc:updated":"12/31/2010","tableSchema":{"columns":[{"name":"GID","titles":"GID","datatype":"string",},{"name":"on_street","titles":"On Street","datatype":"string"},{"name":"species","titles":"Species","datatype":"string"},{"name":"trim_cycle","titles":"Trim Cycle","datatype":"string"},{"name":"inventory_date","titles":"Inventory Date","datatype":{"base":"date","format":"M/d/yyyy"}}]}}

Who,What,,Where,Organization,Sector,Subsector,Department,Municipality#org,#sector,#subsector,#adm1,#adm2UNICEF,Education,Teacher training,Chocó,QuidbóUNICEF,Education,Teacher training,Chocó,Bojayá

{"skipRows":1,"headerRowCount":2}

{"tableSchema":{"columns":[{"titles":["Organization","#org"]},{"titles":["Sector","#sector"]},{"titles":["Subsector","#subsector"]},{"titles":["Department","#adm1"]},{"titles":["Municipality","#adm2"]},]}}

{"tableSchema":{"columns":[{"name":"org","titles":#org" },{"name":"sector","titles":#sector" },{"name":"subsector","titles":#subsector" },{"name":"adm1","titles":#adm1" },{"name":"adm2","titles":#adm2" },]}}

B.1RFC 4180

[RFC4180] defines CSV with the following ABNF grammar:

file = [header CRLF] record *(CRLF record) [CRLF]header = name *(COMMA name)record = field *(COMMA field)name = fieldfield = (escaped / non-escaped)escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTEnon-escaped = *TEXTDATACOMMA = %x2CCR = %x0DDQUOTE =  %x22LF = %x0ACRLF = CR LFTEXTDATA =  %x20-21 / %x23-2B / %x2D-7E

Of particular note here are:

• The production forTEXTDATA indicates that only non-control ASCII characters are permitted within a CSV file. This restriction is routinely ignored in practice, and is impractical on the international web.
• Lines should be ended withCRLF. This makes it harder to produce CSV files on Unix-based systems where the usual line ending isLF.
• The header line is optional; aheader parameter on the media type indicates whether the header is present or not.
• Fields may be escaped by wrapping them in double quotes; any double quotes within the field must be escaped with two double quotes ("").

B.2Excel

Excel is a common tool for both creating and reading CSV documents, and therefore the CSV that it produces is a de facto standard.

B.3Google Spreadsheets

B.4CSV Files in a Tabular Data Package

Tabular Data Packages place the following restrictions on CSV files:

As a starting point, CSV files included in a Tabular Data Package package must conform to the RFC for CSV (4180 - Common Format and MIME Type for Comma-Separated Values (CSV) Files). In addition:

• File namesMUST end with.csv.

• FilesMUST be encoded as UTF-8.

• FilesMUST have a single header row. This rowMUST be the first row in the file.

  • Terminology: each column in the CSV file is termed afield and itsname is the string in that column in the header row.

  • ThenameMUST be unique amongst fields,MUST contain at least one character, andMUST conform to the character restrictions defined for thename property.

• Rows in the fileMUST NOT contain more fields than are in the header row (though they may contain less).

• Each fileMUST have an entry in thetables array in thedatapackage.json file.

• The resource metadataMUST include atableSchema attribute whose valueMUST be a validschema description.

• All fields in the CSV filesMUST be described in theschema description.

CSV files generated by different applications often vary in their syntax, e.g. use of quoting characters, delimiters, etc. To encourage conformance, CSV files in a CSV files in a Tabular Data PackageSHOULD:

• Use "," as field delimiters.
• UseCRLF (U+000D U+000A) orLF (U+000A) as line terminators.

If a CSV file does not follow these rules then its specific CSV dialectMUST be documented. The resource hash for the resource in thedatapackage.json descriptorMUST:

• Include adialect key that conforms to that described in theCSV Dialect Description Format.

Applications processing the CSV fileSHOULD read use thedialect of the CSV file to guide parsing.

D.1Changes since the candidate recommendation of 16 July 2015

• Usetext/tab-separated-values instead of the un-registeredtext/tsv.
• /.well-known/csvm has been registered at IANA

D.2Changes since the working draft of 16 April 2015

• Merging of metadata files has been removed as it was determined not to be necessary.
• Embedded metadata now used for compatibility check only, or as metadata if no other is found.
• Thetitles annotation has been added to rows, and a section added describing the way in which screen readers should announce rows and columns to users
• A Datatype description may have anid annotation to reference an external datatype definition in XSD, OWL, or some other format.
• Renamed the direction annotation totable direction.
• The built-in locations for locating metadata files were removed in favor of a site-wide configuration file, which uses the original values for file-specific and directory-specific metadata locations as the default value. Seesection5.3Default Locations and Site-wide Location Configuration.
• Thepattern for numeric types is now a number format pattern rather than a regular expression.

D.3Changes since the working draft of 08 January 2015

The document has undergone substantial changes since the last working draft. Below are some of the changes made:

• Describe all core annotations on agroup of tables,tables,columns,rows, anddatatypes.
• Removed theCore Tabular Data Model, as this can be derived from theAnnotated Tabular Data Model with an appropriatedialect descriptiondialect description.
• Describe the process oflocating metadata, using this to createembedded metadata,processingtabular data files, andcreating annotated tables.
• Move details ondatatypes andparsing cells. Datatypes include detailed formats for different datatypes.

E.1Normative references

[BCP47]

A. Phillips; M. Davis.Tags for Identifying Languages. September 2009. IETF Best Current Practice. URL:https://tools.ietf.org/html/bcp47

[BIDI]

Mark Davis; Aharon Lanin; Andrew Glass.Unicode Bidirectional Algorithm. 5 June 2014. Unicode Standard Annex #9. URL:http://www.unicode.org/reports/tr9/

[ECMASCRIPT]

ECMAScript Language Specification. URL:https://tc39.github.io/ecma262/

[ISO8601]

Representation of dates and times. International Organization for Standardization. 2004. ISO 8601:2004. URL:http://www.iso.org/iso/catalogue_detail?csnumber=40874

[JSON-LD]

Manu Sporny; Gregg Kellogg; Markus Lanthaler.JSON-LD 1.0. 16 January 2014. W3C Recommendation. URL:http://www.w3.org/TR/json-ld/

[RFC2119]

S. Bradner.Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL:https://tools.ietf.org/html/rfc2119

[RFC3968]

G. Camarillo.The Internet Assigned Number Authority (IANA) Header Field Parameter Registry for the Session Initiation Protocol (SIP). December 2004. Best Current Practice. URL:https://tools.ietf.org/html/rfc3968

[RFC4180]

Y. Shafranovich.Common Format and MIME Type for Comma-Separated Values (CSV) Files. October 2005. Informational. URL:https://tools.ietf.org/html/rfc4180

[RFC5785]

M. Nottingham; E. Hammer-Lahav.Defining Well-Known Uniform Resource Identifiers (URIs). April 2010. Proposed Standard. URL:https://tools.ietf.org/html/rfc5785

[UAX35]

Mark Davis; CLDR committee members.Unicode Locale Data Markup Language (LDML). 15 March 2013. Unicode Standard Annex #35. URL:http://www.unicode.org/reports/tr35/tr35-31/tr35.html

[UNICODE]

The Unicode Standard. URL:http://www.unicode.org/versions/latest/

[URI-TEMPLATE]

J. Gregorio; R. Fielding; M. Hadley; M. Nottingham; D. Orchard.URI Template. March 2012. Proposed Standard. URL:https://tools.ietf.org/html/rfc6570

[tabular-metadata]

Jeni Tennison; Gregg Kellogg.Metadata Vocabulary for Tabular Data. W3C Recommendation. URL:http://www.w3.org/TR/2015/REC-tabular-metadata-20151217/

[xmlschema11-2]

David Peterson; Sandy Gao; Ashok Malhotra; Michael Sperberg-McQueen; Henry Thompson; Paul V. Biron et al.W3C XML Schema Definition Language (XSD) 1.1 Part 2: Datatypes. 5 April 2012. W3C Recommendation. URL:http://www.w3.org/TR/xmlschema11-2/

E.2Informative references

[EBNF-NOTATION]

Tim Bray; Jean Paoli; C. Michael Sperberg-McQueen; Eve Maler; François Yergau.EBNF Notation. W3C Recommendation. URL:http://www.w3.org/TR/xml/#sec-notation

[RFC7111]

M. Hausenblas; E. Wilde; J. Tennison.URI Fragment Identifiers for the text/csv Media Type. January 2014. Informational. URL:https://tools.ietf.org/html/rfc7111

[UAX15]

Mark Davis; Ken Whistler.Unicode Normalization Forms. 31 August 2012. Unicode Standard Annex #15. URL:http://www.unicode.org/reports/tr15

[annotation-model]

Robert Sanderson; Paolo Ciccarese; Benjamin Young.Web Annotation Data Model. 15 October 2015. W3C Working Draft. URL:http://www.w3.org/TR/annotation-model/

[csv2json]

Jeremy Tandy; Ivan Herman.Generating JSON from Tabular Data on the Web. W3C Recommendation. URL:http://www.w3.org/TR/2015/REC-csv2json-20151217/

[csv2rdf]

Jeremy Tandy; Ivan Herman; Gregg Kellogg.Generating RDF from Tabular Data on the Web. W3C Recommendation. URL:http://www.w3.org/TR/2015/REC-csv2rdf-20151217/

[encoding]

Anne van Kesteren; Joshua Bell; Addison Phillips.Encoding. 20 October 2015. W3C Candidate Recommendation. URL:http://www.w3.org/TR/encoding/

[vocab-data-cube]

Richard Cyganiak; Dave Reynolds.The RDF Data Cube Vocabulary. 16 January 2014. W3C Recommendation. URL:http://www.w3.org/TR/vocab-data-cube/