Please refer to theglossary forthis document.
Copyright ©2009W3C® (MIT,ERCIM,Keio), All Rights Reserved. W3Cliability,trademarkanddocumentuse rules apply.
Every day, governments and government agencies publish more data on theInternet. Sharing this data enables greater transparency; delivers moreefficient public services; and encourages greater public and commercial use andre-use of government information. Some governments have even created catalogsor portals (such asdata.gov) to make it easyfor the public to find and use this data.
Although the reasons may vary, the logistics and practicalities of openinggovernment data are the same. To help governments open and share their data,the W3C eGov Interest Group has developed the following guidelines. Thesestraightforward steps emphasize standards and methodologies to encouragepublication of government data, allowing the public to use this data in new andinnovative ways.
This section describes the status of this document at the time of itspublication. Other documents may supersede this document. A list of current W3Cpublications and the latest revision of this technical report can be found intheW3C technical reports index athttp://www.w3.org/TR/.
TheW3C eGovernment InterestGroupis pleased to offer this First Public Working Draft which brieflyexplains how to publish government data on the Web, based on the our gatheredexperience. We encourage anyone with additional experience to join the group orsend written comment to help with the next version.
TheW3C eGovernment InterestGroup views this as an evolving document and seeks public feedback on thisWorking Draft. Please send your comments topublic-egov-ig@w3.org (archives). Ifpossible, please offer specific changes to the text that would address yourconcern. You may also wish to check theWiki Version ofthis document and see if the relevant text has already been updated.
Publication as a Working Draft does not imply endorsement by the W3CMembership. This is a draft document and may be updated, replaced or obsoletedby other documents at any time. It is inappropriate to cite this document asother than work in progress.
This document was produced by a group operating under the5 February 2004 W3CPatent Policy. The group does not expect this document to become a W3CRecommendation. W3C maintains apublic list of any patentdisclosures made in connection with the deliverables of the group; thatpage also includes instructions for disclosing a patent. An individual who hasactual knowledge of a patent which the individual believes containsEssentialClaim(s) must disclose the information in accordance withsection6 of the W3C Patent Policy.
Step 1:The quickest and easiest way to makedata available on the Internet is to publish the data in its raw form (e.g., anXML file of polling data from past elections). However, the data should bewell-structured. Structure allows others to successfully make automated use ofthe data. Well-known formats or structures include XML, RDF and CSV. Formatsthat only allow the data to be seen, rather than extracted (for example,pictures of the data), are not useful and should be avoided.
Step 2:Create an online catalog of the rawdata (complete with documentation) so people can discover what has beenposted.
These raw datasets should be reliably structured and documented, otherwisetheir usefulness is negligible. Most governments already have mechanisms inplace to create and store data (e.g., Excel, Word, and other software-specificfile formats).
Posting raw data, with an online catalog, is a great starting point, andreflects the next-step evolution of the Internet - "website as fileserver".
Step 3:Make the data bothhuman- and machine-readable:
These steps will help the public to easily find, use, cite and understandthe data. The data catalog should explain any rules or regulations that must befollowed in the use of the dataset. Also, the data catalog itself is considered"data" and should be published as structured data, so that third parties canextract data about the datasets. Thoroughly document the parts of the webpage, using valid XHTML, and choose easily patterned and discoverable URLs forthe pages. Also syndicate the data for the catalog (using formats such as RSS)to quickly and easily advertise new datasets upon publication.
The ability to identify things by a URI/URL is the foundation of theInternet. Best practices for "open government data" depend on applying thearchitecture described in Architectureof the World Wide Web, Volume 1 [WEBARCH]. If you give permanent, patterned and/ordiscoverable URI/URLs to your data, processes and people will be able to findand use it more easily. URI/URLs can be used in databases and metadata asuniversal, unique identifiers (e.g. by appending a serial number or otherinternal naming system to a domain:
Use internal IDs to identify specific data for machine re-use (e.g.,http://www.census.gov/main/www/popclock.html). Byusing URIs, and incorporating internal links inside large files or documents,people can authoritatively cite the information on their own websites.
Without documentation, data is not very useful. When possible, you shoulduse industry standards such as those based on XML/RDF, as they tend to beself-documenting. Create a web page with a plain language description of thedataset to help search engines find the data, so people can use it. Thedescription should explain how to find the pages and/or files, and describe thecontents of the files. At a minimum, document the title, description,publication date and authoritative source for the data. Use clear language, andlink to related information to help put the data into context. When possible,also fully document the description of each datum and the data standard and/ordata validation (e.g., if datum is a temperature, also explain the relatedstandard, such as Celsius, Fahrenheit or Kelvin). As standards emerge fordescribing the datasets, take advantage of them. Document search tools andReSTful methods of obtaining the data when applicable.
The next step beyond raw data is fully linked data. In other words, raw datacan only be linked to, there are no links back out - but linked data containslinks out to other data and documentation. There is a spectrum from raw tofully linked data, and standards such as RDF that can help you link yourdata.
Here are four expectations for the linked data web:
Using URIs as described provides for a rich and easily connected web of datathat is easier for the public to search and use. However, many databases donot build URIs, but instead use internally derived unique identifiers. To linkthis data into the larger web, we need to use systems that can anticipate orbuild the URIs prior to publishing the data.
Preserving the data we publish is a top concern for governments. How can weensure that data is findable and can be referenced for as long as people needit? People may continue to cite older data at the original URI decades afterit was initially published.
To maintain the history of a dataset, think about how to integrate new URIsfor upgraded and new datasets, and structure your URIs accordingly. Useversioning of datasets so people can cite and link to present and pastversions. New and upgraded datasets can refer back to original datasets. Ifneeded, provide tools to translate older data. Carefully document the changesbetween versions, and embed the version number/indicator within the data ifpossible. Consider the file format carefully, and opt for open standards thatare not software specific.
See theReferences section formore information on standards and best practices for data preservation.
If you really want to help people discover and explore the data you arepublishing, there are some useful W3C standards which can help. It ispossible to make data human-readable by using XSLT for XML- and RDF-basedformats. Make web pages machine-readable by embedding semantic information inthe HTML, using RDFa and/or Microformats. (Note that search engines can takeadvantage of embedded semantic information, helping more people to find yourdata.) If it is too difficult to append XSL or CSS styles to old data, it isstill possible to document how to transform the data with a specific or genericXSLT engines (for example, seehttp://www.xmldatasets.net).
External parties can create new and exciting interfaces that may not beobvious to the data publishers. For that reason, do not compromise theintegrity of the data to create flashy interfaces. If you must create aninterface, then publish the data separate from the interface, and ensureexternal parties have direct access to the raw data, so they can build theirown interfaces if they wish.
Additionally, since all web documents using (X)HTML, XML and RDF can be usedas an object database or ReSTful API, the public can create software, Webapplications and mash-ups that use the datasets, and link back to theauthoritative version on the government web site. Two W3C standard languagesthat make that possible are XQuery and SPARQL. Once your data is published,governments themselves can also create a user-friendly, transparent interfacethat allows users to play with and refine the information in which they areinterested.
Having a unique identifier for every registered company or school (object)is as important as having information about those companies or schools. Inaddition to using open industry standards, an effort should be made to catalogall reference-able government entities, officers and objects. The URIs canthen be used within all of the data. This aids in discoverability, improvesmetadata, and ensures authenticity.
All data that can be shared with the public should be opened for publicdissemination. Data should be published in compliance with applicable laws andregulations, and only after addressing issues of security and privacy.
First, publish all data that is already available in other formats (such asprint), or already being collected and shared with the public in other ways. Publish both highly structured data and text documents. Publish laws,regulations and codes, and public documents related to government agencymission, vision, values and goals.
There are many standards for such documents (such as AIIM's emergingStrategy Markup Language (StratML), XBRL and others). Use of standards enablesstakeholders to easily discover and provide feedback on items of interest tothem. Creating and publishing new documents in various standard formats iseasier than fixing after the fact.
As you create your documentation for the data, include and reference XMLSchemas, DTDs and taxonomies. Explain how to discover all of thefiles/documents in a collection. Use syndication standards and tools toannounce additions or changes. Provide complete documentation to enableautomatically-generated data dictionaries and provide query services, whichwill make it easier for the public to find your documents and datasets.
There are many different data formats, but which will work best with yourdata? The primary format for human-readable data is (X)HTML.
Raw data is more likely to be produced using formats customized to thespecific data, the tools used, or industry standards. The W3C has pioneeredXML and RDF, which allow for excellent manipulation and standardized tool sets.RDF and XML files can be accessed like databases, using SPARQL, XQuery,JavaScript and many other computer languages. When possible, use establishedopen standards, and tools that allow easy and efficient production andpublishing of the data. See theReferences sectionfor a list of current tools. Also keep in mind the power of linked data.
Once you've made your data available, be sure to document clearly any legalor regulatory restrictions on the use of the data. In many cases, there aredefined standards to insert copyright and/or licensing information right intothe data (see metadata standards, such as Dublin Core). It is very important tobe clear regarding the expected uses, credits, limitations and responsibilitiesthat particular parties have in regard to working with and providing thedata.
We still have much to learn about the best ways for governments to open uptheir data. The W3C eGov Interest Group plans to publish relevant use cases toillustrate working examples of current thinking and best practices. The W3Ctechnologies and approaches described in this document can be implementedquickly, and often with relatively little cost. If you want to learn more,there are a number of communities that can help. If you are a governmentagency, think about joining the W3C and getting involved in e-Governmentactivities. You can connect with people from other governments who are facingsimilar issues and sharing their experiences. Even if you don't join the W3C,please let us know how you are getting on. We can help.