Repository files navigation

This is a part of the Parallel Universal Dependencies (PUD) treebanks createdfor theCoNLL 2017 shared task on Multilingual Parsing from Raw Text toUniversal Dependencies.

There are1000 sentences in each language, always in the same order. (The sentencealignment is 1-1 but occasionally a sentence-level segment actually consistsof two real sentences.) The sentences are taken from the news domain (sentenceid starts in ‘n’) and from Wikipedia (sentence id starts with ‘w’). There areusually only a few sentences from each document, selected randomly, notnecessarily adjacent. The digits on the second and third position in thesentence ids encode the original language of the sentence. The first 750sentences are originally English (01). The remaining 250 sentences areoriginally German (02), French (03), Italian (04) or Spanish (05) and theywere translated to other languages via English. Translation into German,French, Italian, Spanish, Arabic, Hindi, Chinese, Indonesian, Japanese,Korean, Portuguese, Russian, Thai and Turkish has been provided by DFKI andperformed (except for German) by professional translators. Then the data hasbeen annotated morphologically and syntactically by Google according to Googleuniversal annotation guidelines; finally, it has been converted by members ofthe UD community to UD v2 guidelines.

Additional languages have been provided (both translation and native UD v2annotation) by other teams: Czech by Charles University, Finnish by Universityof Turku and Swedish by Uppsala University.

The entire treebank is labeled as test set (and was used for testing in theshared task). If it is used for training in future research, the users shouldemploy ten-fold cross-validation.

In May 2020, we introduced the same coversion method used in UD_Japanese GSD v2.6 for UD_Japanese PUD.

• 2020-05- v2.6

  • Update for v2.6. Introduce the same conversion method used in UD-Japanese GSD v2.6

• 2017-11-15 v2.1

  • First official release after it was used as a surprise dataset in theCoNLL 2017 shared task.

A description of how the treebanks were generated can be found in:

Universal Dependency Annotation for Multilingual ParsingRyan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg,Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang,Oscar Tackstrom, Claudia Bedini, Nuria Bertomeu Castello and Jungmee LeeProceedings of ACL 2013

A more detailed description of each relation type in our harmonized scheme isincluded in the file universal-guidelines.pdf.

Each file is formatted according to the CoNLL 2006/2007 guidelines:

http://ilk.uvt.nl/conll/#dataformat

The treebank annotations use basic Stanford Style dependencies, modifiedminimally to be sufficient for each language and be maximally consistent acrosslanguages. The original English Stanford guidelines can be found here:

http://nlp.stanford.edu/software/dependencies_manual.pdf

In the CoNLL file format there is a coarse part-of-speech tag field (4) and afine-grained part-of-speech tag field (5). In this data release, we use thecoarse field to store the normalized universal part-of-speech tags that areconsistent across languages. The fine-grained field contains potentially richerpart-of-speech information depending on the language, e.g., a richer tagrepresentation for clitics.

We will distinguish between two portions of the data:

1. The underlying text for sentences and corresponding translations. This data Google asserts no ownership over and no copyright over. The source of the texts is randomly selected Wikipedia (www.wikipedia.org) sentences. Some or all of these sentences may be copyrighted in some jurisdictions. Where copyrighted, Google collected these sentences under exceptions to copyright or implied license rights. GOOGLE MAKES THEM AVAILABLE TO YOU under CC-BY-SA 3.0, WITHOUT ANY WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED.See attached LICENSE file for the text of CC BY-SA 3.0.

2. The annotations -- part-of-speech tags and dependency annotations. GOOGLE MAKES THEM AVAILABLE TO YOU 'AS IS', WITHOUT ANY WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED.

slav@google.comtbd

We are greatful to the many people who made this dataset possible:Fernando Pereira, Hans Uszkoreit, Aljoscha Burchardt, Vivien Macketanz,Ali Elkahky, Abhijit Barde, Tolga Kayadelen, ...