Repository files navigation

Distribution vector representations of words and Wikipedia entities appeared in Japanese Wikipedia. The data is created based on the specification of "Japanese-Wikipedia Entity Vectors (in Japanese)" released by Masatoshi Suzuki of Inui-Suzuki Laboratory in Tohoku University.

Here are examples withGensim library.Segmented words are registered as word2vec words.

>>> from gensim.models import KeyedVectors>>> w2v_model = KeyedVectors.load_word2vec_format('entity_vector.model.bin', binary=True, unicode_errors='ignore')>>> print(w2v_model.most_similar(['こと']))[('事', 0.9618349075317383), ('もの', 0.7732754349708557), ('ため', 0.7425500154495239), ... ]

Entities that link to Wikipedia articles are registered as word2vec words with square brackets on either side in the format[entity].

>>> from gensim.models import KeyedVectors>>> w2v_model = KeyedVectors.load_word2vec_format('entity_vector.model.bin', binary=True, unicode_errors='ignore')>>> print(w2v_model.most_similar(['[72時間ホンネテレビ]']))[('[AbemaPrime]', 0.6929168701171875), ('[原宿AbemaNews]', 0.6854027509689331), ...]

Due to the large file size, files are uploaded to Dropbox.

https://www.dropbox.com/sh/601gucye55nr1gq/AABekRrz4IYtp2n0_lTrKsGma

By decompressing an archive with the following tar command, 5 files are created.

tar xvJf jawikivec.[dictionary].yyyyMMdd.tar.xz

An output file saved in binary word2vec format.

An output file saved in text word2vec format.

A tsv file containing terms appeared in a plain text and corresponding Wikipedia entities. More details are described inJapanese-Wikipedia Wikification Corpus.

AYAML-formatted file to store version information for referred dictionary and corpus.

Document regarding licensing.

Distribution vector representations are created in the following settings.