Repository files navigation

BIG FAT WARNING: This codebase is now deprecated and has been replaced by our brand-newskweak framework, please check it out!

Source code associated with the paper "Named Entity Recognition without Labelled Data: a Weak Supervision Approach" accepted to ACL 2020.

You should first make sure that the following Python packages are installed:

• spacy (version >= 2.2)
• hmmlearn
• snips-nlu-parsers
• pandas
• numba
• scikit-learn

You should also install theen_core_web_sm anden_core_web_md models in Spacy.

To run the neural models inner.py, you need also needpytorch,cupy,keras andtensorflow installed.

To run the baselines, you will also need to havesnorkel installed.

Finally, you also need to download the following files and add them to thedata directory:

• conll2003_spacy.tar.gz (unpack the archive in the same directory)
• BTC_spacy.tar.gz (same)
• SEC_spacy.tar.gz (same)
• wikidata.json
• wikidata_small.json
• crunchbase.json
• conll2003.docbin

You should first convert your corpus to SpacyDocBin format.

Then, to run all labelling functions on your corpus, you can simply:

import annotationsannotator = annotations.FullAnnotator().add_all()annotator.annotate_docbin('path_to_your_docbin_corpus')

You can then estimate an HMM model that aggregates all sources:

import labellinghmm = labelling.HMMAnnotator()hmm.train('path_to_your_docbin_corpus')

And run it on your corpus to get the aggregated labels:

hmm.annotate_docbin('path_to_your_docbin_corpus')

More detailed instructions with a step-by-step example are available in the Jupyter NotebookWeak Supervision.ipynb. Don't forget to run it using Jupyter to get the visualisation for the NER annotations.