pySBD: Python Sentence Boundary Disambiguation (SBD)

Highlights

Install

pip install pysbd

Usage

importpysbdtext="My name is Jonas E. Smith. Please turn to p. 55."seg=pysbd.Segmenter(language="en",clean=False)print(seg.segment(text))# ['My name is Jonas E. Smith.', 'Please turn to p. 55.']

importspacyfrompysbd.utilsimportPySBDFactorynlp=spacy.blank('en')# explicitly adding component to pipeline# (recommended - makes it more readable to tell what's going on)nlp.add_pipe(PySBDFactory(nlp))# or you can use it implicitly with keyword# pysbd = nlp.create_pipe('pysbd')# nlp.add_pipe(pysbd)doc=nlp('My name is Jonas E. Smith. Please turn to p. 55.')print(list(doc.sents))# [My name is Jonas E. Smith., Please turn to p. 55.]

Contributing

Citation

@inproceedings{sadvilkar-neumann-2020-pysbd,    title = "{P}y{SBD}: Pragmatic Sentence Boundary Disambiguation",    author = "Sadvilkar, Nipun  and      Neumann, Mark",    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",    month = nov,    year = "2020",    address = "Online",    publisher = "Association for Computational Linguistics",    url = "https://www.aclweb.org/anthology/2020.nlposs-1.15",    pages = "110--114",    abstract = "We present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter which we ported to Python with additional improvements and functionality. PySBD passes 97.92{\%} of the Golden Rule Set examplars for English, an improvement of 25{\%} over the next best open source Python tool.",}

Credit