Repository files navigation

This module can be used to replace keywords in sentences or extract keywords from sentences. It is based on theFlashText algorithm.

$ pip install flashtext

Documentation can be found atFlashText Read the Docs.

Extract keywords

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>># keyword_processor.add_keyword(<unclean name>, <standardised name>)>>> keyword_processor.add_keyword('Big Apple','New York')>>> keyword_processor.add_keyword('Bay Area')>>> keywords_found= keyword_processor.extract_keywords('I love Big Apple and Bay Area.')>>> keywords_found>>># ['New York', 'Bay Area']

Replace keywords

>>> keyword_processor.add_keyword('New Delhi','NCR region')>>> new_sentence= keyword_processor.replace_keywords('I love Big Apple and new delhi.')>>> new_sentence>>># 'I love New York and NCR region.'

Case Sensitive example

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor(case_sensitive=True)>>> keyword_processor.add_keyword('Big Apple','New York')>>> keyword_processor.add_keyword('Bay Area')>>> keywords_found= keyword_processor.extract_keywords('I love big Apple and Bay Area.')>>> keywords_found>>># ['Bay Area']

Span of keywords extracted

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_processor.add_keyword('Big Apple','New York')>>> keyword_processor.add_keyword('Bay Area')>>> keywords_found= keyword_processor.extract_keywords('I love big Apple and Bay Area.',span_info=True)>>> keywords_found>>># [('New York', 7, 16), ('Bay Area', 21, 29)]

Get Extra information with keywords extracted

>>>from flashtextimport KeywordProcessor>>> kp= KeywordProcessor()>>> kp.add_keyword('Taj Mahal', ('Monument','Taj Mahal'))>>> kp.add_keyword('Delhi', ('Location','Delhi'))>>> kp.extract_keywords('Taj Mahal is in Delhi.')>>># [('Monument', 'Taj Mahal'), ('Location', 'Delhi')]>>>#NOTE: replace_keywords feature won't work with this.

No clean name for Keywords

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_processor.add_keyword('Big Apple')>>> keyword_processor.add_keyword('Bay Area')>>> keywords_found= keyword_processor.extract_keywords('I love big Apple and Bay Area.')>>> keywords_found>>># ['Big Apple', 'Bay Area']

Add Multiple Keywords simultaneously

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_dict= {>>>"java": ["java_2e","java programing"],>>>"product management": ["PM","product manager"]>>> }>>># {'clean_name': ['list of unclean names']}>>> keyword_processor.add_keywords_from_dict(keyword_dict)>>># Or add keywords from a list:>>> keyword_processor.add_keywords_from_list(["java","python"])>>> keyword_processor.extract_keywords('I am a product manager for a java_2e platform')>>># output ['product management', 'java']

To Remove keywords

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_dict= {>>>"java": ["java_2e","java programing"],>>>"product management": ["PM","product manager"]>>> }>>> keyword_processor.add_keywords_from_dict(keyword_dict)>>>print(keyword_processor.extract_keywords('I am a product manager for a java_2e platform'))>>># output ['product management', 'java']>>> keyword_processor.remove_keyword('java_2e')>>># you can also remove keywords from a list/ dictionary>>> keyword_processor.remove_keywords_from_dict({"product management": ["PM"]})>>> keyword_processor.remove_keywords_from_list(["java programing"])>>> keyword_processor.extract_keywords('I am a product manager for a java_2e platform')>>># output ['product management']

To check Number of terms in KeywordProcessor

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_dict= {>>>"java": ["java_2e","java programing"],>>>"product management": ["PM","product manager"]>>> }>>> keyword_processor.add_keywords_from_dict(keyword_dict)>>>print(len(keyword_processor))>>># output 4

To check if term is present in KeywordProcessor

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_processor.add_keyword('j2ee','Java')>>>'j2ee'in keyword_processor>>># output: True>>> keyword_processor.get_keyword('j2ee')>>># output: Java>>> keyword_processor['colour']='color'>>> keyword_processor['colour']>>># output: color

Get all keywords in dictionary

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_processor.add_keyword('j2ee','Java')>>> keyword_processor.add_keyword('colour','color')>>> keyword_processor.get_all_keywords()>>># output: {'colour': 'color', 'j2ee': 'Java'}

For detecting Word Boundary currently any character other than this \w [A-Za-z0-9_] is considered a word boundary.

To set or add characters as part of word characters

>>>from flashtextimport KeywordProcessor>>> keyword_processor= KeywordProcessor()>>> keyword_processor.add_keyword('Big Apple')>>>print(keyword_processor.extract_keywords('I love Big Apple/Bay Area.'))>>># ['Big Apple']>>> keyword_processor.add_non_word_boundary('/')>>>print(keyword_processor.extract_keywords('I love Big Apple/Bay Area.'))>>># []

$ git clone https://github.com/vi3k6i5/flashtext$ cd flashtext$ pip install pytest$ python setup.py test

$ git clone https://github.com/vi3k6i5/flashtext$ cd flashtext/docs$ pip install sphinx$ make html$ # open _build/html/index.html in browser to view it locally

It's a custom algorithm based onAho-Corasick algorithm andTrie Dictionary.

Time taken by FlashText to find terms in comparison to Regex.

Time taken by FlashText to replace terms in comparison to Regex.

Link to code for benchmarking theFind Feature andReplace Feature.

The idea for this library came from the followingStackOverflow question.

The original paper published onFlashText algorithm.

@ARTICLE{2017arXiv171100046S,   author = {{Singh}, V.},    title = "{Replace or Retrieve Keywords In Documents at Scale}",  journal = {ArXiv e-prints},archivePrefix = "arXiv",   eprint = {1711.00046}, primaryClass = "cs.DS", keywords = {Computer Science - Data Structures and Algorithms},     year = 2017,    month = oct,   adsurl = {http://adsabs.harvard.edu/abs/2017arXiv171100046S},  adsnote = {Provided by the SAO/NASA Astrophysics Data System}}

The article published onMedium freeCodeCamp.

• Issue Tracker:https://github.com/vi3k6i5/flashtext/issues
• Source Code:https://github.com/vi3k6i5/flashtext/

The project is licensed under the MIT license.