Repository files navigation

Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It providessimple,performant &accurate NLP annotations for machine learning pipelines thatscale easily in a distributed environment.

Spark NLP comes with100000+ pretrainedpipelines andmodels in more than200+ languages.It also offers tasks such asTokenization,Word Segmentation,Part-of-Speech Tagging, Word and SentenceEmbeddings,Named Entity Recognition,Dependency Parsing,Spell Checking,Text Classification,Sentiment Analysis,Token Classification,Machine Translation (+180 languages),Summarization,Question Answering,Table Question Answering,Text Generation,Image Classification,Image to Text (captioning),Automatic Speech Recognition,Zero-Shot Learning, and many moreNLP tasks.

Spark NLP is the only open-source NLP library inproduction that offers state-of-the-art transformers such asBERT,CamemBERT,ALBERT,ELECTRA,XLNet,DistilBERT,RoBERTa,DeBERTa,XLM-RoBERTa,Longformer,ELMO,Universal Sentence Encoder,Llama-2,M2M100,BART,Instructor,E5,Google T5,MarianMT,OpenAI GPT2,Vision Transformers (ViT),OpenAI Whisper,Llama,Mistral,Phi,Qwen2, and many more not only toPython andR, but also toJVM ecosystem (Java,Scala, andKotlin) atscale by extendingApache Spark natively.

Spark NLP provides easy support for importing models from various popular frameworks:

• TensorFlow
• ONNX
• OpenVINO
• Llama.cpp (GGUF)

This wide range of support allows you to seamlessly integrate models from different sources into your Spark NLP workflows, enhancing flexibility and compatibility with existing machine learning ecosystems.

Take a look at our official Spark NLP page:https://sparknlp.org/ for userdocumentation and examples

• Text Preprocessing
• Parsing and Analysis
• Sentiment and Classification
• Embeddings
• Classification and Question Answering
• Machine Translation and Generation
• Image and Speech
• Integration and Interoperability (ONNX, OpenVINO)
• Pre-trained Models (36000+ in +200 languages)
• Multi-lingual Support

This is a quick example of how to use a Spark NLP pre-trained pipeline in Python and PySpark:

$ java -version# should be Java 8 or 11 (Oracle or OpenJDK)$ conda create -n sparknlp python=3.7 -y$ conda activate sparknlp# spark-nlp by default is based on pyspark 3.x$ pip install spark-nlp==6.2.3 pyspark==3.3.1

In Python console or JupyterPython3 kernel:

# Import Spark NLPfromsparknlp.baseimport*fromsparknlp.annotatorimport*fromsparknlp.pretrainedimportPretrainedPipelineimportsparknlp# Start SparkSession with Spark NLP# start() functions has 3 parameters: gpu, apple_silicon, and memory# sparknlp.start(gpu=True) will start the session with GPU support# sparknlp.start(apple_silicon=True) will start the session with macOS M1 & M2 support# sparknlp.start(memory="16G") to change the default driver memory in SparkSessionspark=sparknlp.start()# Download a pre-trained pipelinepipeline=PretrainedPipeline('explain_document_dl',lang='en')# Your testing datasettext="""The Mona Lisa is a 16th century oil painting created by Leonardo.It's held at the Louvre in Paris."""# Annotate your testing datasetresult=pipeline.annotate(text)# What's in the pipelinelist(result.keys())Output: ['entities','stem','checked','lemma','document','pos','token','ner','embeddings','sentence']# Check the resultsresult['entities']Output: ['Mona Lisa','Leonardo','Louvre','Paris']

For more examples, you can visit our dedicatedexamples to showcase all Spark NLP use cases!

This is a cheatsheet for corresponding Spark NLP Maven package to Apache Spark / PySpark major version:

NOTE:M1/M2 andAArch64 are underexperimental support. Access and support to these architectures are limited by thecommunity and we had to build most of the dependencies by ourselves to make them compatible. We support these twoarchitectures, however, they may not work in some environments.

For a quick example of using pipelines and models take a look at our officialdocumentation

Spark NLP6.2.3 has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x

Find out more aboutSpark NLP versions from ourrelease notes.

Find out more about 4.xSparkNLP versions in our officialdocumentation

Spark NLP 6.2.3 has been tested and is compatible with the following runtimes:

We are compatible with older runtimes. For a full list check databricks support in our officialdocumentation

Spark NLP 6.2.3 has been tested and is compatible with the following EMR releases:

We are compatible with older EMR releases. For a full list check EMR support in our officialdocumentation

Full list ofAmazon EMR 6.x releasesFull list ofAmazon EMR 7.x releases

NOTE: The EMR 6.1.0 and 6.1.1 are not supported.

To install spark-nlp packages through command line followthese instructions from our official documentation

Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x versions. Our packages aredeployed to Maven central. To add any of our packages as a dependency in your application you can followthese instructionsfrom our official documentation.

If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in yourprojectsSpark NLP Starter

Spark NLP supports Python 3.7.x and above depending on your major PySpark version.Check all available installations for Python in our officialdocumentation

To compile the jars from source followthese instructions from our official documentation

For detailed instructions on how to use Spark NLP on supported platforms, please refer to our official documentation:

Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet.Please checkthese instructions from our official documentationto use Spark NLP offline.

You can change Spark NLP configurations via Spark properties configuration.Please checkthese instructions from our official documentation.

In Spark NLP we can define S3 locations to:

• Export log files of training models
• Store tensorflow graphs used inNerDLApproach

Please checkthese instructions from our official documentation.

Need moreexamples? Check out our dedicatedSpark NLP Examplesrepository to showcase all Spark NLP use cases!

Also, don't forget to checkSpark NLP in Action built by Streamlit.

Check our Articles and Videos page here

We have published apaper that you can cite forthe Spark NLP library:

@article{KOCAMAN2021100058,title ={Spark NLP: Natural language understanding at scale},journal ={Software Impacts},pages ={100058},year ={2021},issn ={2665-9638},doi ={https://doi.org/10.1016/j.simpa.2021.100058},url ={https://www.sciencedirect.com/science/article/pii/S2665963.2.300063},author ={Veysel Kocaman and David Talby},keywords ={Spark, Natural language processing, Deep learning, Tensorflow, Cluster},abstract ={Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}    }}

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas,and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

We appreciate any sort of contributions:

• ideas
• feedback
• documentation
• bug reports
• NLP training and testing corpora
• Development and testing

Clone the repo and submit your pull-requests! Or directly create issues in this repo.

http://johnsnowlabs.com