Grammar is one of the main building blocks of language. Each human language, and programming language for that matter, has a set of rules that every person speaking it must follow, otherwise risking not being understood. These grammatical rules can be uncovered using NLP and are useful for extracting data from sentences. For example, using information about the grammatical structure of text, we can parse out subjects, objects, and relations betweendifferent entities.

In this chapter, you will learn how to use different packages to reveal the grammatical structure of words and sentences, as well as extract certain parts of sentences. These are the topics covered inthis chapter:

• Counting nouns – plural andsingular nouns
• Getting thedependency parse
• Extractingnoun chunks
• Extracting the subjects and objects ofthe sentence
• Finding patterns in text usinggrammatical information

Please follow the installation requirements given inChapter 1 to run the notebooks inthis chapter.

In this recipe, we will do two things: determine whether a noun is plural or singular and turn plural nouns into singular, andvice versa.

You might need these two things for a variety of tasks. For example, you might want to count the word statistics, and for that, you most likely need to count the singular and plural nouns together. In order to count the plural nouns together with singular ones, you need a way to recognize that a word is pluralor singular.

Getting ready

To determine whether a noun is singular or plural, we will usespaCy via two different methods: by looking at thedifference between the lemma and the actual word and by looking at themorph attribute. To inflect these nouns, or turn singular nouns into plural or vice versa we will use thetextblob package. We will also see how to determine the noun’s number using GPT-3 through the OpenAI API. The code for this section is locatedathttps://github.com/PacktPublishing/Python-Natural-Language-Processing-Cookbook-Second-Edition/tree/main/Chapter02.

How to do it…

We will first usespaCy’s lemma information to infer whether a noun is singular or plural. Then, we will use themorph attribute ofToken objects. We will then create a function that uses one of those methods. Finally, we will use GPT-3.5 to find out the numberof nouns:

1. Run the code in the file and languageutility notebooks. If you run into an error saying that the small or large models do not exist, you need to open thelang_utils.ipynb file, uncomment, and run the statement that downloadsthe model:

   %run -i "../util/file_utils.ipynb"%run -i "../util/lang_utils.ipynb"

2. Initialize thetext variable and process it using thespaCy small model to get the resultingDoc object:

   text = "I have five birds"doc = small_model(text)

3. In this step, we loop through theDoc object. For each token in the object, we check whether it’s a noun and whether the lemma is the same as the word itself. Since the lemma is the basic form of the word, if the lemma is different from the word, that tokenis plural:

   for token in doc:    if (token.pos_ == "NOUN" and token.lemma_ != token.text):        print(token.text, "plural")

   The result should beas follows:

   birds plural

4. Now, we will check the number of a noun using a different method: themorph features of aToken object. Themorph features are the morphological features of a word, such as number, case, and so on. Since we know that token3 is a noun, we directly access themorph features and get theNumber to get the same resultas previously:

   doc = small_model("I have five birds.")print(doc[3].morph.get("Number"))

   Here isthe result:

   ['Plur']

5. In this step, we prepare to define a function that returns a tuple,(noun, number). In order to better encode the noun number, we use anEnum class that assigns numbers to different values. We assign1 to singular and2 to plural. Once we create the class, we can directly refer to the noun number variables asNoun_number.SINGULARandNoun_number.PLURAL:

   class Noun_number(Enum):    SINGULAR = 1    PLURAL = 2

6. In this step, we define the function. It takes as input the text, thespaCy model, and the method of determining the noun number. The two methods arelemma andmorph, the same two methods we used insteps 3 and4, respectively. The function outputs a list of tuples, each of the format(<noun text>, <noun number>), where the noun number is expressed using theNoun_number class defined instep 5:

   def get_nouns_number(text, model, method="lemma"):    nouns = []    doc = model(text)    for token in doc:        if (token.pos_ == "NOUN"):            if method == "lemma":                if token.lemma_ != token.text:                    nouns.append((token.text,                         Noun_number.PLURAL))                else:                    nouns.append((token.text,                        Noun_number.SINGULAR))            elif method == "morph":                if token.morph.get("Number") == "Sing":                    nouns.append((token.text,                        Noun_number.PLURAL))                else:                    nouns.append((token.text,                        Noun_number.SINGULAR))    return nouns

7. We can use the preceding function and see its performance with differentspaCy models. In this step, we usethe smallspaCy model with the function we just defined. Usingboth methods, we see that thespaCy model gets the number of the irregular noungeese incorrectly:

   text = "Three geese crossed the road"nouns = get_nouns_number(text, small_model, "morph")print(nouns)nouns = get_nouns_number(text, small_model)print(nouns)

   The result should beas follows:

   [('geese', <Noun_number.SINGULAR: 1>), ('road', <Noun_number.SINGULAR: 1>)][('geese', <Noun_number.SINGULAR: 1>), ('road', <Noun_number.SINGULAR: 1>)]

8. Now, let’s do the same using the large model. If you have not yet downloaded the large model, do so by running the first line. Otherwise, you can comment it out. Here, we see that although themorph method still incorrectly assigns singular togeese, thelemma method provides thecorrect answer:

   !python -m spacy download en_core_web_lglarge_model = spacy.load("en_core_web_lg")nouns = get_nouns_number(text, large_model, "morph")print(nouns)nouns = get_nouns_number(text, large_model)print(nouns)

   The result should beas follows:

   [('geese', <Noun_number.SINGULAR: 1>), ('road', <Noun_number.SINGULAR: 1>)][('geese', <Noun_number.PLURAL: 2>), ('road', <Noun_number.SINGULAR: 1>)]

9. Let’s now use GPT-3.5 to get the nounnumber. In the results, we see that GPT-3.5 gives us an identicalresult and correctly identifies both the number forgeese and the numberforroad:

   from openai import OpenAIclient = OpenAI(api_key=OPEN_AI_KEY)prompt="""Decide whether each noun in the following text is singular or plural.Return the list in the format of a python tuple: (word, number). Do not provide any additional explanations.Sentence: Three geese crossed the road."""response = client.chat.completions.create(    model="gpt-3.5-turbo",    temperature=0,    max_tokens=256,    top_p=1.0,    frequency_penalty=0,    presence_penalty=0,    messages=[        {"role": "system", "content": "You are a helpful             assistant."},        {"role": "user", "content": prompt}    ],)print(response.choices[0].message.content)

   The result should beas follows:

   ('geese', 'plural')('road', 'singular')

There’s more…

We can also change the nounsfrom plural to singular, and vice versa. We will use thetextblob package for that. The package should be installed automatically via thePoetryenvironment:

1. Import theTextBlob class fromthe package:

   from textblob import TextBlob

2. Initialize a list of text variables and process them using theTextBlob class via alist comprehension:

   texts = ["book", "goose", "pen", "point", "deer"]blob_objs = [TextBlob(text) for text in texts]

3. Use thepluralize function of the object to get the plural. This function returns a list and we access its first element. Printthe result:

   plurals = [blob_obj.words.pluralize()[0]     for blob_obj in blob_objs]print(plurals)

   The result should beas follows:

   ['books', 'geese', 'pens', 'points', 'deer']

4. Now, we will do the reverse. We use the precedingplurals list to turn the plural nouns intoTextBlob objects:

   blob_objs = [TextBlob(text) for text in plurals]

5. Turn the nouns into singular using thesingularize functionand print:

   singulars = [blob_obj.words.singularize()[0]     for blob_obj in blob_objs]print(singulars)

   The result should be the same as the list we started with instep 2:

   ['book', 'goose', 'pen', 'point', 'deer']

A dependency parse is a tool that shows dependencies in a sentence. For example, in the sentenceThe cat wore a hat, the root of the sentence is the verb,wore, and both the subject,the cat, and the object,a hat, are dependents. The dependency parse can be very useful in many NLP tasks since it shows the grammatical structure of the sentence, with the subject, the main verb, the object, and so on. It can then be used indownstream processing.

ThespaCy NLP engine does the dependencyparse as part of its overall analysis. The dependency parse tags explain the role of each word in the sentence.ROOT is the main word that all other words depend on, usuallythe verb.

Getting ready

We will usespaCy to create the dependency parse. The required packages are part of thePoetry environment.

How to do it…

We will take a few sentences from thesherlock_holmes1.txt file to illustrate the dependency parse. The steps areas follows:

1. Run the file and languageutility notebooks:

   %run -i "../util/file_utils.ipynb"%run -i "../util/lang_utils.ipynb"

2. Define the sentence we willbe parsing:

   sentence = 'I have seldom heard him mention her under any other name.'

3. Define a function that will print the word, its grammatical function embedded in thedep_ attribute, and the explanation of that attribute. Thedep_ attribute of theToken object shows the grammatical function of the word inthe sentence:

   def print_dependencies(sentence, model):    doc = model(sentence)    for token in doc:        print(token.text, "\t", token.dep_, "\t",             spacy.explain(token.dep_))

4. Now, let’s use this function on the first sentence in our list. We can see that the verbheard is theROOT word of the sentence, with all other words dependingon it:

   print_dependencies(sentence, small_model)

   The result should beas follows:

   I    nsubj    nominal subjecthave    aux    auxiliaryseldom    advmod    adverbial modifierheard    ROOT    roothim    nsubj    nominal subjectmention    ccomp    clausal complementher    dobj    direct objectunder    prep    prepositional modifierany    det    determinerother    amod    adjectival modifiername    pobj    object of preposition.    punct    punctuation

5. To explore the dependency parse structure, we can use the attributes of theToken class. Using theancestors andchildren attributes, we can get the tokens that this token depends on and the tokens that depend on it, respectively. The function to print the ancestors isas follows:

   def print_ancestors(sentence, model):    doc = model(sentence)    for token in doc:        print(token.text, [t.text for t in token.ancestors])

6. Now, let’s use this function on the first sentence inour list:

   print_ancestors(sentence, small_model)

   The output will be as follows. In the result, we see thatheard has no ancestors since it is the main word in the sentence. All other words depend on it, and in fact, containheard in theirancestor lists.

   The dependency chain can be seen by following the ancestor links for each word. For example, if we look at the wordname, we see that its ancestors areunder,mention, andheard. The immediate parent ofname isunder, the parent ofunder ismention, and the parent ofmention isheard. A dependency chain will always lead to the root, or the main word, ofthe sentence:

   I ['heard']have ['heard']seldom ['heard']heard []him ['mention', 'heard']mention ['heard']her ['mention', 'heard']under ['mention', 'heard']any ['name', 'under', 'mention', 'heard']other ['name', 'under', 'mention', 'heard']name ['under', 'mention', 'heard']. ['heard']

7. To see all the children, use the following function. This function prints out each word and the words that depend on it,itschildren:

   def print_children(sentence, model):    doc = model(sentence)    for token in doc:        print(token.text,[t.text for t in token.children])

8. Now, let’s use this function on the first sentence inour list:

   print_children(sentence, small_model)

   The result should be as follows. Now, the wordheard has a list of words that depend on it since it is the main word inthe sentence:

   I []have []seldom []heard ['I', 'have', 'seldom', 'mention', '.']him []mention ['him', 'her', 'under']her []under ['name']any []other []name ['any', 'other']. []

9. We can also see left and right children in separate lists. In the following function, we print the childrenas two separate lists, left and right. This can be useful when doing grammatical transformations inthe sentence:

   def print_lefts_and_rights(sentence, model):    doc = model(sentence)    for token in doc:        print(token.text,            [t.text for t in token.lefts],            [t.text for t in token.rights])

10. Let’s use this function on the first sentence inour list:

    print_lefts_and_rights(sentence, small_model)

    The result should beas follows:

    I [] []have [] []seldom [] []heard ['I', 'have', 'seldom'] ['mention', '.']him [] []mention ['him'] ['her', 'under']her [] []under [] ['name']any [] []other [] []name ['any', 'other'] []. [] []

11. We can also see thesubtree that the token is in by usingthis function:

    def print_subtree(sentence, model):    doc = model(sentence)    for token in doc:        print(token.text, [t.text for t in token.subtree])

12. Let’s use this function on the first sentence inour list:

    print_subtree(sentence, small_model)

    The result should be as follows. From the subtrees that each word is part of, we can see the grammatical phrases that appear in the sentence, such as thenoun phrase,any other name, and theprepositional phrase,under anyother name:

    I ['I']have ['have']seldom ['seldom']heard ['I', 'have', 'seldom', 'heard', 'him', 'mention', 'her', 'under', 'any', 'other', 'name', '.']him ['him']mention ['him', 'mention', 'her', 'under', 'any', 'other', 'name']her ['her']under ['under', 'any', 'other', 'name']any ['any']other ['other']name ['any', 'other', 'name']. ['.']

See also

The dependency parse can be visualized graphically using thedisplaCy package, which is part ofspaCy. Please seeChapter 87,Visualizing Text Data, for a detailed recipe on how to dothe visualization.

Noun chunks are known in linguistics as noun phrases. They represent nouns and any words that depend on and accompany nouns. For example, in the sentenceThe big red apple fell on the scared cat, the noun chunks arethe big red apple andthe scared cat. Extracting these noun chunks is instrumental to many other downstream NLP tasks, such as named entity recognition and processing entities and relations between them. In this recipe, we will explore how to extract named entities froma text.

Getting ready

We will use thespaCy package, which has a function for extracting noun chunks, and the text from thesherlock_holmes_1.txt file asan example.

How to do it…

Use the following steps to get the noun chunks froma text:

1. Run the file and languageutility notebooks:

   %run -i "../util/file_utils.ipynb"%run -i "../util/lang_utils.ipynb"

2. Define the function that will print out the noun chunks. The noun chunks are contained in thedoc.noun_chunksclass variable:

   def print_noun_chunks(text, model):    doc = model(text)    for noun_chunk in doc.noun_chunks:        print(noun_chunk.text)

3. Read the text from thesherlock_holmes_1.txt file and use the function on theresulting text:

   sherlock_holmes_part_of_text = read_text_file("../data/sherlock_holmes_1.txt")print_noun_chunks(sherlock_holmes_part_of_text, small_model)

   This is the partial result. See the output of the notebook athttps://github.com/PacktPublishing/Python-Natural-Language-Processing-Cookbook-Second-Edition/blob/main/Chapter02/noun_chunks_2.3.ipynbfor the full printout. The function gets the pronouns, nouns, and noun phrases that are in thetext correctly:

   Sherlock Holmesshethe_ womanIhimherany other namehis eyesshethe whole…

There’s more…

Noun chunks arespaCySpan objects and have all their properties. See the official documentationathttps://spacy.io/api/token.

Let’s explore some properties ofnoun chunks:

1. We will define a function that will print out the different properties of noun chunks. It will print the text of the noun chunk, its start and end indices within theDoc object, the sentence it belongs to (useful when there is more than one sentence), the root of the noun chunk (its main word), and the chunk’s similarity to the wordemotions. Finally, it will print out the similarity of the whole input sentencetoemotions:

   def explore_properties(sentence, model):    doc = model(sentence)    other_span = "emotions"    other_doc = model(other_span)    for noun_chunk in doc.noun_chunks:        print(noun_chunk.text)        print("Noun chunk start and end", "\t",            noun_chunk.start, "\t", noun_chunk.end)        print("Noun chunk sentence:", noun_chunk.sent)        print("Noun chunk root:", noun_chunk.root.text)        print(f"Noun chunk similarity to '{other_span}'",            noun_chunk.similarity(other_doc))    print(f"Similarity of the sentence '{sentence}' to         '{other_span}':",        doc.similarity(other_doc))

2. Set the sentence toAll emotions, and that one particularly, were abhorrent to his cold, precise but admirablybalanced mind:

   sentence = "All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind."

3. Use theexplore_properties function onthe sentence using thesmall model:

   explore_properties(sentence, small_model)

   This isthe result:

   All emotionsNoun chunk start and end    0    2Noun chunk sentence: All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind.Noun chunk root: emotionsNoun chunk similarity to 'emotions' 0.4026421588260174his cold, precise but admirably balanced mindNoun chunk start and end    11    19Noun chunk sentence: All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind.Noun chunk root: mindNoun chunk similarity to 'emotions' -0.036891259527462Similarity of the sentence 'All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind.' to 'emotions': 0.03174900767577446

   You will also see a warning messagesimilar to this one due to the fact that the small model does not ship with word vectors ofits own:

   /tmp/ipykernel_1807/2430050149.py:10: UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Span.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.  print(f"Noun chunk similarity to '{other_span}'", noun_chunk.similarity(other_doc))

4. Now, let’s apply the same function to the same sentence with thelarge model:

   sentence = "All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind."explore_properties(sentence, large_model)

   The large model does come with its own word vectors and does not result ina warning:

   All emotionsNoun chunk start and end    0    2Noun chunk sentence: All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind.Noun chunk root: emotionsNoun chunk similarity to 'emotions' 0.6302678068015664his cold, precise but admirably balanced mindNoun chunk start and end    11    19Noun chunk sentence: All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind.Noun chunk root: mindNoun chunk similarity to 'emotions' 0.5744456705692561Similarity of the sentence 'All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind.' to 'emotions': 0.640366414527618

   We see that the similarity of theAll emotions noun chunk is high in relation to the wordemotions, as compared to the similarity of thehis cold, precise but admirably balanced mindnoun chunk.

Important note

A largerspaCy model, such asen_core_web_lg, takes up more space but ismore precise.

See also

The topic of semantic similarity will be explored in more detail inChapter 3.

Sometimes, we might need to findthe subject and direct objects of the sentence, and that is easily accomplished with thespaCy package.

Getting ready

We will be using the dependency tags fromspaCy to find subjects and objects. The code uses thespaCy engine to parse the sentence. Then, the subject function loops through the tokens, and if the dependency tag containssubj, it returns that token’s subtree, aSpan object. There are different subject tags, includingnsubj for regular subjects andnsubjpass for subjects of passive sentences, thus we want to lookfor both.

How to do it…

We will use thesubtree attribute of tokens to find the complete noun chunk that is the subject or direct object of the verb (see theGetting the dependency parse recipe). We will define functions to find the subject, direct object, dative phrase, andprepositional phrases:

1. Run the file and languageutility notebooks:

   %run -i "../util/file_utils.ipynb"%run -i "../util/lang_utils.ipynb"

2. We will use two functions to find the subject and the direct object of the sentence. These functions will loop through the tokens and return the subtree that contains the token withsubj ordobj in the dependency tag, respectively. Here is the subject function. It looks for the token that has a dependency tag that containssubj and then returns the subtree that contains that token. There are several subject dependency tags, includingnsubj andnsubjpass (for the subject of a passive sentence), so we look for the mostgeneral pattern:

   def get_subject_phrase(doc):    for token in doc:        if ("subj" in token.dep_):            subtree = list(token.subtree)            start = subtree[0].i            end = subtree[-1].i + 1            return doc[start:end]

3. Here is the direct object function. It works similarly toget_subject_phrase but looks for thedobj dependency tag instead of a tag that containssubj. If the sentence does not have a direct object, it willreturnNone:

   def get_object_phrase(doc):    for token in doc:        if ("dobj" in token.dep_):            subtree = list(token.subtree)            start = subtree[0].i            end = subtree[-1].i + 1            return doc[start:end]

4. Assign a list of sentences to a variable, loop through them, and use the preceding functions to print out their subjectsand objects:

   sentences = [    "The big black cat stared at the small dog.",    "Jane watched her brother in the evenings.",    "Laura gave Sam a very interesting book."]for sentence in sentences:    doc = small_model(sentence)    subject_phrase = get_subject_phrase(doc)    object_phrase = get_object_phrase(doc)    print(sentence)    print("\tSubject:", subject_phrase)    print("\tDirect object:", object_phrase)

   The result will be as follows. Since the first sentence does not have a direct object,None is printed out. For thesentenceThe big black cat stared at the small dog, the subject isthe big black cat and there is no direct object (the small dog is the object of the prepositionat). For the sentenceJane watched her brother in the evenings, the subject isJane and the direct object isher brother. In the sentenceLaura gave Sam a very interesting book, the subject isLaura and the direct object isa veryinteresting book:

   The big black cat stared at the small dog.  Subject: The big black cat  Direct object: NoneJane watched her brother in the evenings.  Subject: Jane  Direct object: her brotherLaura gave Sam a very interesting book.  Subject: Laura  Direct object: a very interesting book

There’s more…

We can look for other objects, for example, the dative objects of verbs such asgive and objects of prepositional phrases. The functions will look very similar, with the main difference being the dependency tags:dative for the dative object function, andpobj for the prepositional object function. The prepositional object function will return a list since there can be more than one prepositional phrase ina sentence:

1. The dative object function checks the tokens for thedative tag. It returnsNone if there are nodative objects:

   def get_dative_phrase(doc):    for token in doc:        if ("dative" in token.dep_):            subtree = list(token.subtree)            start = subtree[0].i            end = subtree[-1].i + 1            return doc[start:end]

2. We can also combine the subject, object, and dative functions into one with an argument that specifies which object tolook for:

   def get_phrase(doc, phrase):    # phrase is one of "subj", "obj", "dative"    for token in doc:        if (phrase in token.dep_):            subtree = list(token.subtree)            start = subtree[0].i            end = subtree[-1].i + 1            return doc[start:end]

3. Let us now define a sentence with a dative object and run the function for all three typesof phrases:

   sentence = "Laura gave Sam a very interesting book."doc = small_model(sentence)subject_phrase = get_phrase(doc, "subj")object_phrase = get_phrase(doc, "obj")dative_phrase = get_phrase(doc, "dative")print(sentence)print("\tSubject:", subject_phrase)print("\tDirect object:", object_phrase)print("\tDative object:", dative_phrase)

   The result will be as follows. The dative objectisSam:

   Laura gave Sam a very interesting book.  Subject: Laura  Direct object: a very interesting book  Dative object: Sam

4. Here is the prepositional object function. It returns a list of objects of prepositions, which will be empty if thereare none:

   def get_prepositional_phrase_objs(doc):    prep_spans = []    for token in doc:        if ("pobj" in token.dep_):            subtree = list(token.subtree)            start = subtree[0].i            end = subtree[-1].i + 1            prep_spans.append(doc[start:end])    return prep_spans

5. Let’s define a list of sentences and run the two functionson them:

   sentences = [    "The big black cat stared at the small dog.",    "Jane watched her brother in the evenings."]for sentence in sentences:    doc = small_model(sentence)    subject_phrase = get_phrase(doc, "subj")    object_phrase = get_phrase(doc, "obj")    dative_phrase = get_phrase(doc, "dative")    prepositional_phrase_objs = \        get_prepositional_phrase_objs(doc)    print(sentence)    print("\tSubject:", subject_phrase)    print("\tDirect object:", object_phrase)    print("\tPrepositional phrases:", prepositional_phrase_objs)

   The result will beas follows:

   The big black cat stared at the small dog.  Subject: The big black cat  Direct object: the small dog  Prepositional phrases: [the small dog]Jane watched her brother in the evenings.  Subject: Jane  Direct object: her brother  Prepositional phrases: [the evenings]

   There is one prepositional phrase in each sentence. In the sentenceThe big black cat stared at the small dog, it isat the small dog, and in the sentenceJane watched her brother in the evenings, it isinthe evenings.

It is left as an exercise for you to find the actual prepositional phrases with prepositions intact instead of just the noun phrases that are dependent onthese prepositions.

In this section, we will use thespaCyMatcher object to find patterns in the text. We will use the grammatical properties of the words to create these patterns. For example, we might be looking for verb phrases instead of noun phrases. We can specify grammatical patterns to matchverb phrases.

Getting ready

We will be using thespaCyMatcher object to specify and find patterns. It can match different properties, not just grammatical. You can find out more in the documentationathttps://spacy.io/usage/rule-based-matching/.

How to do it…

Your steps should be formattedlike so:

1. Run the file and languageutility notebooks:

   %run -i "../util/file_utils.ipynb"%run -i "../util/lang_utils.ipynb"

2. Import theMatcher object and initialize it. We need to put in the vocabulary object, which is the same as the vocabulary of the model we will be using to processthe text:

   from spacy.matcher import Matchermatcher = Matcher(small_model.vocab)

3. Create a list of patterns and add them to the matcher. Each pattern is a list of dictionaries, where each dictionary describes a token. In our patterns, we only specify the part of speech for each token. We then add these patterns to theMatcher object. The patterns we will be using are a verb by itself (for example,paints), an auxiliary followed by a verb (for example,was observing), an auxiliary followed by an adjective (for example,were late), and an auxiliary followed by a verb and a preposition (for example,were staring at). This is not an exhaustive list; feel free to come up withother examples:

   patterns = [    [{"POS": "VERB"}],    [{"POS": "AUX"}, {"POS": "VERB"}],    [{"POS": "AUX"}, {"POS": "ADJ"}],    [{"POS": "AUX"}, {"POS": "VERB"}, {"POS": "ADP"}]]matcher.add("Verb", patterns)

4. Read in the small part of theSherlock Holmes text and process it using thesmall model:

   sherlock_holmes_part_of_text = read_text_file("../data/sherlock_holmes_1.txt")doc = small_model(sherlock_holmes_part_of_text)

5. Now, we find the matches using theMatcher object and the processed text. We then loop through the matches and print out the match ID, the string ID (the identifier of the pattern), the start and end of the match, and the text ofthe match:

   matches = matcher(doc)for match_id, start, end in matches:    string_id = small_model.vocab.strings[match_id]    span = doc[start:end]    print(match_id, string_id, start, end, span.text)

   The result will beas follows:

   14677086776663181681 Verb 14 15 heard14677086776663181681 Verb 17 18 mention14677086776663181681 Verb 28 29 eclipses14677086776663181681 Verb 31 32 predominates14677086776663181681 Verb 43 44 felt14677086776663181681 Verb 49 50 love14677086776663181681 Verb 63 65 were abhorrent14677086776663181681 Verb 80 81 take14677086776663181681 Verb 88 89 observing14677086776663181681 Verb 94 96 has seen14677086776663181681 Verb 95 96 seen14677086776663181681 Verb 103 105 have placed14677086776663181681 Verb 104 105 placed14677086776663181681 Verb 114 115 spoke14677086776663181681 Verb 120 121 save14677086776663181681 Verb 130 132 were admirable14677086776663181681 Verb 140 141 drawing14677086776663181681 Verb 153 154 trained14677086776663181681 Verb 157 158 admit14677086776663181681 Verb 167 168 adjusted14677086776663181681 Verb 171 172 introduce14677086776663181681 Verb 173 174 distracting14677086776663181681 Verb 178 179 throw14677086776663181681 Verb 228 229 was

The code finds some of the verbphrases in the text. Sometimes, it finds a partial match that is part of another match. Weeding out these partial matches is left asan exercise.

See also

We can use other attributes apart from parts of speech. It is possible to match on the text itself, its length, whether it is alphanumeric, the punctuation, the word’s case, thedep_ andmorph attributes, lemma, entity type, and others. It is also possible to use regular expressions on the patterns. Formore information, see the spaCydocumentation:https://spacy.io/usage/rule-based-matching.