Pattern is a web mining module for the Pythonprogramming language.

It has tools for data mining (Google, Twitter andWikipedia API, a web crawler, a HTML DOM parser), natural languageprocessing (part-of-speech taggers, n-gram search, sentiment analysis,WordNet), machine learning (vector space model, clustering, SVM),network analysis and <canvas> visualization.

The module is free, well-document and bundled with 50+ examples and 350+unit tests.

Pattern 3.6 |download 3.6 master (.zip, 23MB) Requires: Python 3.6 (current version) or Python 2.5+ (previous version) on Windows | Mac | Linux Licensed underBSD Latest releases:3.6 master |2.7 |2.6 | 2.5 | 2.4 |2.3 | 2.2 | 2.1 | 2.0 Authors:  Tom De Smedt (tom at organisms.be)  Walter Daelemans  __Reference__: De Smedt, T. & Daelemans, W. (2012). Pattern for Python.Journal of Machine Learning Research, 13: 2031–2035. __SHA256__ checksum of the .zip: 003aa285adc23b72255ca8b9f052c6a2644b245b3a9666af007e086428a950be

Tweet

Modules pattern.web pattern.db pattern.en | es |de |fr |it | nl pattern.search pattern.vector pattern.graph  *Helper modules* pattern.metrics canvas.js *Command-line* Command-line interface

Contribute Developer documentation GitHub repository Google group

Pattern is written for Python 2.5+ and recently added support for Python 3.6 -- however, note that these wiki instructions have not yet been updated with the instructions for Python 3.6.

The module has no external dependencies, exceptLSA in the pattern.vector module, whichrequiresNumPy (installed by default on MacOS X). 

To install Pattern so that the module is available in all Pythonscripts, from the command line do:

> cd pattern-2.6> python setup.py install 

If you have pip, you can automatically download and install from thePyPi repository:

> pip install pattern

If none of the above works, you can make Python aware of the module inthree ways:

• Put thepattern subfolder in the.zip archive in the same folder as your script.

• Put thepattern subfolder in thestandard location for modules so it is available to all scripts:
  c:\\python27\\Lib\\site-packages\\ (Windows),
  /Library/Python/2.7/site-packages/ (Mac),

  /usr/lib/python2.7/site-packages/ (Unix).
  

• Add the location of the module to sys.path in your Python script, beforeimporting it:

  import sys; sys.path.append('/users/tom/desktop/pattern')from pattern.web import Twitter

The pattern.web module is a web toolkit thatcontains API's (Google, Gmail, Bing, Twitter, Facebook, Wikipedia,Wiktionary, DBPedia, Flickr, ...), a robust HTML DOM parser and a webcrawler.

>>> from pattern.web import Twitter, plaintext>>>>>> twitter = Twitter(language='en') >>> for tweet in twitter.search('"more important than"', cached=False):>>>    print plaintext(tweet.text)'The mobile web is more important than mobile apps.''Start slowly, direction is more important than speed.''Imagination is more important than knowledge. - Albert Einstein'...

The pattern.en module is a natural languageprocessing (NLP) toolkit for English. Because language is ambiguous(e.g.,I can ↔a can) it uses statistical approaches + regularexpressions. This means that it is fast, quite accurate and occasionallyincorrect. It has a part-of-speech tagger that identifies word types(e.g., noun, verb, adjective), word inflection (conjugation,singularization) and a WordNet API.

>>> from pattern.en import parse>>>  >>> s = 'The mobile web is more important than mobile apps.'>>> s = parse(s, relations=True, lemmata=True)>>> print s'The/DT/B-NP/O/NP-SBJ-1/the mobile/JJ/I-NP/O/NP-SBJ-1/mobile' ...

The text has been annotated with word types, for example nouns (NN), verbs(VB), adjectives (JJ)and determiners (DT), word types(e.g., sentence subject SBJ) andprepositional noun phrases (PNP). To iterateover the parts in the tagged text we can construct aparse tree.

>>> from pattern.en import parsetree>>>  >>> s = 'The mobile web is more important than mobile apps.'>>> s = parsetree(s) >>> for sentence in s: >>>     for chunk in sentence.chunks:>>>         for word in chunk.words:>>>             print word,>>>         printWord(u'The/DT') Word(u'mobile/JJ') Word(u'web/NN')Word(u'is/VBZ')Word(u'more/RBR') Word(u'important/JJ')Word(u'than/IN')Word(u'mobile/JJ') Word(u'apps/NNS')

Parsers for Spanish, French, Italian, German and Dutch are alsoavailable:
pattern.es |pattern.fr |pattern.it| pattern.de | pattern.nl

The pattern.search module contains a searchalgorithm to retrieve sequences of words (calledn-grams) from taggedtext.

>>> from pattern.en import parsetree>>> from pattern.search import search>>>  >>> s = 'The mobile web is more important than mobile apps.'>>> s = parsetree(s, relations=True, lemmata=True)>>>>>> for match in search('NP be RB?+ important than NP', s):>>>     print match.constituents()[-1], '=>', \>>>           match.constituents()[0]Chunk('mobile apps/NP') => Chunk('The mobile web/NP-SBJ-1')

The search pattern NPbeRB?+importantthanNPmeans any noun phrase (NP) followed by theverbto be, followed by zero or more adverbs (RB, e.g., much,more), followed by the wordsimportant than, followed by any noun phrase. It will also match "Themobile webwillbemuchless important than mobileapps" and other grammatical variations.

The pattern.vector module is a toolkit formachine learning, based on a vector space model of bag-of-wordsdocuments with weighted features (e.g., tf-idf) and distance metrics(e.g., cosine similarity, infogain). Models can be used for clustering(k-means, hierarchical), classification (Naive Bayes,Perceptron, *k-*NN, SVM) and latent semantic analysis (LSA).

>>> from pattern.web    import Twitter>>> from pattern.en     import tag>>> from pattern.vector import KNN, count>>> >>> twitter, knn = Twitter(), KNN()>>> >>> for i in range(1, 10):>>>     for tweet in twitter.search('#win OR #fail', start=i, count=100):>>>         s = tweet.text.lower()>>>         p = '#win' in s and 'WIN' or 'FAIL'>>>         v = tag(s)>>>         v = [word for word, pos in v if pos == 'JJ'] # JJ = adjective>>>         v = count(v) >>>         if v:>>>             knn.train(v, type=p)>>> >>> print knn.classify('sweet potato burger')>>> print knn.classify('stupid autocorrect')'WIN''FAIL'

This example trains a classifier on adjectives mined from Twitter.First, tweets with hashtag #win or #fail are mined. For example:"$20tip off asweetlittleold lady today #win". Theword part-of-speech tags are parsed, keeping only adjectives. Each tweetis transformed to a vector, a dictionary of adjective → count items,labeledWIN orFAIL. The classifier uses the vectors tolearn which other, unknown tweets look more like WIN (e.g.,sweet potato burger) or morelikeFAIL (e.g.,stupid autocorrect).

The pattern.graph module provides a graph datastructure that represents relations between nodes (e.g., terms,concepts). Graphs can be exported as HTML<canvas> animations (demo). In theexample below, morecentral nodes (= more incoming traffic) arecolored in blue.

>>> from pattern.web    import Bing, plaintext>>> from pattern.en     import parsetree>>> from pattern.search import search>>> from pattern.graph  import Graph>>>  >>> g = Graph()>>> for i in range(10):>>>     for result in Bing().search('"more important than"', start=i+1, count=50):>>>         s = r.text.lower() >>>         s = plaintext(s)>>>         s = parsetree(s)>>>         p = '{NP} (VP) more important than {NP}' >>>         for m in search(p, s):>>>             x = m.group(1).string # NP left>>>             y = m.group(2).string # NP right>>>             if x not in g:>>>                 g.add_node(x)>>>             if y not in g:>>>                 g.add_node(y)>>>             g.add_edge(g[x], g[y], stroke=(0,0,0,0.75)) # R,G,B,A>>>  >>> g = g.split()[0] # Largest subgraph.>>>  >>> for n in g.sorted()[:40]: # Sort by Node.weight.>>>     n.fill = (0, 0.5, 1, 0.75 * n.weight)>>>  >>> g.export('test', directed=True, weighted=0.6)

Some relations (= edges) could use some extra post-processing, e.g., innothing is more important than life,nothing isnot more important thanlife.

Case studies with hands-on source code examples.