Repository files navigation

Purpose: The focus of the package ‘polmineR’ is the interactiveanalysis of corpora using R. Core objectives for the development of thepackage are performance, usability, and a modular design.

Aims: Key aims for developing the package are:

• To keep the original text accessible. A seamless integration ofqualitative and quantitative steps in corpus analysis supportsvalidation, based on inspecting the text behind the numbers.

• To provide a library with standard tasks. It is an open sourceplatform that will make text mining more productive, avoidingprohibitive costs to reimplement basics, or to run many lines ofcode to perform a basic tasks.

• To create a package that makes the creation and analysis ofsubcorpora (‘partitions’) easy. A particular strength of the packageis to support contrastive/comparative research.

• To offer performance for users with a standard infrastructure. Thepackage picks up the idea of a three-tier software design. Corpusdata are managed and indexed by using theOpen Corpus Workbench(CWB). The CWB is particularly efficient for storing large corporaand offers a powerful language for querying corpora, the CorpusQuery Processor (CQP).

• To support sharing consolidated and documented data, following theideas of reproducible research.

Background: The polmineR-package was specifically developed to makefull use of the XML annotation structure of the corpora created in thePolMine project (see polmine.sowi.uni-due.de). The core PolMine corporaare corpora of plenary protocols. In these corpora, speakers, partiesetc. are structurally annotated. The polmineR-package is meant to helpmaking full use of the rich annotation structure.

To demonstrate the core functionality of package, we load polmineR.

library(polmineR)

The package includes two small sample corpora (REUTERS andGERMAPARLMINI). Here we want two use somewhat bigger “real life” corpora(Europarl and GermaParl). Thecwbtools package offersan installation mechanism, so we install this package first.

install.packages("cwbtools")

We now install Europarl …

europarl<-"http://corpora.linguistik.uni-erlangen.de/demos/download/Europarl3-CWB-2010-02-28.tar.gz"cwbtools::corpus_install(tarball=europarl)

… and the GermaParl corpus of parliamentary debates.

cwbtools::corpus_install(doi="10.5281/zenodo.3742113")

All methods can be applied to a whole corpus, as well as to partitions(i.e. subcorpora). Use the metadata of a corpus (so-called s-attributes)to define a subcorpus.

ep2005<- partition("EUROPARL-EN",text_year="2006")#> ... get encoding: latin1#> ... get cpos and strucssize(ep2005)#> [1] 3100529

barroso<- partition("EUROPARL-EN",speaker_name="Barroso",regex=TRUE)#> ... get encoding: latin1#> ... get cpos and strucssize(barroso)#> [1] 98142

Partitions can be bundled into partition_bundle objects, and mostmethods can be applied to a whole corpus, a partition, or apartition_bundle object alike. Consult the package vignette to learnmore.

Counting occurrences of a feature in a corpus, a partition or in thepartitions of a partition_bundle is a basic operation. By offeringaccess to the query syntax of the Corpus Query Processor (CQP), polmineRpackage exposes a query syntax that goes far beyond regular expressions.See theCQPdocumentationto learn more.

count("EUROPARL-EN","France")#>     query count         freq#> 1: France  5517 0.0001399122count("EUROPARL-EN", c("France","Germany","Britain","Spain","Italy","Denmark","Poland"))#>      query count         freq#> 1:  France  5517 1.399122e-04#> 2: Germany  4196 1.064114e-04#> 3: Britain  1708 4.331523e-05#> 4:   Spain  3378 8.566676e-05#> 5:   Italy  3209 8.138089e-05#> 6: Denmark  1615 4.095673e-05#> 7:  Poland  1820 4.615557e-05count("EUROPARL-EN",'"[pP]opulism"')#>            query count         freq#> 1: "[pP]opulism"   107 2.713542e-06

The dispersion method is there to analyse the dispersion of a query, ora set of queries across one or two dimensions (absolute and relativefrequencies). The CQP syntax can be used.

populism<- dispersion("EUROPARL-EN","populism",s_attribute="text_year",progress=FALSE)pop_regex<- dispersion("EUROPARL-EN",'"[pP]opulism"',s_attribute="text_year",cqp=TRUE,progress=FALSE)

The cooccurrences method is used to analyse the context of a query(including some statistics).

islam<- cooccurrences("EUROPARL-EN",query='Islam',left=10,right=10)islam<- subset(islam,rank_ll<=100)dotplot(islam)

Compare partitions to identify features / keywords (using statisticaltests such as chi square).

ep_2002<- partition("EUROPARL-EN",text_year="2002",p_attribute="word")ep_pre_2002<- partition("EUROPARL-EN",text_year=1997:2001,p_attribute="word")features(ep_2002,ep_pre_2002,included=FALSE) %>%  subset(rank_chisquare<=10) %>%  format() %>%knitr::kable(format="markdown")

So what happens in the context of a word, or a CQP query? To attainvalid research results, reading will often be necessary. The kwic methodwill help, and uses the conveniences of DataTables, outputted in theViewer pane of RStudio.

kwic("EUROPARL-EN","Islam",meta= c("text_date","speaker_name")) %>%  as.data.frame() %>%.[1:8,] %>%knitr::kable(format="markdown",escape=FALSE)

Corpus analysis involves moving from text to numbers, and back again.Use the read method, to inspect the full text of a partition (a speechgiven by chancellor Angela Merkel in this case).

merkel<- partition("GERMAPARL",speaker="Angela Merkel",date="2013-09-03")read(merkel)

Many advanced methods in text mining require term document matrices asinput. Based on the metadata of a corpus, these data structures can beobtained in a fast and flexible manner, for performing topic modelling,machine learning etc.

speakers<- partition_bundle("EUROPARL-EN",s_attribute="speaker_id",progress=FALSE,verbose=FALSE)speakers_count<- count(speakers,p_attribute="word",progress=TRUE)tdm<- as.TermDocumentMatrix(speakers_count,col="count")dim(tdm)

The following instructions assume that you have installed R. If not,install it fromCRAN. Aninstallation ofRStudiois highly recommended.

The CRAN release of polmineR can be installed usinginstall.packages(), all dependencies will be installed, too.

install.packages("polmineR")

To install the most recent development version that is hosted in aGitHub repository, use the installation mechanism offered by thedevtools package.

install.packages("devtools")devtools::install_github("PolMine/polmineR",ref="dev")

Check the installation by loading polmineR and activating the corporaincluded in the package.

library(polmineR)corpus()

CRAN offerspolmineR as abinary package both for Intel processors (x86_64 architecture) and thenewer Apple silicon chips (arm64 architecture). If R and RStudio are notyet installed, follow these preparatory steps.

• InstallingXQuartz is recommended. Theavailable image works for Intel and Apple chips. Note that XQuartzcapability is configured and used by R only if XQuartz has beeninstalledbefore installing R.

• Install R. Note that packages are available for both Intel 64-bitand Apple silicon arm64 chip architectures. Install what applies foryou, see theR for macOSsite.

• InstallRStudio.The free version of RStudio Desktop is enough. Starting with version1.4, Apple silicon is supported. When installing RStudio, users witha Apple silicon chips are be asked to installRosetta (say yes).

• When starting RStudio the first time, you may be asked to installtheCommand Line DeveloperTools. This is notnecessary for basic polmineR usage, but recommended. Note thatdownloading the Command Line Developer Tools may require a stableinternet connection and still take some time.

Then run this command for installing polmineR:

install.packages("polmineR")

The installation mechanism will determine which binary version yourequire and install all required dependencies.

For installing the development version of polmineR and building thepackage from source, the Command Line Developer Tools need to beinstalled. Install them from a terminal window as follows.

xcode-select --install

If you haven’t done so already, installXQuartz, R and RStudio (see previousinstructions for binary installation).

Thedevtools packageexposes a convenient and commonly used installation mechanism forinstalling a package from GitHub. First install the devtools package,which involves the installation of several dependencies.

install.packages("devtools")# unless devtools is already installed

Then use theinstall_github() function as follows.

devtools::install_github("PolMine/polmineR",ref="dev")

The development version of polmineR may require the installation of adevelopment version of the RcppCWB: polmineR interacts with the CorpusWorkbench (CWB) via RcppCWB, an R package which exposes the C-levelfunctions of the CWB. If you want or need to install a developmentversion of RcppCWB, several system dependencies need to be fulfilled forcompiling the package from source. See theREADME of the RcppCWB GitHubrepository for instructions.

Check whether everything works by loading polmineR, and see whether yousee the demo corpora included in the package.

library(polmineR)corpus()

If you have not yet installed R on your Ubuntu machine, there is a goodinstruction atubuntuuser. To installbase R, enter in the terminal.

sudo apt-get install r-base r-recommended

Make sure that you have installed the latest version of R. The followingcommands will add the R repository to the package sources and run anupdate. The second line assumes that you are using Ubuntu 16.04.

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E084DAB9sudo add-apt-repository'deb http://ftp5.gwdg.de/pub/misc/cran/bin/linux/ubuntu xenial/'sudo apt-get updatesudo apt-get upgrade

It is highly recommended to installRStudio,a powerful IDE for R. Output of polmineR methods is generally optimizedto be displayed using RStudio facilities. If you are working on a remoteserver, running RStudio Server may be an interesting option to consider.

The RcppCWB package, the interface used by polmineR to query CWBcorpora, will require the pcre, glib and pkg-config libraries. They canbe installed as follows. In addition libxml2 is installed, a dependencyof the R package xml2 that is used for manipulating html output.

sudo apt-get install libglib2.0-dev libssl-dev libcurl4-openssl-devsudo apt-get install libxml2-devsudo apt-get install libprotobuf-dev protobuf-compiler

The system requirements will now be fulfilled. From R, installdependencies for rcqp/polmineR first, and then rcqp and polmineR.

install.packages("RcppCWB")install.packages("polmineR")

Use devtools to install the development version of polmineR from GitHub.

install.packages("devtools")devtools::install_github("PolMine/polmineR",ref="dev")

You may want to install packaged corpora to run examples in thevignette, and the man packages.

library(polmineR)corpus()

To have access to all package functions and to run all package tests,the installation of further system requirements and packages isrequired. The xlsx dependency requires that rJava is installed andconfigured for R. That is done on the shell:

sudo apt-get install openjdk-8-jresudo R CMD javareconf

To run package tests including (re-)building the manual and vignettes, aworking installation of Latex is required, too. Be aware that this maybe a time-consuming operation.

sudo apt-get install texlive-full texlive-xetex

Now install the remaining packages from within R.

install.packages(pkgs= c("rJava","xlsx","tidytext"))

The polmineR package has been developed to be useful for research. Ifyou publish research results making use of polmineR, the followingcitation is suggested to be included in publications.

Blaette, Andreas (2020). polmineR: Verbs and Nouns for Corpus Analysis.R package version v0.8.5.http://doi.org/10.5281/zenodo.4042093