This article has multiple issues. Please helpimprove it or discuss these issues on thetalk page.(Learn how and when to remove these messages) The topic of this articlemay not meet Wikipedia'sgeneral notability guideline. Please help to demonstrate the notability of the topic by citingreliable secondary sources that areindependent of the topic and provide significant coverage of it beyond a mere trivial mention. If notability cannot be shown, the article is likely to bemerged,redirected, ordeleted. Find sources: "Programming with Big Data in R" – news ·newspapers ·books ·scholar ·JSTOR(June 2013) (Learn how and when to remove this message) A major contributor to this article appears to have aclose connection with its subject. It may require cleanup to comply with Wikipedia's content policies, particularlyneutral point of view. Please discuss further on thetalk page.(June 2013) (Learn how and when to remove this message) (Learn how and when to remove this message)

Programming with Big Data in R (pbdR)^[1] is a series ofR packages and an environment forstatistical computing withbig data by using high-performance statistical computation.^[2][3] The pbdR uses the same programming language as R withS3/S4 classes and methods which is used amongstatisticians anddata miners for developingstatistical software. The significant difference between pbdR and R code is that pbdR mainly focuses ondistributed memory systems, where data are distributed across several processors and analyzed in abatch mode, while communications between processors are based onMPI that is easily used in largehigh-performance computing (HPC) systems. R system mainly focuses^{[citation needed]} on singlemulti-core machines for data analysis via an interactive mode such asGUI interface.

Two main implementations inR usingMPI are Rmpi^[4] and pbdMPI of pbdR.

• The pbdR built on pbdMPI usesSPMD parallelism where every processor is considered as worker and owns parts of data. TheSPMD parallelism introduced in mid 1980 is particularly efficient in homogeneous computing environments for large data, for example, performingsingular value decomposition on a large matrix, or performingclustering analysis on high-dimensional large data. On the other hand, there is no restriction to usemanager/workers parallelism inSPMD parallelism environment.
• The Rmpi^[4] usesmanager/workers parallelism where one main processor (manager) serves as the control of all other processors (workers). Themanager/workers parallelism introduced around early 2000 is particularly efficient for large tasks in smallclusters, for example,bootstrap method andMonte Carlo simulation in applied statistics sincei.i.d. assumption is commonly used in moststatistical analysis. In particular, task pull parallelism has better performance for Rmpi in heterogeneous computing environments.

The idea ofSPMD parallelism is to let every processor do the same amount of work, but on different parts of a large data set. For example, a modernGPU is a large collection of slower co-processors that can simply apply the same computation on different parts of relatively smaller data, but the SPMD parallelism ends up with an efficient way to obtain final solutions (i.e. time to solution is shorter).^[5]

Programming with pbdR requires usage of various packages developed by pbdR core team. Packages developed are the following.

Among these packages, pbdMPI provides wrapper functions toMPI library, and it also produces ashared library and a configuration file for MPI environments. All other packages rely on this configuration for installation and library loading that avoids difficulty of library linking and compiling. All other packages can directly use MPI functions easily.

• pbdMPI --- an efficient interface to MPI eitherOpenMPI orMPICH2 with a focus on Single Program/Multiple Data (SPMD) parallel programming style
• pbdSLAP --- bundles scalable dense linear algebra libraries in double precision for R, based onScaLAPACK version 2.0.2 which includes several scalable linear algebra packages (namelyBLACS,PBLAS, andScaLAPACK).
• pbdNCDF4 --- interface to Parallel UnidataNetCDF4 format data files
• pbdBASE --- low-levelScaLAPACK codes and wrappers
• pbdDMAT --- distributed matrix classes and computational methods, with a focus on linear algebra and statistics
• pbdDEMO --- set of package demonstrations and examples, and this unifying vignette
• pmclust --- parallelmodel-based clustering using pbdR
• pbdPROF --- profiling package for MPI codes and visualization of parsed stats
• pbdZMQ --- interface toØMQ
• remoter --- R client with remote R servers
• pbdCS --- pbdR client with remote pbdR servers
• pbdRPC --- remote procedure call
• kazaam --- very tall and skinny distributed matrices
• pbdML --- machine learning toolbox

Among those packages, the pbdDEMO package is a collection of 20+ package demos which offer example uses of the various pbdR packages, and contains a vignette that offers detailed explanations for the demos and provides some mathematical or statistical insight.

Hello World! Save the following code in a file called "demo.r"

### Initial MPIlibrary(pbdMPI,quiet=TRUE)init()comm.cat("Hello World!\n")### Finishfinalize()

and use the command

mpiexec-np2Rscriptdemo.r

to execute the code whereRscript is one of command line executable program.

The following example modified from pbdMPI illustrates the basicsyntax of the language of pbdR.Since pbdR is designed inSPMD, all the R scripts are stored in files and executed from the command line via mpiexec, mpirun, etc. Save the following code in a file called "demo.r"

### Initial MPIlibrary(pbdMPI,quiet=TRUE)init().comm.size<-comm.size().comm.rank<-comm.rank()### Set a vector x on all processors with different valuesN<-5x<-(1:N)+N*.comm.rank### All reduce x using summation operationy<-allreduce(as.integer(x),op="sum")comm.print(y)y<-allreduce(as.double(x),op="sum")comm.print(y)### Finishfinalize()

and use the command

mpiexec-np4Rscriptdemo.r

to execute the code whereRscript is one of command line executable program.

The following example modified from pbdDEMO illustrates the basic ddmatrix computation of pbdR which performssingular value decomposition on a given matrix.Save the following code in a file called "demo.r"

# Initialize process gridlibrary(pbdDMAT,quiet=T)if(comm.size()!=2)comm.stop("Exactly 2 processors are required for this demo.")init.grid()# Setup for the remaindercomm.set.seed(diff=TRUE)M<-N<-16BL<-2# blocking --- passing single value BL assumes BLxBL blockingdA<-ddmatrix("rnorm",nrow=M,ncol=N,mean=100,sd=10)# LA SVDsvd1<-La.svd(dA)comm.print(svd1$d)# Finishfinalize()

and use the command

mpiexec-np2Rscriptdemo.r

to execute the code whereRscript is one of command line executable program.

• Raim, A.M. (2013).Introduction to distributed computing with pbdR at the UMBC High Performance Computing Facility(PDF) (Technical report). UMBC High Performance Computing Facility, University of Maryland, Baltimore County. HPCF-2013-2. Archived fromthe original(PDF) on 2014-02-04. Retrieved2013-06-26.
• Bachmann, M.G., Dyas, A.D., Kilmer, S.C. and Sass, J. (2013).Block Cyclic Distribution of Data in pbdR and its Effects on Computational Efficiency(PDF) (Technical report). UMBC High Performance Computing Facility, University of Maryland, Baltimore County. HPCF-2013-11. Archived fromthe original(PDF) on 2014-02-04. Retrieved2014-02-01.{{cite tech report}}: CS1 maint: multiple names: authors list (link)
• Bailey, W.J., Chambless, C.A., Cho, B.M. and Smith, J.D. (2013).Identifying Nonlinear Correlations in High Dimensional Data with Application to Protein Molecular Dynamics Simulations(PDF) (Technical report). UMBC High Performance Computing Facility, University of Maryland, Baltimore County. HPCF-2013-12. Archived fromthe original(PDF) on 2014-02-04. Retrieved2014-02-01.{{cite tech report}}: CS1 maint: multiple names: authors list (link)
• Dirk Eddelbuettel (13 November 2022)."High-Performance and Parallel Computing with R".
• "R at 12,000 Cores".
  This article was read 22,584 times in 2012 since it posted on October 16, 2012, and ranked number 3^[6]
• Google Summer of Code - R 2013."Profiling Tools for Parallel Computing with R". Archived fromthe original on 2013-06-29.{{cite web}}:|author= has generic name (help)CS1 maint: numeric names: authors list (link)
• Wush Wu (2014)."在雲端運算環境使用R和MPI".{{cite web}}: CS1 maint: numeric names: authors list (link)
• Wush Wu (2013)."快速在AWS建立R和pbdMPI的使用環境".YouTube.{{cite web}}: CS1 maint: numeric names: authors list (link)

• Official website

• Cross-platform free software
• Data mining and machine learning software
• Data-centric programming languages
• Free statistical software
• Functional languages
• Numerical analysis software for Linux
• Numerical analysis software for macOS
• Numerical analysis software for Windows
• Parallel computing

• CS1 maint: multiple names: authors list
• Articles with topics of unclear notability from June 2013
• All articles with topics of unclear notability
• Wikipedia articles with possible conflicts of interest from June 2013
• Articles with multiple maintenance issues
• All articles with unsourced statements
• Articles with unsourced statements from July 2013
• CS1 errors: generic name
• CS1 maint: numeric names: authors list
• Official website not in Wikidata