Overview

memshare enables multicore computation in R withoutredundant memory copies. Large vectors, matrices, or lists are storedonce in shared memory and exposed to R processes asALTREPviews. This allows workers in a PSOCK cluster to operate on thesame physical data while avoiding serialization overhead.

Key features:

• Shared memory allocation via C++17 back-end (shm_openon Unix,MapViewOfFile on Windows).
• ALTREP wrappers so R sees shared objects as native vectors ormatrices.
• High-level parallels toparallel::parApply andparallel::parLapply:
  • memApply() — apply a function row/column-wise over amatrix in shared memory.
  • memLapply() — apply a function element-wise over a listin shared memory.
• Automatic cleanup of shared objects when all views are released orwhen the package is unloaded.
• Tested on Linux, macOS, and Windows.

Installation

From CRAN:

install.packages("memshare")

From GitHub (development version):

remotes::install_github("mthrun/memshare")

System requirements: R ≥ 4.0 and a C++17 compiler.

Quick start

Example 1:Parallel correlations with a matrix

library(memshare)library(parallel)set.seed(1)n=10000p=2000X=matrix(rnorm(n* p), n, p)y=rnorm(n)res=memApply(X = X,MARGIN =2,FUN =function(v, y)cor(v, y),VARS =list(y = y))str(res)

Example 2: List operations

library(memshare)library(parallel)list_length=1000matrix_dim=100ListV=lapply(1:list_length,function(i)matrix(rnorm(matrix_dim* matrix_dim),nrow = matrix_dim,ncol = matrix_dim))y=rnorm(matrix_dim)namespace="ID123"res= memshare::memLapply(ListV,function(el, y) {   el%*% y},NAMESPACE=namespace,VARS=list(y=y),MAX.CORES =1)

Each element el of ListV is multiplied by y in parallel. The listresides once in shared memory.

Concepts

• Pages: memory regions owned by the current R session that loadedthe package.

• Views: ALTREP wrappers exposing shared memory variables(read/write capable).

• Namespaces: string identifiers defining a shared memory contextacross sessions.

When the package is detached, all handles and associated sharedmemory pages are released, unless another R process still holdsreferences.

Shared-memory lifecycle

memshare exposes explicit lifecycle functions so you cancontrol when data is placed in shared memory and when it is freed.

library(memshare)ns<-"my_namespace"X<-matrix(rnorm(1e4),100,100)y<-rnorm(100)registerVariables(ns,list(X = X,y = y))# Now X and y live once in shared memory and can be accessed from other R processes

# After all workers have released their views:releaseVariables(ns,c("X","y"))

vlist<-retrieveViews(ns,c("X","y"))# use vlist$X and vlist$yreleaseViews(ns,c("X","y"))