numpy.memmap#

classnumpy.memmap(filename,dtype=<class'numpy.ubyte'>,mode='r+',offset=0,shape=None,order='C')[source]#

Create a memory-map to an array stored in abinary file on disk.

Memory-mapped files are used for accessing small segments of large fileson disk, without reading the entire file into memory. NumPy’smemmap’s are array-like objects. This differs from Python’smmapmodule, which uses file-like objects.

This subclass of ndarray has some unpleasant interactions withsome operations, because it doesn’t quite fit properly as a subclass.An alternative to using this subclass is to create themmapobject yourself, then create an ndarray with ndarray.__new__ directly,passing the object created in its ‘buffer=’ parameter.

This class may at some point be turned into a factory functionwhich returns a view into an mmap buffer.

Flush the memmap instance to write the changes to the file. Currently thereis no API to close the underlyingmmap. It is tricky to ensure theresource is actually closed, since it may be shared between differentmemmap instances.

Parameters:

filenamestr, file-like object, or pathlib.Path instance

The file name or file object to be used as the array data buffer.

dtypedata-type, optional

The data-type used to interpret the file contents.Default isuint8.

mode{‘r+’, ‘r’, ‘w+’, ‘c’}, optional

The file is opened in this mode:

‘r’	Open existing file for reading only.
‘r+’	Open existing file for reading and writing.
‘w+’	Create or overwrite existing file for reading and writing.Ifmode=='w+' thenshape must also be specified.
‘c’	Copy-on-write: assignments affect data in memory, butchanges are not saved to disk. The file on disk isread-only.

Default is ‘r+’.

offsetint, optional

In the file, array data starts at this offset. Sinceoffset ismeasured in bytes, it should normally be a multiple of the byte-sizeofdtype. Whenmode!='r', even positive offsets beyond end offile are valid; The file will be extended to accommodate theadditional data. By default,memmap will start at the beginning ofthe file, even iffilename is a file pointerfp andfp.tell()!=0.

shapeint or sequence of ints, optional

The desired shape of the array. Ifmode=='r' and the numberof remaining bytes afteroffset is not a multiple of the byte-sizeofdtype, you must specifyshape. By default, the returned arraywill be 1-D with the number of elements determined by file sizeand data-type.

Changed in version 2.0:The shape parameter can now be any integer sequence type, previouslytypes were limited to tuple and int.

order{‘C’, ‘F’}, optional

Specify the order of the ndarray memory layout:row-major, C-style orcolumn-major,Fortran-style. This only has an effect if the shape isgreater than 1-D. The default order is ‘C’.

See also

lib.format.open_memmap

Create or load a memory-mapped.npy file.

Notes

The memmap object can be used anywhere an ndarray is accepted.Given a memmapfp,isinstance(fp,numpy.ndarray) returnsTrue.

Memory-mapped files cannot be larger than 2GB on 32-bit systems.

When a memmap causes a file to be created or extended beyond itscurrent size in the filesystem, the contents of the new part areunspecified. On systems with POSIX filesystem semantics, the extendedpart will be filled with zero bytes.

Examples

Try it in your browser!

>>>importnumpyasnp>>>data=np.arange(12,dtype='float32')>>>data.resize((3,4))

This example uses a temporary file so that doctest doesn’t writefiles to your directory. You would use a ‘normal’ filename.

>>>fromtempfileimportmkdtemp>>>importos.pathaspath>>>filename=path.join(mkdtemp(),'newfile.dat')

Create a memmap with dtype and shape that matches our data:

>>>fp=np.memmap(filename,dtype='float32',mode='w+',shape=(3,4))>>>fpmemmap([[0., 0., 0., 0.],        [0., 0., 0., 0.],        [0., 0., 0., 0.]], dtype=float32)

Write data to memmap array:

>>>fp[:]=data[:]>>>fpmemmap([[  0.,   1.,   2.,   3.],        [  4.,   5.,   6.,   7.],        [  8.,   9.,  10.,  11.]], dtype=float32)

>>>fp.filename==path.abspath(filename)True

Flushes memory changes to disk in order to read them back

>>>fp.flush()

Load the memmap and verify data was stored:

>>>newfp=np.memmap(filename,dtype='float32',mode='r',shape=(3,4))>>>newfpmemmap([[  0.,   1.,   2.,   3.],        [  4.,   5.,   6.,   7.],        [  8.,   9.,  10.,  11.]], dtype=float32)

Read-only memmap:

>>>fpr=np.memmap(filename,dtype='float32',mode='r',shape=(3,4))>>>fpr.flags.writeableFalse

Copy-on-write memmap:

>>>fpc=np.memmap(filename,dtype='float32',mode='c',shape=(3,4))>>>fpc.flags.writeableTrue

It’s possible to assign to copy-on-write array, but values are onlywritten into the memory copy of the array, and not written to disk:

>>>fpcmemmap([[  0.,   1.,   2.,   3.],        [  4.,   5.,   6.,   7.],        [  8.,   9.,  10.,  11.]], dtype=float32)>>>fpc[0,:]=0>>>fpcmemmap([[  0.,   0.,   0.,   0.],        [  4.,   5.,   6.,   7.],        [  8.,   9.,  10.,  11.]], dtype=float32)

File on disk is unchanged:

>>>fprmemmap([[  0.,   1.,   2.,   3.],        [  4.,   5.,   6.,   7.],        [  8.,   9.,  10.,  11.]], dtype=float32)

Offset into a memmap:

>>>fpo=np.memmap(filename,dtype='float32',mode='r',offset=16)>>>fpomemmap([  4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.], dtype=float32)

Go BackOpen In Tab

Attributes:

filenamestr or pathlib.Path instance

Path to the mapped file.

offsetint

Offset position in the file.

modestr

File mode.

Methods

flush()

Write any changes in the array to the file on disk.