Creating an array#

Zarr has several functions for creating arrays. For example:

>>>importzarr>>>store=zarr.storage.MemoryStore()>>>z=zarr.create_array(store=store,shape=(10000,10000),chunks=(1000,1000),dtype='int32')>>>z<Array memory://... shape=(10000, 10000) dtype=int32>

The code above creates a 2-dimensional array of 32-bit integers with 10000 rowsand 10000 columns, divided into chunks where each chunk has 1000 rows and 1000columns (and so there will be 100 chunks in total). The data is written to azarr.storage.MemoryStore (e.g. an in-memory dict). SeePersistent arrays for details on storing arrays in other stores, and seeArray data types for an in-depth look at the data types supported by Zarr.

For a complete list of array creation routines see thezarrmodule documentation.

Reading and writing data#

Zarr arrays support a similar interface toNumPyarrays for reading and writing data. For example, the entire array can be filledwith a scalar value:

>>>z[:]=42

Regions of the array can also be written to, e.g.:

>>>importnumpyasnp>>>>>>z[0,:]=np.arange(10000)>>>z[:,0]=np.arange(10000)

The contents of the array can be retrieved by slicing, which will load therequested region into memory as a NumPy array, e.g.:

>>>z[0,0]array(0, dtype=int32)>>>z[-1,-1]array(42, dtype=int32)>>>z[0,:]array([   0,    1,    2, ..., 9997, 9998, 9999],      shape=(10000,), dtype=int32)>>>z[:,0]array([   0,    1,    2, ..., 9997, 9998, 9999],      shape=(10000,), dtype=int32)>>>z[:]array([[   0,    1,    2, ..., 9997, 9998, 9999],       [   1,   42,   42, ...,   42,   42,   42],       [   2,   42,   42, ...,   42,   42,   42],       ...,       [9997,   42,   42, ...,   42,   42,   42],       [9998,   42,   42, ...,   42,   42,   42],       [9999,   42,   42, ...,   42,   42,   42]],      shape=(10000, 10000), dtype=int32)

Read more about NumPy-style indexing can be found in theNumPy documentation.

Persistent arrays#

In the examples above, compressed data for each chunk of the array was stored inmain memory. Zarr arrays can also be stored on a file system, enablingpersistence of data between sessions. To do this, we can change the storeargument to point to a filesystem path:

>>>z1=zarr.create_array(store='data/example-1.zarr',shape=(10000,10000),chunks=(1000,1000),dtype='int32')

The array above will store its configuration metadata and all compressed chunkdata in a directory called'data/example-1.zarr' relative to the current workingdirectory. Thezarr.create_array() function provides a convenient wayto create a new persistent array or continue working with an existingarray. Note, there is no need to close an array: data are automaticallyflushed to disk, and files are automatically closed whenever an array is modified.

Persistent arrays support the same interface for reading and writing data,e.g.:

>>>z1[:]=42>>>z1[0,:]=np.arange(10000)>>>z1[:,0]=np.arange(10000)

Check that the data have been written and can be read again:

>>>z2=zarr.open_array('data/example-1.zarr',mode='r')>>>np.all(z1[:]==z2[:])np.True_

If you are just looking for a fast and convenient way to save NumPy arrays todisk then load back into memory later, the functionszarr.save() andzarr.load() may beuseful. E.g.:

>>>a=np.arange(10)>>>zarr.save('data/example-2.zarr',a)>>>zarr.load('data/example-2.zarr')array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Please note that there are a number of other options for persistent arraystorage, see theStorage Guide guide for more details.

Resizing and appending#

A Zarr array can be resized, which means that any of its dimensions can beincreased or decreased in length. For example:

>>>z=zarr.create_array(store='data/example-3.zarr',shape=(10000,10000),dtype='int32',chunks=(1000,1000))>>>z[:]=42>>>z.shape(10000, 10000)>>>z.resize((20000,10000))>>>z.shape(20000, 10000)

Note that when an array is resized, the underlying data are not rearranged inany way. If one or more dimensions are shrunk, any chunks falling outside thenew array shape will be deleted from the underlying store.

zarr.Array.append() is provided as a convenience function, which can beused to append data to any axis. E.g.:

>>>a=np.arange(10000000,dtype='int32').reshape(10000,1000)>>>z=zarr.create_array(store='data/example-4.zarr',shape=a.shape,dtype=a.dtype,chunks=(1000,100))>>>z[:]=a>>>z.shape(10000, 1000)>>>z.append(a)(20000, 1000)>>>z.append(np.vstack([a,a]),axis=1)(20000, 2000)>>>z.shape(20000, 2000)

Compressors#

A number of different compressors can be used with Zarr. Zarr includes Blosc,Zstandard and Gzip compressors. Additional compressors are available througha separate package calledNumCodecs which provides variouscompressor libraries including LZ4, Zlib, BZ2 and LZMA.Different compressors can be provided via thecompressors keywordargument accepted by all array creation functions. For example:

>>>compressors=zarr.codecs.BloscCodec(cname='zstd',clevel=3,shuffle=zarr.codecs.BloscShuffle.bitshuffle)>>>data=np.arange(100000000,dtype='int32').reshape(10000,10000)>>>z=zarr.create_array(store='data/example-5.zarr',shape=data.shape,dtype=data.dtype,chunks=(1000,1000),compressors=compressors)>>>z[:]=data>>>z.compressors(BloscCodec(typesize=4, cname=<BloscCname.zstd: 'zstd'>, clevel=3, shuffle=<BloscShuffle.bitshuffle: 'bitshuffle'>, blocksize=0),)

This array above will use Blosc as the primary compressor, using the Zstandardalgorithm (compression level 3) internally within Blosc, and with thebit-shuffle filter applied.

When using a compressor, it can be useful to get some diagnostics on thecompression ratio. Zarr arrays provide thezarr.Array.info propertywhich can be used to print useful diagnostics, e.g.:

>>>z.infoType               : ArrayZarr format        : 3Data type          : Int32(endianness='little')Fill value         : 0Shape              : (10000, 10000)Chunk shape        : (1000, 1000)Order              : CRead-only          : FalseStore type         : LocalStoreFilters            : ()Serializer         : BytesCodec(endian=<Endian.little: 'little'>)Compressors        : (BloscCodec(typesize=4, cname=<BloscCname.zstd: 'zstd'>, clevel=3, shuffle=<BloscShuffle.bitshuffle: 'bitshuffle'>, blocksize=0),)No. bytes          : 400000000 (381.5M)

Thezarr.Array.info_complete() method inspects the underlying store andprints additional diagnostics, e.g.:

>>>z.info_complete()Type               : ArrayZarr format        : 3Data type          : Int32(endianness='little')Fill value         : 0Shape              : (10000, 10000)Chunk shape        : (1000, 1000)Order              : CRead-only          : FalseStore type         : LocalStoreFilters            : ()Serializer         : BytesCodec(endian=<Endian.little: 'little'>)Compressors        : (BloscCodec(typesize=4, cname=<BloscCname.zstd: 'zstd'>, clevel=3, shuffle=<BloscShuffle.bitshuffle: 'bitshuffle'>, blocksize=0),)No. bytes          : 400000000 (381.5M)No. bytes stored   : 3558573 (3.4M)Storage ratio      : 112.4Chunks Initialized : 100

If you don’t specify a compressor, by default Zarr uses the Zstandardcompressor.

In addition to Blosc and Zstandard, other compression libraries can also be used. For example,here is an array using Gzip compression, level 1:

>>>data=np.arange(100000000,dtype='int32').reshape(10000,10000)>>>z=zarr.create_array(store='data/example-6.zarr',shape=data.shape,dtype=data.dtype,chunks=(1000,1000),compressors=zarr.codecs.GzipCodec(level=1))>>>z[:]=data>>>z.compressors(GzipCodec(level=1),)

Here is an example using LZMA fromNumCodecs with a custom filter pipeline including LZMA’sbuilt-in delta filter:

>>>importlzma>>>fromnumcodecs.zarr3importLZMA>>>>>>lzma_filters=[dict(id=lzma.FILTER_DELTA,dist=4),dict(id=lzma.FILTER_LZMA2,preset=1)]>>>compressors=LZMA(filters=lzma_filters)>>>data=np.arange(100000000,dtype='int32').reshape(10000,10000)>>>z=zarr.create_array(store='data/example-7.zarr',shape=data.shape,dtype=data.dtype,chunks=(1000,1000),compressors=compressors)>>>z.compressors(LZMA(codec_name='numcodecs.lzma', codec_config={'filters': [{'id': 3, 'dist': 4}, {'id': 33, 'preset': 1}]}),)

To disable compression, setcompressors=None when creating an array, e.g.:

>>>z=zarr.create_array(store='data/example-8.zarr',shape=(100000000,),chunks=(1000000,),dtype='int32',compressors=None)>>>z.compressors()

Filters#

In some cases, compression can be improved by transforming the data in someway. For example, if nearby values tend to be correlated, then shuffling thebytes within each numerical value or storing the difference between adjacentvalues may increase compression ratio. Some compressors provide built-in filtersthat apply transformations to the data prior to compression. For example, theBlosc compressor has built-in implementations of byte- and bit-shuffle filters,and the LZMA compressor has a built-in implementation of a deltafilter. However, to provide additional flexibility for implementing and usingfilters in combination with different compressors, Zarr also provides amechanism for configuring filters outside of the primary compressor.

Here is an example using a delta filter with the Blosc compressor:

>>>fromnumcodecs.zarr3importDelta>>>>>>filters=[Delta(dtype='int32')]>>>compressors=zarr.codecs.BloscCodec(cname='zstd',clevel=1,shuffle=zarr.codecs.BloscShuffle.shuffle)>>>data=np.arange(100000000,dtype='int32').reshape(10000,10000)>>>z=zarr.create_array(store='data/example-9.zarr',shape=data.shape,dtype=data.dtype,chunks=(1000,1000),filters=filters,compressors=compressors)>>>z.info_complete()Type               : ArrayZarr format        : 3Data type          : Int32(endianness='little')Fill value         : 0Shape              : (10000, 10000)Chunk shape        : (1000, 1000)Order              : CRead-only          : FalseStore type         : LocalStoreFilters            : (Delta(codec_name='numcodecs.delta', codec_config={'dtype': 'int32'}),)Serializer         : BytesCodec(endian=<Endian.little: 'little'>)Compressors        : (BloscCodec(typesize=4, cname=<BloscCname.zstd: 'zstd'>, clevel=1, shuffle=<BloscShuffle.shuffle: 'shuffle'>, blocksize=0),)No. bytes          : 400000000 (381.5M)No. bytes stored   : 826Storage ratio      : 484261.5Chunks Initialized : 0

For more information about available filter codecs, see theNumcodecs documentation.

Advanced indexing#

Zarr arrays support several methods for advanced or “fancy”indexing, which enable a subset of data items to be extracted or updated in anarray without loading the entire array into memory.

Note that although this functionality is similar to some of the advancedindexing capabilities available on NumPy arrays and on h5py datasets,the ZarrAPI for advanced indexing is different from both NumPy and h5py, so pleaseread this section carefully. For a complete description of the indexing API,see the documentation for thezarr.Array class.

>>>data=np.arange(10)**2>>>z=zarr.create_array(store='data/example-10.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>z[:]array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])>>>z.get_coordinate_selection([2,5])array([ 4, 25])

>>>z.set_coordinate_selection([2,5],[-1,-2])>>>z[:]array([ 0,  1, -1,  9, 16, -2, 36, 49, 64, 81])

>>>data=np.arange(15).reshape(3,5)>>>z=zarr.create_array(store='data/example-11.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>z[:]array([[ 0,  1,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, 13, 14]])>>>z.get_coordinate_selection(([0,2],[1,3]))array([ 1, 13])>>>z.set_coordinate_selection(([0,2],[1,3]),[-1,-2])>>>z[:]array([[ 0, -1,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, -2, 14]])

>>>z.vindex[[0,2],[1,3]]array([-1, -2])>>>z.vindex[[0,2],[1,3]]=[-3,-4]>>>z[:]array([[ 0, -3,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, -4, 14]])>>>z[[0,2],[1,3]]array([-3, -4])

>>>z[1,[1,3]]array([6, 8])>>>z[[1,1],[1,3]]array([6, 8])

>>>data=np.arange(10)**2>>>z=zarr.create_array(store='data/example-12.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>z[:]array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])>>>sel=np.zeros_like(z,dtype=bool)>>>sel[2]=True>>>sel[5]=True>>>z.get_mask_selection(sel)array([ 4, 25])>>>z.set_mask_selection(sel,[-1,-2])>>>z[:]array([ 0,  1, -1,  9, 16, -2, 36, 49, 64, 81])

>>>data=np.arange(15).reshape(3,5)>>>z=zarr.create_array(store='data/example-13.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>z[:]array([[ 0,  1,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, 13, 14]])>>>sel=np.zeros_like(z,dtype=bool)>>>sel[0,1]=True>>>sel[2,3]=True>>>z.get_mask_selection(sel)array([ 1, 13])>>>z.set_mask_selection(sel,[-1,-2])>>>z[:]array([[ 0, -1,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, -2, 14]])

>>>z.vindex[sel]array([-1, -2])>>>z.vindex[sel]=[-3,-4]>>>z[:]array([[ 0, -3,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, -4, 14]])

>>>data=np.arange(15).reshape(3,5)>>>z=zarr.create_array(store='data/example-14.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>z[:]array([[ 0,  1,  2,  3,  4],       [ 5,  6,  7,  8,  9],       [10, 11, 12, 13, 14]])>>>z.get_orthogonal_selection(([0,2],slice(None)))# select first and third rowsarray([[ 0,  1,  2,  3,  4],       [10, 11, 12, 13, 14]])>>>z.get_orthogonal_selection((slice(None),[1,3]))# select second and fourth columnsarray([[ 1,  3],       [ 6,  8],       [11, 13]])>>>z.get_orthogonal_selection(([0,2],[1,3]))# select rows [0, 2] and columns [1, 4]array([[ 1,  3],       [11, 13]])

>>>z.set_orthogonal_selection(([0,2],[1,3]),[[-1,-2],[-3,-4]])

>>>data=np.arange(15).reshape(3,5)>>>z=zarr.create_array(store='data/example-15.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>z.oindex[[0,2],:]# select first and third rowsarray([[ 0,  1,  2,  3,  4],       [10, 11, 12, 13, 14]])>>>z.oindex[:,[1,3]]# select second and fourth columnsarray([[ 1,  3],       [ 6,  8],       [11, 13]])>>>z.oindex[[0,2],[1,3]]# select rows [0, 2] and columns [1, 4]array([[ 1,  3],       [11, 13]])>>>z.oindex[[0,2],[1,3]]=[[-1,-2],[-3,-4]]>>>z[:]array([[ 0, -1,  2, -2,  4],       [ 5,  6,  7,  8,  9],       [10, -3, 12, -4, 14]])

>>>data=np.arange(15).reshape(3,5)>>>z=zarr.create_array(store='data/example-16.zarr',shape=data.shape,dtype=data.dtype)>>>z[:]=data>>>np.all(z.oindex[[0,2],:]==z[[0,2],:])np.True_

>>>data=np.arange(100).reshape(10,10)>>>z=zarr.create_array(store='data/example-17.zarr',shape=data.shape,dtype=data.dtype,chunks=(3,3))>>>z[:]=data

>>>z.get_block_selection(1)array([[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],       [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],       [50, 51, 52, 53, 54, 55, 56, 57, 58, 59]])

>>>z[3:6]array([[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],       [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],       [50, 51, 52, 53, 54, 55, 56, 57, 58, 59]])

>>>z.blocks[1]array([[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],       [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],       [50, 51, 52, 53, 54, 55, 56, 57, 58, 59]])

>>>z.blocks[0,1:3]array([[ 3,  4,  5,  6,  7,  8],       [13, 14, 15, 16, 17, 18],       [23, 24, 25, 26, 27, 28]])

>>>z=zarr.create_array(store='data/example-18.zarr',shape=(6,6),dtype=int,chunks=(2,2))

>>>z.set_block_selection((1,0),1)>>>z[...]array([[0, 0, 0, 0, 0, 0],       [0, 0, 0, 0, 0, 0],       [1, 1, 0, 0, 0, 0],       [1, 1, 0, 0, 0, 0],       [0, 0, 0, 0, 0, 0],       [0, 0, 0, 0, 0, 0]])

>>>z.blocks[:,2]=7>>>z[...]array([[0, 0, 0, 0, 7, 7],       [0, 0, 0, 0, 7, 7],       [1, 1, 0, 0, 7, 7],       [1, 1, 0, 0, 7, 7],       [0, 0, 0, 0, 7, 7],       [0, 0, 0, 0, 7, 7]])

>>>z.blocks[2,1:3]array([[0, 0, 7, 7],       [0, 0, 7, 7]])>>>>>>root=zarr.create_group('data/example-19.zarr')>>>foo=root.create_array(name='foo',shape=(1000,100),chunks=(10,10),dtype='float32')>>>bar=root.create_array(name='foo/bar',shape=(100,),dtype='int32')>>>foo[:,:]=np.random.random((1000,100))>>>bar[:]=np.arange(100)>>>root.tree()/└── foo (1000, 100) float32

Sharding#

Using small chunk shapes in very large arrays can lead to a very large number of chunks.This can become a performance issue for file systems and object storage.With Zarr format 3, a new sharding feature has been added to address this issue.

With sharding, multiple chunks can be stored in a single storage object (e.g. a file).Within a shard, chunks are compressed and serialized separately.This allows individual chunks to be read independently.However, when writing data, a full shard must be written in one go for optimalperformance and to avoid concurrency issues.That means that shards are the units of writing and chunks are the units of reading.Users need to configure the chunk and shard shapes accordingly.

Sharded arrays can be created by providing theshards parameter tozarr.create_array().

>>>a=zarr.create_array('data/example-20.zarr',shape=(10000,10000),shards=(1000,1000),chunks=(100,100),dtype='uint8')>>>a[:]=(np.arange(10000*10000)%256).astype('uint8').reshape(10000,10000)>>>a.info_complete()Type               : ArrayZarr format        : 3Data type          : UInt8()Fill value         : 0Shape              : (10000, 10000)Shard shape        : (1000, 1000)Chunk shape        : (100, 100)Order              : CRead-only          : FalseStore type         : LocalStoreFilters            : ()Serializer         : BytesCodec(endian=None)Compressors        : (ZstdCodec(level=0, checksum=False),)No. bytes          : 100000000 (95.4M)No. bytes stored   : 3981473 (3.8M)Storage ratio      : 25.1Shards Initialized : 100

In this example a shard shape of (1000, 1000) and a chunk shape of (100, 100) is used.This means that 10*10 chunks are stored in each shard, and there are 10*10 shards in total.Without theshards argument, there would be 10,000 chunks stored as individual files.

Missing features in 3.0#

The following features have not been ported to 3.0 yet.