Python Pandas - Sparse Data

Sparse data structures in Pandas are used to store data in a compressed format. They are particularly useful when you have large datasets with many repeated values (such as NaN). The compression is achieved by not storing these repeated values, making the storage more efficient.

Pandas provides specialized data structures for efficiently storing sparse data. Unlike typical sparse structures that mostly store zeros, Pandas' sparse objects allow you to compress data by omitting any values matching a specific fill value (like NaN). This compression leads to significant memory savings, especially with large datasets.

In this tutorial we will learn about the Sparse objects in pandas.

Sparse Arrays and Dtypes

Pandas offers theSparseArray class for handling sparse data at the array level. You can access the dtype information, which includes both the data type of the stored elements and the fill value.

Example

Let's see an example of creating a series with sparse data structures and verifying it's datatype.

import pandas as pdimport numpy as np# Generate random dataarr = np.random.randn(10)arr[2:-2] = np.nan  # Introduce NaN values# Convert to a sparse Seriessparse_series = pd.Series(pd.arrays.SparseArray(arr))print("Output sparse Series:\n",sparse_series)print("DataType of the Series:",sparse_series.dtype)

Itsoutput is as follows −

Output sparse Series: 0    0.7638301    0.8213922         NaN3         NaN4         NaN5         NaN6         NaN7         NaN8    0.5324639    1.169153dtype: Sparse[float64, nan]DataType of the Series: Sparse[float64, nan]

Notice the dtypeSparse[float64, nan]. Thenan indicates thatNaN values are not actually stored, only the non-NaN elements are.

Memory Efficiency with Sparse DataFrames

Sparse objects are ideal for enhancing memory efficiency when working with large datasets containing many NaN values.

Example

Let us now assume you had a DataFrame with mostly NaN values and execute the following code −

import pandas as pdimport numpy as np# Create a DataFrame and introduce NaN values df = pd.DataFrame(np.random.randn(10000, 4))df.iloc[:9998] = np.nan# Convert to a sparse DataFramesparse_df = df.astype(pd.SparseDtype("float", np.nan))# Display the first few rows and data typesprint("Sparse DataFrame: \n",sparse_df.head())print("\nDataType:\n",sparse_df.dtypes)# Compare memory usageprint("\nMemory Comparison:")print('Dense: {:.2f} KB'.format(df.memory_usage().sum() / 1e3))print('Sparse: {:.2f} KB'.format(sparse_df.memory_usage().sum() / 1e3))

Itsoutput is as follows −

Sparse DataFrame:      0   1   2   30 NaN NaN NaN NaN1 NaN NaN NaN NaN2 NaN NaN NaN NaN3 NaN NaN NaN NaN4 NaN NaN NaN NaNDataType: 0    Sparse[float64, nan]1    Sparse[float64, nan]2    Sparse[float64, nan]3    Sparse[float64, nan]dtype: objectMemory Comparison:Dense: 320.13 KBSparse: 0.22 KB

By converting the DataFrame to a sparse format, memory usage is significantly reduced.

Converting Sparse Arrays to Dense

Any sparse object can be converted back to the standard dense form by callingsparse.to_dense() −

import pandas as pdimport numpy as npdf = pd.DataFrame(np.random.randn(10, 2), columns=['A', 'B'])df.iloc[:5] = np.nan# Convert to a sparse DataFramesparse_df = df.astype(pd.SparseDtype("float", np.nan))# Display input the sparse objectprint("sparse object:\n",sparse_df.dtypes)result = sparse_df.sparse.to_dense()# Output Denseprint("Output Dense:\n", result.dtypes)

Itsoutput is as follows −

sparse object: A    Sparse[float64, nan]B    Sparse[float64, nan]dtype: objectOutput Dense: A    float64B    float64dtype: object

Working with Sparse Accessor

Pandas offers a.sparse accessor to work with sparse data structures, similar to.str for string data or.dt for datetime data.

Sparse data should have the same dtype as its dense representation. Currently,float64,int64 andbooldtypes are supported. Depending on the originaldtype,fill_value default changes −

• float64 − np.nan

• int64 − 0

• bool − False

Example

Let us execute the following code to understand the working of the sparse accessor −

import pandas as pdimport numpy as np# Create a sparse objectsparse_series = pd.Series([0, 0, 1, 2], dtype="Sparse[int]")# Display input of the sparse objectprint("sparse object:\n",sparse_series)# Output of working with the Sparse Accessorprint("Percent of non-fill_value points:",sparse_series.sparse.density)print("Fill value:", sparse_series.sparse.fill_value)print("The number of non- fill_value points:", sparse_series.sparse.npoints)print("Non fill value:", sparse_series.sparse.sp_values)

Itsoutput is as follows −

sparse object:0    01    02    13    2dtype: Sparse[int64, 0]Percent of non-fill_value points: 0.5Fill value: 0The number of non- fill_value points: 2Non fill value: [1 2]