Python Pandas - Transformation of Group Data

---

---

Transformation of group data is refers to applying a function to each group and producing the results with the same index structure as the original data. Unlike aggregations, transformations do not reduce the dimensionality of the data, therefore we will get the original group structure. Common transformation methods include calculating cumulative sums, differences between values, and filling missing values within groups.

In data analysis, group transformations allow us to modify groups of data without changing the grouping structure. In Pandas, thetransform() method in thegroupby() operations provides a convenient way to apply transformations to each group independently.

In this tutorial we will learn about transformation of group data in Pandas, using thetransform() function and built-in transformation methods.

Built-in Transformation Methods in Pandas GroupBy

Pandas provides various built-in transformation functions that can be applied to each group are −

• Cumulative Sum (cumsum()): Calculates cumulative sum.

• Difference (diff()): Computes the difference between adjacent elements within each group.

• Cumulative Count (cumcount()): Calculates the cumulative count within each group.

• Cumulative Max (cummax()): Calculates the cumulative max within each group.

• Cumulative Min (cummin()): Finds the cumulative minimum value within each group.

• Cumulative Product (cumprod()): Calculates the cumulative product.

• Back Fill (bfill()): Back fill the missing (NA) values within each group.

• Forward Fill (ffill()): Forward fill the missing (NA) values within each group.

• Percentage Change (pct_change()): Calculates the percentage change between consecutive values.

• Rank (rank()): Ranks each value within the group.

• Shift (shift()): Shifts values up or down within each group.

Example

Here is a basis example of applying the built-in transformation functions to the Groupby data in Pandas.

# import the pandas libraryimport pandas as pdimport numpy as npipl_data = {'Team': ['Riders', 'Riders', 'Devils', 'Devils', 'Kings',   'kings', 'Kings', 'Kings', 'Riders', 'Royals', 'Royals', 'Riders'],   'Rank': [1, 2, 2, 3, 3,4 ,1 ,1,2 , 4,1,2],   'Year': [2014,2015,2014,2015,2014,2015,2016,2017,2016,2014,2015,2017],   'Points':[876,789,863,673,741,812,756,788,694,701,804,690]}df = pd.DataFrame(ipl_data)# Display the Original DataFrameprint("Original DataFrame:")print(df)# Grouping data by 'Team' and applying transformationsgrouped = df.groupby("Team")["Points"]# Cumulative sum within each groupcumulative_sum = grouped.cumsum()print("Cumulative Sum:\n", cumulative_sum)# Difference within each groupdifference = grouped.diff()print("\nDifference:\n", difference)

Following is the output of the above code −

Original DataFrame:

	Team	Rank	Year	Points
0	Riders	1	2014	876
1	Riders	2	2015	789
2	Devils	2	2014	863
3	Devils	3	2015	673
4	Kings	3	2014	741
5	kings	4	2015	812
6	Kings	1	2016	756
7	Kings	1	2017	788
8	Riders	2	2016	694
9	Royals	4	2014	701
10	Royals	1	2015	804
11	Riders	2	2017	690

Cumulative Sum:0 8761 16652 8633 15364 7415 8126 14977 22858 23599 70110 150511 3049Name: Points, dtype: int64Difference:0 NaN1 -87.02 NaN3 -190.04 NaN5 NaN6 15.07 32.08 -95.09 NaN10 103.011 -4.0Name: Points, dtype: float64

The cumulative sum adds each row's points to the previous, while the diff() method finds the difference between consecutive values.

Applying Transformation Using the transform()

Thetransform() method in Pandas is used to apply both built-in (as a string) and user-defined functions to each group. This method broadcasts the transformed data across the group, resulting the same index alignment of the original DataFrame.

Example: Applying Built-in Functions with transform()

TheDataFrameGroupBy.transform() method can accept string aliases for built-in functions, such assum,cumsum, andmean. Here's an example of applying thecumsum andcummin transformation functions with theDataFrameGroupBy.transform() method.

# import the pandas libraryimport pandas as pdimport numpy as npipl_data = {'Team': ['Riders', 'Riders', 'Devils', 'Devils', 'Kings',   'kings', 'Kings', 'Kings', 'Riders', 'Royals', 'Royals', 'Riders'],   'Rank': [1, 2, 2, 3, 3,4 ,1 ,1,2 , 4,1,2],   'Year': [2014,2015,2014,2015,2014,2015,2016,2017,2016,2014,2015,2017],   'Points':[876,789,863,673,741,812,756,788,694,701,804,690]}df = pd.DataFrame(ipl_data)# Grouping data by 'Team' and applying transformationsgrouped = df.groupby("Team")[["Points"]]# Using transform with a built-in functioncumulative_min= grouped.transform("cummin")print("Cumulative Min:")print(cumulative_min)# Using transform() with a cumulative sumcumulative_sum= grouped.transform("cumsum")print("\nCumulative Sum:")print(cumulative_sum)

Following is the output of the above code −

Cumulative Min:    Points0      8761      7892      8633      6734      7415      8126      7417      7418      6949      70110     70111     690Cumulative Sum:    Points0      8761     16652      8633     15364      7415      8126     14977     22858     23599      70110    150511    3049

Applying User-Defined Functions Using the transform()

In addition to built-in methods, thetransform() function also accepts custom functions defined by the user for customized transformations within each group. The user-defined function should return a result that matches the size of the group chunk.

Example

The following example demonstrates applying the user-defined functions using thetransform() method to the grouped data.

# import the pandas libraryimport pandas as pdimport numpy as npipl_data = {'Team': ['Riders', 'Riders', 'Devils', 'Devils', 'Kings',   'kings', 'Kings', 'Kings', 'Riders', 'Royals', 'Royals', 'Riders'],   'Rank': [1, 2, 2, 3, 3,4 ,1 ,1,2 , 4,1,2],   'Year': [2014,2015,2014,2015,2014,2015,2016,2017,2016,2014,2015,2017],   'Points':[876,789,863,673,741,812,756,788,694,701,804,690]}df = pd.DataFrame(ipl_data)grouped = df.groupby('Team')score = lambda x: (x - x.mean()) / x.std()*10print(grouped.transform(score))

Itsoutput is as follows −

i	Rank	Year	Points
0	-15.000000	-11.618950	12.843272
1	5.000000	-3.872983	3.020286
2	-7.071068	-7.071068	7.071068
3	7.071068	7.071068	-7.071068
4	11.547005	-10.910895	-8.608621
5	NaN	NaN	NaN
6	-5.773503	2.182179	-2.360428
7	-5.773503	8.728716	10.969049
8	5.000000	3.872983	-7.705963
9	7.071068	-7.071068	-7.071068
10	-7.071068	7.071068	7.071068
11	5.000000	11.618950	-8.157595