Python Pandas - Statistical Functions

In data analysis, understanding the patterns and relationships within your data is crucial. Statistical methods in Pandas help to extract meaningful information, patterns and relationships from data, enabling you to make decisions and analyzing the behavior of data.

In this tutorial, we will explore some key statistical functions available in Pandas. These functions are designed to help you summarize and understand your data in different ways. Whether you want to measure changes over time, assess relationships between variables, or rank your data, Pandas provides the tools you need.

Analyzing Fractional Change

Thepct_change() function in Pandas calculates the fractional change between the current and a prior element. It is a valuable tool for understanding how data evolves over time, commonly used in financial data analysis.

Example

Following is the example of calculating the fractional change between the current and a prior element of Pandas Series and DataFrame using thepct_change() method.

import pandas as pdimport numpy as nps = pd.Series([1,2,3,4,5,4])print(s.pct_change())df = pd.DataFrame(np.random.randn(5, 2))print(df.pct_change())

Itsoutput is as follows −

0        NaN1   1.0000002   0.5000003   0.3333334   0.2500005  -0.200000dtype: float64            0          10         NaN        NaN1  -15.151902   0.1747302  -0.746374   -1.4490883  -3.582229   -3.1658364   15.601150  -1.860434

By default, thepct_change() operates on columns; if you want to apply the same row wise, then useaxis=1() argument.

Understanding Covariance

Covariance measures how two variables change together. In Pandas, the cov() method computes the covariance between two Series objects or across all pairs of columns in a DataFrame.

Example

Here is the example of calculating the covariance between two Series objects using theSeries.cov() method.

import pandas as pdimport numpy as nps1 = pd.Series(np.random.randn(10))s2 = pd.Series(np.random.randn(10))print(s1.cov(s2))

Itsoutput is as follows −

0.02429227824398636

Example

Covariance method when applied on a DataFrame, computescov() between all the columns.

import pandas as pdimport numpy as npframe = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])print(frame['a'].cov(frame['b']))print(frame.cov())

Itsoutput is as follows −

-0.58312921152741437           a           b           c           d            ea   1.780628   -0.583129   -0.185575    0.003679    -0.136558b  -0.583129    1.297011    0.136530   -0.523719     0.251064c  -0.185575    0.136530    0.915227   -0.053881    -0.058926d   0.003679   -0.523719   -0.053881    1.521426    -0.487694e  -0.136558    0.251064   -0.058926   -0.487694     0.960761

Note: Observe thecov betweena andb column in the first statement and the same is the value returned by cov on DataFrame.

Measuring Correlation

Correlation shows the linear relationship between any two array of values (series). Pandascorr() function supports different correlation methods, including Pearson (default), Spearman, and Kendall.

Example

This example calculates the correlation between two columns of a DataFrame using thecorr() function.

import pandas as pdimport numpy as npframe = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])print(frame['a'].corr(frame['b']))print(frame.corr())

Itsoutput is as follows −

-0.383712785514           a          b          c          d           ea   1.000000  -0.383713  -0.145368   0.002235   -0.104405b  -0.383713   1.000000   0.125311  -0.372821    0.224908c  -0.145368   0.125311   1.000000  -0.045661   -0.062840d   0.002235  -0.372821  -0.045661   1.000000   -0.403380e  -0.104405   0.224908  -0.062840  -0.403380    1.000000

If any non-numeric column is present in the DataFrame, it is excluded automatically.

Ranking Data

Therank() function assigns ranks to elements in a Series or DataFrame. In cases where multiple elements have the same value, it assigns the average rank by default, but this behavior can be adjusted.

Example

Following is the example of calculating the numerical data ranks of the Series elements using therank() method.

import pandas as pdimport numpy as nps = pd.Series(np.random.randn(5), index=list('abcde'))s['d'] = s['b'] # so there's a tieprint(s.rank())

Itsoutput is as follows −

a  1.0b  3.5c  2.0d  3.5e  5.0dtype: float64

Rank optionally takes a parameter ascending which by default is true; when false, data is reverse-ranked, with larger values assigned a smaller rank. It supports different tie-breaking methods, specified with the method parameter −

• average: average rank of tied group

• min: lowest rank in the group

• max: highest rank in the group

• first: ranks assigned in the order they appear in the array