Perform the Same Processing on Multiple Sources

Suppose you want a function that’s equally happy grouping data from either a pandas or Polars DataFrame, or from any other Narwhals-supported format. Create a file nameduniversal_processing.py in your project’s folder, and add in auniversal_groupby_v1() function as shown:

importnarwhalsasnwfromnarwhals.typingimportIntoFrameTdefuniversal_groupby_v1(df:IntoFrameT)->IntoFrameT:return(nw.from_native(df).group_by("party_name").agg(nw.col("last_name").count()).sort("party_name").to_native())

To begin with, you import the Narwhals library into your code using its standardnw alias. You’ve also decided to use theIntoFrameT type hint, so you’ve imported that fromnarwhals.typing. TheIntoFrameTtype hint indicates an object that’s convertible to either a Narwhals DataFrame or LazyFrame. Your function will accept an instance of this type as its input and return it back to the caller as output.

Youruniversal_groupby_v1() function takes a single parameter,df, of typeIntoFrameT. The function then uses thenw.from_native() function to add a wrapper around the original input. This wrapper allows the Narwhals API to access your DataFrame. The specific object that Narwhals creates depends on the original input format.

Narwhals then performs a grouping operation. In this example, it groups the data byparty_name, counts the number of entries in thelast_name column for each group, and then sorts the results byparty_name.

If you’re familiar withPolars expressions, then this syntax will be immediately familiar to you. Remember, Narwhals uses a subset of Polars. TheAPI Completeness section of the documentation will tell you precisely what is or isn’t supported.

Finally, Narwhals takes the result of its analysis and uses.to_native() to return it in its original form. So if you pass in a Polars DataFrame, that’s what you’ll get back, and similarly for any other supported format.

One final point to note is that theuniversal_groupby_v1() function requires only theNarwhals library and the optionalnarwhals.typing.IntoFrameT. Nowhere do you import the source data’s library.

Now it’s time to test your function. Open aPython REPL within your project’s folder and run the following code:

>>>importpandasaspd>>>importpolarsaspl>>>fromuniversal_processingimportuniversal_groupby_v1>>>presidents_pd=pd.read_parquet("presidents.parquet")>>>presidents_pl=pl.read_parquet("presidents.parquet")

After importing both the pandas and Polars libraries, you import youruniversal_groupby_v1() function. You also create both thepresidents_pd pandas DataFrame and thepresidents_pl Polars DataFrame.

To test youruniversal_groupby_v1() function, you first pass in a pandas DataFrame:

>>>universal_groupby_v1(presidents_pd)              party_name  last_name3             Democratic         172  Democratic Republican          41             Federalist          15             Republican         200           Unaffiliated          14                   Whig          4

As you can see from the results, Narwhals can cope with your pandas DataFrame. The result is a pandas DataFrame because the function uses.to_native() to return the analyzed data in its original format.

Next, you’ll see if your function can also cope with a Polars DataFrame:

>>>universal_groupby_v1(presidents_pl)shape: (6, 2)┌───────────────────────┬───────────┐│ party_name            ┆ last_name ││ ---                   ┆ ---       ││ str                   ┆ u32       │╞═══════════════════════╪═══════════╡│ Democratic            ┆ 17        ││ Democratic Republican ┆ 4         ││ Federalist            ┆ 1         ││ Republican            ┆ 20        ││ Unaffiliated          ┆ 1         ││ Whig                  ┆ 4         │└───────────────────────┴───────────┘

Yes, it can. This time, your function has returned a Polars DataFrame containing the same result.

Use Some Syntactic Sugar

Although the previous example works perfectly well, the Narwhals library provides some useful decoratorsyntactic sugar to make your code simpler and more readable.

Add a new version of your earlier function nameduniversal_groupby_v2() into your existinguniversal_processing.py file:

importnarwhalsasnwfromnarwhals.typingimportFrameT,IntoFrameT# ...@nw.narwhalifydefuniversal_groupby_v2(df:FrameT)->FrameT:return(df.group_by("party_name").agg(nw.col("last_name").count()).sort("party_name"))

This time, you’ve made use of the@nw.narwhalify functiondecorator to simplify your code. Usually, code needs to explicitly convert the DataFrame to a Narwhals format, analyze it, and then convert it back again to its original format before returning it. Because@nw.narwhalify does both conversions for you implicitly, you can concentrate on the data analysis steps.

Take a look back at the example above, and you’ll see neitherfrom_native() nor.to_native() is explicitly used.

This time, you use theFrameT type hint instead ofIntoFrameT. The Narwhals documentation definesFrameT as a type that represents anw.DataFrame ornw.LazyFrame. The documentation favorsFrameT when@nw.narwhalify is used. Because type hints aren’t enforced, theIntoFrameT hint would also work equally well.

But does version two of your function still produce the same result as version one did? Before you can find out, you’ll first need to exit your current Python REPL session by typingexit or pressingCtrl+D, and then start a fresh session. Now, for the moment of truth:

>>>importpandasaspd>>>fromuniversal_processingimportuniversal_groupby_v2>>>presidents_pd=pd.read_parquet("presidents.parquet")>>>universal_groupby_v2(presidents_pd)              party_name  last_name3             Democratic         172  Democratic Republican          41             Federalist          15             Republican         200           Unaffiliated          14                   Whig          4

It certainly gives you what you’d expect when a pandas DataFrame has been passed to it. Feel free to try it using your earlierpresidents_pl DataFrame. You shouldn’t be disappointed.

Return a Different Output Format

So far, the returned DataFrame has been of the same type as the original DataFrame that you passed into your function. The DataFrame’s.to_native() method ensured this. You can specify an alternative output format using one of a range of available methods, such as.to_pandas() and.to_polars(), but you can’t use the@nw.narwhalify decorator.

You’ll next create a third version of your function thatalways returns a Polars DataFrame. Add this new version to your existing file:

# ...defuniversal_groupby_v3(df:IntoFrameT)->IntoFrameT:return(nw.from_native(df).group_by("party_name").agg(nw.col("last_name").count()).sort("party_name").to_polars())

This version looks very similar to version one. However, you specify that you always want a Polars DataFrame returned by using.to_polars() instead of.to_native().

As usual, restart your Python REPL and test your new function:

>>>importpandasaspd>>>importpolarsaspl>>>fromuniversal_processingimportuniversal_groupby_v3>>>presidents_pd=pd.read_parquet("presidents.parquet")>>>presidents_pl=pl.read_parquet("presidents.parquet")>>>type(universal_groupby_v3(presidents_pd))<class 'polars.dataframe.frame.DataFrame'>>>>type(universal_groupby_v3(presidents_pl))<class 'polars.dataframe.frame.DataFrame'>

First of all, you pass a pandas DataFrame into your function, and then you pass a Polars DataFrame. In both cases, a Polars DataFrame is produced. Feel free to print both results to your console, and you’ll see that they’re indeed identical.

Work With Both Eager and Lazy Modes

If you’ve worked with Polars, you might be aware that it provides both eager and lazy APIs. Narwhals also supports both through itsnarwhals.DataFrame andnarwhals.LazyFrame classes.

In eager mode, a query is executed immediately, whereas in lazy mode, a query plan is created by analyzing the query and determining its most efficient execution plan. Execution of the query to obtain the result is deferred until the code requests it.

Despite its somewhat derogatory name,lazy evaluation delivers significant performance gains, particularly when handling large volumes of data.

In general, if you pass a DataFrame or LazyFrame into Narwhals, that’s what.to_native() will return to you. You can test this using either the first or second version of your function because both versions return an object in the same format as they receive it:

>>>importpolarsaspl>>>fromuniversal_processingimportuniversal_groupby_v1>>>presidents_df=pl.read_parquet("presidents.parquet")>>>presidents_lf=pl.scan_parquet("presidents.parquet")>>>type(universal_groupby_v1(presidents_df))<class 'polars.dataframe.frame.DataFrame'>>>>type(universal_groupby_v1(presidents_lf))<class 'polars.lazyframe.frame.LazyFrame'>

As you can see, by passing in a Polars DataFrame to youruniversal_analysis_v1() function, you get a DataFrame back, and similarly for passing a LazyFrame. However, as with Polars, you need to take care when using Narwhals’ functions with LazyFrames, because not all analysis can be performed in lazy mode. More specifically, any analysis that needs to read data can only work in eager mode.

To investigate a situation where a LazyFrame fails, you decide to create a newuniversal_pivot_v1() function that produces apivot table of your data:

# ...defuniversal_pivot_v1(df:IntoFrameT)->IntoFrameT:return(nw.from_native(df).pivot(on="party_name",index="century",values="last_name",aggregate_function="count",).to_native())

This function performs a pivot table operation on the presidential data, allowing you to see a count of the number of presidents from each party in each century. To do this, you use Narwhals’ version of.pivot().

To display a separate column for each political party, you passon="party_name", and to return a separate row for each of the different centuries, you passindex="century". With the rows and columns defined, you then tell.pivot() to produce a count of each century-and-party combination by counting each president’s last name. You useaggregate_function="count" andvalues="last_name" to do this.

You decide to try out your function with a Polars DataFrame, but you could also pass it a pandas DataFrame:

>>>importpolarsaspl>>>fromuniversal_processingimportuniversal_pivot_v1>>>presidents_df=pl.read_parquet("presidents.parquet")>>>universal_pivot_v1(presidents_df)shape: (4, 7)┌─────────┬──────────────┬────────────┬───────────────────────┬  \│ century ┆ Unaffiliated ┆ Federalist ┆ Democratic Republican ┆  \│ ---     ┆ ---          ┆ ---        ┆ ---                   ┆  \│ str     ┆ u32          ┆ u32        ┆ u32                   ┆  \╞═════════╪══════════════╪════════════╪═══════════════════════╪  \│ 18th    ┆ 1            ┆ 1          ┆ 0                     ┆  \│ 19th    ┆ 0            ┆ 0          ┆ 4                     ┆  \│ 20th    ┆ 0            ┆ 0          ┆ 0                     ┆  \│ 21st    ┆ 0            ┆ 0          ┆ 0                     ┆  \└─────────┴──────────────┴────────────┴───────────────────────┴  \────────────┬──────┬────────────┐ Democratic ┆ Whig ┆ Republican │ ---        ┆ ---  ┆ ---        │ u32        ┆ u32  ┆ u32        │════════════╪══════╪════════════╡ 0          ┆ 0    ┆ 0          │ 8          ┆ 4    ┆ 7          │ 7          ┆ 0    ┆ 10         │ 2          ┆ 0    ┆ 3          │────────────┴──────┴────────────┘

It worked! Next, you decide to try passing in a LazyFrame:

>>>presidents_lf=pl.scan_parquet("presidents.parquet")>>>universal_pivot_v1(presidents_lf)Traceback (most recent call last):...AttributeError:'LazyFrame' object has no attribute 'pivot'.⮑ Did you mean: 'unpivot'?

This time, your analysis fails. Instead of a result, you get anAttributeErrorexception. The problem, as the exception’s message points out, is that.pivot() doesn’t apply to LazyFrames. You can only use.pivot() with DataFrames because they contain the data that.pivot() needs to perform its calculations.

If you need to write Narwhals code that uses lazy mode, then make sure to check that the functionality you need is supported. The documentation’sAPI Completeness section provides bothsupported DataFrame methods andsupported LazyFrame methods, giving you a heads-up on what’s available in each mode. Also, theAPI Reference section provides details on how each method works.

If you want youruniversal_pivot() function to work with both DataFrames and LazyFrames, you could use the.collect() method on the LazyFrame passed to it before you pass it to.pivot(). To see this working, you decide to create auniversal_pivot_v2() function as shown:

# ...defuniversal_pivot_v2(df:IntoFrameT)->IntoFrameT:df=nw.from_native(df)ifisinstance(df,nw.LazyFrame):df=df.collect()returndf.pivot(on="party_name",index="century",values="last_name",aggregate_function="count",).to_native()

This time, version two of your function tests for the presence of a LazyFrame using Python’s built-inisinstance() function. If a LazyFrame is present, afterfrom_native() adds a Narwhals wrapper, you call.collect() to populate it with data and create a DataFrame. You then pass the DataFrame through the same pivoting operation as before.

If either a DataFrame or LazyFrame is passed touniversal_pivot_v2() , it’s analyzed in the same way as it was earlier withuniversal_pivot_v1(). Feel free to test it for yourself, and you’ll see it produces the same results in both cases.

The final aspect of Narwhals you should be aware of is that its syntax is similar to that of Polars. You’ll investigate this next.