Python lambda expressions unleashed

Carl Kadie, Ph.D., is a research developer in Microsoft Research/TnR working on Genomics.

Lambda expressions provide a way to pass functionality into a function. Sadly, Python puts two annoying restrictions on lambda expressions. First, lambdas can only contain anexpression, notstatements. Second, lambdas can’t be serialized to disk. This blog shows how we can work around these restrictions and unleash the full power of lambdas.

So what are lambda’s good for? Suppose, you have a list of words from a string.

In [1]:

"This is a test string from Carl".split()

Out[1]:

['This', 'is', 'a', 'test', 'string', 'from', 'Carl']

You can sort the words withsorted().

In [2]:

sorted("This is a test string from Carl".split())

Out[2]:

['Carl', 'This', 'a', 'from', 'is', 'string', 'test']

Notice, however, that all the capitalized words, sort before all the lower-case words. This can be fixed by passing a lambda expression as thekey argument to thesorted() function.

In [3]:

sorted("This is a test string from Carl".split(),key=lambdaword:word.lower())# `key=str.lower` also works.

Out[3]:

['a', 'Carl', 'from', 'is', 'string', 'test', 'This']

Lambda can be more complicated. Suppose we want to sort the words based on their (lower-case) back-to-front letters? As a reminder, here is a Python way to reverse the lower-case letters of a word:

In [4]:

str.lower("Hello")[::-1]

Out[4]:

'olleh'

And here is how to pass this functionality tosorted() using a lambda:

In [5]:

sorted("This is a test string from Carl".split(),key=lambdaword:word.lower()[::-1])

Out[5]:

['a', 'string', 'Carl', 'from', 'is', 'This', 'test']

But what if you want even more complex functionality? For example, functionality that requiresif statements and multiple lines with unique scoping? Sadly, Python restricts lambdas to expressions only. But there is a workaround!

Define a function that

defines an inner function and …
returns that inner function.

Note that the inner function can refer to variables in the outer function, giving you that private scoping.

In this examplelower_sorted() is the outer function. It has an argument calledback_to_front. Insidelower_sorted, we define and return an inner function calledinner_lower_sorted(). That inner function has multiple lines including anif statement that referencesback_to_front.

In [6]:

deflower_sorted(back_to_front=False):definner_lower_sorted(word):result=word.lower()ifback_to_front:#The inner function can refer to outside variablesresult=result[::-1]returnresultreturninner_lower_sortedprint(sorted("This is a test string from Carl".split(),key=lower_sorted()))print(sorted("This is a test string from Carl".split(),key=lower_sorted(back_to_front=True)))

['a', 'Carl', 'from', 'is', 'string', 'test', 'This']['a', 'string', 'Carl', 'from', 'is', 'This', 'test']

You may find lambdas and these inner functions handy enough that you’d like to serialize one to disk for use later. Sadly, if you try to seralize withpickle module, you’ll get an error message like “TypeError: can’t pickle function objects”.

A nice workaround is to use thedill project in place ofpickle. Thedill project is a third-party package that is now included in the standardAnaconda distribution. Here is an example:

In [7]:

!pip install dill

Requirement already satisfied (use --upgrade to upgrade): dill in /home/nbcommon/anaconda3_23/lib/python3.4/site-packagesYou are using pip version 8.1.1, however version 8.1.2 is available.You should consider upgrading via the 'pip install --upgrade pip' command.

In [8]:

importdillaspicklewithopen("temp.p",mode="wb")asf:pickle.dump(lower_sorted(back_to_front=True),f)withopen("temp.p",mode="rb")asf:some_functionality=pickle.load(f)sorted("This is a test string from Carl".split(),key=some_functionality)

Out[8]:

['a', 'string', 'Carl', 'from', 'is', 'This', 'test']

Serialization of lambdas and these inner functions opens exciting possibilities. For example, we use it in one of our libraries to run work in different processes and even on different machines in a cluster.

We’ve seen that lambdas are a handy way to pass functionality into a function. Python’s implementation of lambdas has two restrictions, but each restriction has a workaround.

Multiple lines not allowed.
- Workaround: Define a function that defines and returns an inner function. The inner function can use variables outside itself.
Can’t pickle lambdas or inner functions.
- Workaround: Replacepickle withdill.

Python offers features such aslist comprehensions that makes lambdas less used that in other languages. When you do need lambdas, however, they will now be unleashed.