Extracting Even Numbers

As a first example, say you need to process a list of integer numbers and build a new list containing the even numbers. Your first approach to this problem might be to use afor loop like this:

>>>numbers=[1,3,10,45,6,50]>>>defextract_even(numbers):...even_numbers=[]...fornumberinnumbers:...ifnumber%2==0:# Filtering condition...even_numbers.append(number)...returneven_numbers...>>>extract_even(numbers)[10, 6, 50]

Here,extract_even() takes an iterable of integer numbers and returns a list containing only those that are even. The conditional statement plays the role of a filter that tests every number to find out if it’s even or not.

When you run into code like this, you can extract the filtering logic into a small predicate function and use it withfilter(). This way, you can perform the same computation without using an explicit loop:

>>>numbers=[1,3,10,45,6,50]>>>defis_even(number):...returnnumber%2==0# Filtering condition...>>>list(filter(is_even,numbers))[10, 6, 50]

Here,is_even() takes an integer and returnsTrue if it’s even andFalse otherwise. The call tofilter() does the hard work and filters out the odd numbers. As a result, you get a list of the even numbers. This code is shorter and more efficient than its equivalentfor loop.

Finding Prime Numbers

Another interesting example might be to extract all theprime numbers in a given interval. To do that, you can start by coding a predicate function that takes an integer as an argument and returnsTrue if the number is prime andFalse otherwise. Here’s how you can do that:

>>>importmath>>>defis_prime(n):...ifn<=1:...returnFalse...foriinrange(2,int(math.sqrt(n))+1):...ifn%i==0:...returnFalse...returnTrue...>>>is_prime(5)True>>>is_prime(12)False

The filtering logic is now inis_prime(). The function iterates through the integers between2 and thesquare root ofn. Inside the loop, theconditional statement checks if the current number is divisible by any other in the interval. If so, then the function returnsFalse because the number isn’t prime. Otherwise, it returnsTrue to signal that the input number is prime.

Withis_prime() in place and tested, you can usefilter() to extract prime numbers from an interval like this:

>>>list(filter(is_prime,range(1,51)))[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]

This call tofilter() extracts all the prime numbers in the range between1 and50. The algorithm used inis_prime() comes from Wikipedia’s article aboutprimality tests. You can check out that article if you need more efficient approaches.

Removing Outliers in a Sample

When you’re trying todescribe and summarize a sample of data, you probably start by finding its mean, or average. The mean is a quite popularcentral tendency measurement and is often the first approach to analyzing a dataset. It gives you a quick idea of the center, orlocation, of the data.

In some cases, the mean isn’t a good enough central tendency measure for a given sample.Outliers are one of the elements that affect how accurate the mean is. Outliers aredata points that differ significantly from other observations in a sample or population. Other than that, there is no unique mathematical definition for them in statistics.

However, innormally distributed samples, outliers are often defined as data points that lie more than twostandard deviations from the sample mean.

Now suppose you have a normally distributed sample with some outliers that are affecting the mean accuracy. You’ve studied the outliers, and you know they’re incorrect data points. Here’s how you can use a couple of functions from thestatistics module along withfilter() to clean up your data:

>>>importstatisticsasst>>>sample=[10,8,10,8,2,7,9,3,34,9,5,9,25]>>># The mean before removing outliers>>>mean=st.mean(sample)>>>mean10.692307692307692>>>stdev=st.stdev(sample)>>>low=mean-2*stdev>>>high=mean+2*stdev>>>clean_sample=list(filter(lambdax:low<=x<=high,sample))>>>clean_sample[10, 8, 10, 8, 2, 7, 9, 3, 9, 5, 9, 25]>>># The mean after removing outliers>>>st.mean(clean_sample)8.75

In the highlighted line, thelambda function returnsTrue if a given data point lies between the mean and two standard deviations. Otherwise, it returnsFalse. When you filter thesample with this function,34 is excluded. After this cleanup, the mean of the sample has a significantly different value.

Validating Python Identifiers

You can also usefilter() with iterables containing nonnumeric data. For example, say you need to process a list ofstrings and extract those that are valid Pythonidentifiers. After doing some research, you find out that Python’sstr provides a method called.isidentifier() that can help you out with that validation.

Here’s how you can usefilter() along withstr.isidentifier() to quickly validate identifiers:

>>>words=["variable","file#","header","_non_public","123Class"]>>>list(filter(str.isidentifier,words))['variable', 'header', '_non_public']

In this case,filter() runs.isidentifier() on every string inwords. If the string is a valid Python identifier, then it’s included in the final result. Otherwise, the word is filtered out. Note that you need to usestr to access.isidentifier() in the call tofilter().

Finally, an interesting exercise might be to take the example further and check if the identifier is also akeyword. Go ahead and give it a try! Hint: you can use.kwlist from thekeyword module.

Finding Palindrome Words

An exercise that often arises when you’re getting familiar with Python strings is to findpalindrome words in a list of strings. A palindrome word reads the same backward as forward. Typical examples are “madam” and “racecar.”

To solve this problem, you’ll start by coding a predicate function that takes a string and checks if it reads the same in both directions, backward and forward. Here’s a possible implementation:

>>>defis_palindrome(word):...reversed_word="".join(reversed(word))...returnword.lower()==reversed_word.lower()...>>>is_palindrome("Racecar")True>>>is_palindrome("Python")False

Inis_palindrome(), you first reverse the originalword and store it inreversed_word. Then you return the result of comparing both words for equality. In this case, you use.lower() to prevent case-related differences. If you call the function with a palindrome word, then you getTrue. Otherwise, you getFalse.

You already have a working predicate function to identify palindrome words. Here’s how you can usefilter() to do the hard work:

>>>words=("filter","Ana","hello","world","madam","racecar")>>>list(filter(is_palindrome,words))['Ana', 'madam', 'racecar']

Cool! Your combination offilter() andis_palindrome() works properly. It’s also concise, readable, and efficient. Good job!

The Square of Even Numbers:filter() andmap()

Sometimes you need to take an iterable, process each of its items with atransformation function, and produce a new iterable with the resulting items. In that case, you can usemap(). The function has the following signature:

map(function,iterable[,iterable1,...,iterableN])

The arguments work like this:

1. function holds the transformation function. This function should take as many arguments as iterables you pass intomap().
2. iterable holds a Python iterable. Note that you can provide several iterables tomap(), but that’s optional.

map() appliesfunction to each item initerable to transform it into a different value with additional features. Thenmap() yields each transformed item on demand.

To illustrate how you can usefilter() along withmap(), say you need to compute the square value of all the even numbers in a given list. In that case, you can usefilter() to extract the even numbers and thenmap() to calculate the square values:

>>>numbers=[1,3,10,45,6,50]>>>defis_even(number):...returnnumber%2==0...>>>even_numbers=list(filter(is_even,numbers))>>>even_numbers[10, 6, 50]>>>list(map(lambdan:n**2,even_numbers))[100, 36, 2500]>>>list(map(lambdan:n**2,filter(is_even,numbers)))[100, 36, 2500]

First, you get the even numbers usingfilter() andis_even() just like you’ve done so far. Then you callmap() with alambda function that takes a number and returns its square value. The call tomap() applies thelambda function to each number ineven_numbers, so you get a list of square even numbers. The final example shows how to combinefilter() andmap() in a single expression.

The Sum of Even Numbers:filter() andreduce()

Another functional programming tool in Python isreduce(). Unlikefilter() andmap(), which are still built-in functions,reduce() was moved to thefunctools module. This function is useful when you need to apply a function to an iterable and reduce it to a single cumulative value. This kind of operation is commonly known as areduction or folding.

The signature ofreduce() is like this:

reduce(function,iterable,initial)

Here’s what the arguments mean:

1. function holds any Python callable that accepts two arguments and returns a single value.
2. iterable holds any Python iterable.
3. initial holds a value that serves as a starting point for the first partial computation or reduction. It’s an optional argument.

A call toreduce() starts by applyingfunction to the first two items initerable. This way, it computes the first cumulative result, called anaccumulator. Thenreduce() uses the accumulator and the third item initerable to compute the next cumulative result. The process continues until the function returns with a single value.

If you supply a value toinitial, thenreduce() runs the first partial computation usinginitial and the first item ofiterable.

Here’s an example that combinesfilter() andreduce() to cumulatively calculate the total sum of all the even numbers in a list:

>>>fromfunctoolsimportreduce>>>numbers=[1,3,10,45,6,50]>>>defis_even(number):...returnnumber%2==0...>>>even_numbers=list(filter(is_even,numbers))>>>reduce(lambdaa,b:a+b,even_numbers)66>>>reduce(lambdaa,b:a+b,filter(is_even,numbers))66

Here, the first call toreduce() computes the sum of all the even numbers thatfilter() provides. To do that,reduce() uses alambda function that adds two numbers at a time.

The final example shows how to chainfilter() andreduce() to produce the same result you got before.

Extracting Odd Numbers

You already coded a predicate function calledis_even() to check if a number is even or not. With that function and the help offilterfalse(), you can build an iterator that yields odd numbers without having to code anis_odd() function:

>>>fromitertoolsimportfilterfalse>>>numbers=[1,3,10,45,6,50]>>>defis_even(number):...returnnumber%2==0...>>>list(filterfalse(is_even,numbers))[1, 3, 45]

In this example,filterfalse() returns an iterator that yields the odd numbers from the input iterator. Note that the call tofilterfalse() is straightforward and readable.

Filtering Out NaN Values

Sometimes when you’re working withfloating-point arithmetic, you can face the issue of havingNaN (not a number) values. For example, say you’re calculating the mean of a sample of data that contains NaN values. If you use Python’sstatistics module for this computation, then you get the following result:

>>>importstatisticsasst>>>sample=[10.1,8.3,10.4,8.8,float("nan"),7.2,float("nan")]>>>st.mean(sample)nan

In this example, the call tomean() returnsnan, which isn’t the most informative value you can get. NaN values can have different origins. They can be due to invalid inputs, corrupted data, and so on. You should find the right strategy to deal with them in your applications. One alternative might be to remove them from your data.

Themath module provides a convenient function calledisnan() that can help you out with this problem. The function takes a numberx as an argument and returnsTrue ifx is a NaN andFalse otherwise. You can use this function to provide the filtering criteria in afilterfalse() call:

>>>importmath>>>importstatisticsasst>>>fromitertoolsimportfilterfalse>>>sample=[10.1,8.3,10.4,8.8,float("nan"),7.2,float("nan")]>>>st.mean(filterfalse(math.isnan,sample))8.96

Usingmath.isnan() along withfilterfalse() allows you to exclude all the NaN values from the mean computation. Note that after the filtering, the call tomean() returns a value that provides a better description of your sample data.

Replacingfilter() With a List Comprehension

You can use the following pattern to quickly replace a call tofilter() with an equivalent list comprehension:

# Generating a list with filter()list(filter(function,iterable))# Generating a list with a list comprehension[itemforiteminiterableiffunction(item)]

In both cases, the final purpose is to create a list object. The list comprehension approach is more explicit than its equivalentfilter() construct. A quick read through the comprehension reveals the iteration and also the filtering functionality in theif clause.

Using list comprehensions instead offilter() is probably the path most Python developers take nowadays. However, list comprehensions have some drawbacks compared tofilter(). The most notable one is the lack of lazy evaluation. Also, when developers start reading code that usesfilter(), they immediately know that the code is performing filtering operations. However, that’s not so evident in code that uses list comprehensions with the same goal.

A detail to notice when turning afilter() construct into a list comprehension is that if you passNone to the first argument offilter(), then the equivalent list comprehension looks like this:

# Generating a list with filter() and Nonelist(filter(None,iterable))# Equivalent list comprehension[itemforiteminiterableifitem]

In this case, theif clause in the list comprehension testsitem for its truth value. This test follows the standard Python rules about truth values you already saw.

Here’s an example of replacingfilter() with a list comprehension to build a list of even numbers:

>>>numbers=[1,3,10,45,6,50]>>># Filtering function>>>defis_even(x):...returnx%2==0...>>># Use filter()>>>list(filter(is_even,numbers))[10, 6, 50]>>># Use a list comprehension>>>[numberfornumberinnumbersifis_even(number)][10, 6, 50]

In this example, you can see that the list comprehension variant is more explicit. It reads almost like plain English. The list comprehension solution also avoids having to calllist() to build the final list.

Replacingfilter() With a Generator Expression

The natural replacement forfilter() is agenerator expression. That’s becausefilter() returns an iterator that yields items on demand just like a generator expression does. Python iterators are known to be memory efficient. That’s whyfilter() now returns an iterator instead of a list.

Here’s how you can use generator expressions to write the example in the above section:

>>>numbers=[1,3,10,45,6,50]>>># Filtering function>>>defis_even(x):...returnx%2==0...>>># Use filter()>>>even_numbers=filter(is_even,numbers)>>>even_numbers<filter object at 0x7f58691de4c0>>>>list(even_numbers)[10, 6, 50]>>># Use a generator expression>>>even_numbers=(numberfornumberinnumbersifis_even(number))>>>even_numbers<generator object <genexpr> at 0x7f586ade04a0>>>>list(even_numbers)[10, 6, 50]

A generator expression is as efficient as a call tofilter() in terms of memory consumption. Both tools return iterators that yield items on demand. Using either one might be a question of taste, convenience, or style. So, you’re in charge!