Understandingmap()

map() loops over the items of an input iterable (or iterables) and returns an iterator that results from applying a transformation function to every item in the original input iterable.

According to thedocumentation,map() takes a function object and an iterable (or multiple iterables) as arguments and returns an iterator that yields transformed items on demand. The function’s signature is defined as follows:

map(function,iterable[,iterable1,iterable2,...,iterableN])

map() appliesfunction to each item initerable in a loop and returns a new iterator that yields transformed items on demand.function can be any Python function that takes a number of arguments equal to the number of iterables you pass tomap().

This first argument tomap() is atransformation function. In other words, it’s the function that transforms each original item into a new (transformed) item. Even though the Python documentation calls this argumentfunction, it can be any Python callable. This includesbuilt-in functions,classes,methods,lambda functions, anduser-defined functions.

The operation thatmap() performs is commonly known as amapping because it maps every item in an input iterable to a new item in a resulting iterable. To do that,map() applies a transformation function to all the items in the input iterable.

To better understandmap(), suppose you need to take a list of numeric values and transform it into a list containing the square value of every number in the original list. In this case, you can use afor loop and code something like this:

>>>numbers=[1,2,3,4,5]>>>squared=[]>>>fornuminnumbers:...squared.append(num**2)...>>>squared[1, 4, 9, 16, 25]

When you run this loop onnumbers, you get a list of square values. Thefor loop iterates overnumbers and applies a power operation on each value. Finally, it stores the resulting values insquared.

You can achieve the same result without using an explicit loop by usingmap(). Take a look at the following reimplementation of the above example:

>>>defsquare(number):...returnnumber**2...>>>numbers=[1,2,3,4,5]>>>squared=map(square,numbers)>>>list(squared)[1, 4, 9, 16, 25]

square() is a transformation function that maps a number to its square value. The call tomap() appliessquare() to all of the values innumbers and returns an iterator that yields square values. Then you calllist() onmap() to create a list object containing the square values.

Sincemap() is written inC and is highly optimized, its internal implied loop can be more efficient than a regular Pythonfor loop. This is one advantage of usingmap().

A second advantage of usingmap() is related to memory consumption. With afor loop, you need to store the whole list in your system’s memory. Withmap(), you get items on demand, and only one item is in your system’s memory at a given time.

For another example, say you need to convert all the items in a list from astring to an integer number. To do that, you can usemap() along withint() as follows:

>>>str_nums=["4","8","6","5","3","2","8","9","2","5"]>>>int_nums=map(int,str_nums)>>>int_nums<map object at 0x7fb2c7e34c70>>>>list(int_nums)[4, 8, 6, 5, 3, 2, 8, 9, 2, 5]>>>str_nums["4", "8", "6", "5", "3", "2", "8", "9", "2", "5"]

map() appliesint() to every value instr_nums. Sincemap() returns an iterator (a map object), you’ll need calllist() so that you can exhaust the iterator and turn it into a list object. Note that the original sequence doesn’t get modified in the process.

Usingmap() With Different Kinds of Functions

You can use any kind of Python callable withmap(). The only condition would be that the callable takes an argument and returns a concrete and useful value. For example, you can use classes, instances that implement aspecial method called__call__(),instance methods, class methods, static methods, and functions.

There are some built-in functions that you can use withmap(). Consider the following examples:

>>>numbers=[-2,-1,0,1,2]>>>abs_values=list(map(abs,numbers))>>>abs_values[2, 1, 0, 1, 2]>>>list(map(float,numbers))[-2.0, -1.0, 0.0, 1.0, 2.0]>>>words=["Welcome","to","Real","Python"]>>>list(map(len,words))[7, 2, 4, 6]

You can use any built-in function withmap(), provided that the function takes an argument and returns a value.

A common pattern that you’ll see when it comes to usingmap() is to use alambda function as the first argument.lambda functions are handy when you need to pass an expression-based function tomap(). For example, you can reimplement the example of square values using alambda function as follows:

>>>numbers=[1,2,3,4,5]>>>squared=map(lambdanum:num**2,numbers)>>>list(squared)[1, 4, 9, 16, 25]

lambda functions are quite useful when it comes to usingmap(). They can play the role of the first argument tomap(). You can uselambda functions along withmap() to quickly process and transform your iterables.

Processing Multiple Input Iterables Withmap()

If you supply multiple iterables tomap(), then the transformation function must take as many arguments as iterables you pass in. Each iteration ofmap() will pass one value from each iterable as an argument tofunction. The iteration stops at the end of the shortest iterable.

Consider the following example that usespow():

>>>first_it=[1,2,3]>>>second_it=[4,5,6,7]>>>list(map(pow,first_it,second_it))[1, 32, 729]

pow() takes two arguments,x andy, and returnsx to the power ofy. In the first iteration,x will be1,y will be4, and the result will be1. In the second iteration,x will be2,y will be5, and the result will be32, and so on. The final iterable is only as long as the shortest iterable, which isfirst_it in this case.

This technique allows you to merge two or more iterables of numeric values using different kinds of math operations. Here are some examples that uselambda functions to perform different math operations on several input iterables:

>>>list(map(lambdax,y:x-y,[2,4,6],[1,3,5]))[1, 1, 1]>>>list(map(lambdax,y,z:x+y+z,[2,4],[1,3],[7,8]))[10, 15]

In the first example, you use a subtraction operation to merge two iterables of three items each. In the second example, you add together the values of three iterables.

Using the Methods ofstr

A quite common approach tostring manipulation is to use some of themethods of the classstr to transform a given string into a new string. If you’re dealing with iterables of strings and need to apply the same transformation to each string, then you can usemap() along with various string methods:

>>>string_it=["processing","strings","with","map"]>>>list(map(str.capitalize,string_it))['Processing', 'Strings', 'With', 'Map']>>>list(map(str.upper,string_it))['PROCESSING', 'STRINGS', 'WITH', 'MAP']>>>list(map(str.lower,string_it))['processing', 'strings', 'with', 'map']

There are a few transformations that you can perform on every item instring_it usingmap() and string methods. Most of the time, you’d use methods that don’t take additional arguments, likestr.capitalize(),str.lower(),str.swapcase(),str.title(), andstr.upper().

You can also use some methods that take additional arguments with default values, such asstr.strip(), which takes an optional argument calledchar that defaults to removing whitespace:

>>>with_spaces=["processing ","  strings","with   "," map   "]>>>list(map(str.strip,with_spaces))['processing', 'strings', 'with', 'map']

When you usestr.strip() like this, you rely on the default value ofchar. In this case, you usemap() to remove all the whitespace in the items ofwith_spaces.

>>>with_dots=["processing..","...strings","with....","..map.."]>>>list(map(lambdas:s.strip("."),with_dots))['processing', 'strings', 'with', 'map']

This technique can be handy when, for example, you’re processing text files in which lines can have trailing spaces (or other characters) and you need to remove them. If this is the case, then you need to consider that usingstr.strip() without a customchar will remove the newline character as well.

Removing Punctuation

When it comes to processing text, you sometimes need to remove the punctuation marks that remain after you split the text into words. To deal with this problem, you can create a custom function that removes the punctuation marks from a single word using aregular expression that matches the most common punctuation marks.

Here’s a possible implementation of this function usingsub(), which is a regular expression function that lives in there module in Python’s standard library:

>>>importre>>>defremove_punctuation(word):...returnre.sub(r'[!?.:;,"()-]',"",word)>>>remove_punctuation("...Python!")'Python'

Insideremove_punctuation(), you use a regular expression pattern that matches the most common punctuation marks that you’ll find in any text written in English. The call tore.sub() replaces the matched punctuation marks using an empty string ("") and returns a cleanedword.

With your transformation function in place, you can usemap() to run the transformation on every word in your text. Here’s how it works:

>>>text="""Some people, when confronted with a problem, think..."I know, I'll use regular expressions."...Now they have two problems. Jamie Zawinski""">>>words=text.split()>>>words['Some', 'people,', 'when', 'confronted', 'with', 'a', 'problem,', 'think', '"I', 'know,', "I'll", 'use', 'regular', 'expressions."', 'Now', 'they', 'have', 'two', 'problems.', 'Jamie', 'Zawinski']>>>list(map(remove_punctuation,words))['Some', 'people', 'when', 'confronted', 'with', 'a', 'problem', 'think','I', 'know', "I'll", 'use', 'regular', 'expressions', 'Now', 'they', 'have', 'two', 'problems', 'Jamie', 'Zawinski']

In this piece of text, some words include punctuation marks. For example, you have'people,' instead of'people','problem,' instead of'problem', and so on. The call tomap() appliesremove_punctuation() to every word and removes any punctuation mark. So, in the secondlist, you have cleaned words.

Note that the apostrophe (') isn’t in your regular expression because you want to keep contractions likeI'll as they are.

Implementing a Caesar Cipher Algorithm

Julius Caesar, the Roman statesman, used to protect the messages he sent to his generals by encrypting them using a cipher. ACaesar cipher shifts each letter by a number of letters. For example, if you shift the lettera by three, then you get the letterd, and so on.

If the shift goes beyond the end of the alphabet, then you just need to rotate back to the beginning of the alphabet. In the case of a rotation by three,x would becomea. Here’s how the alphabet would look after the rotation:

• Original alphabet:abcdefghijklmnopqrstuvwxyz
• Alphabet rotated by three:defghijklmnopqrstuvwxyzabc

The following code implementsrotate_chr(), a function that takes a character and rotates it by three.rotate_chr() will return the rotated character. Here’s the code:

 1defrotate_chr(c): 2rot_by=3 3c=c.lower() 4alphabet="abcdefghijklmnopqrstuvwxyz" 5# Keep punctuation and whitespace 6ifcnotinalphabet: 7returnc 8rotated_pos=ord(c)+rot_by 9# If the rotation is inside the alphabet10ifrotated_pos<=ord(alphabet[-1]):11returnchr(rotated_pos)12# If the rotation goes beyond the alphabet13returnchr(rotated_pos-len(alphabet))

Insiderotate_chr(), you first check if the character is in the alphabet. If not, then you return the same character. This has the purpose of keeping punctuation marks and other unusual characters. In line 8, you calculate the new rotated position of the character in the alphabet. To do this, you use the built-in functionord().

ord() takes aUnicode character and returns an integer that represents theUnicode code point of the input character. For example,ord("a") returns97, andord("b") returns98:

>>>ord("a")97>>>ord("b")98

ord() takes a character as an argument and returns the Unicode code point of the input character.

If you add this integer to the target number ofrot_by, then you’ll get the rotated position of the new letter in the alphabet. In this example,rot_by is3. So, the letter"a" rotated by three will become the letter at position100, which is the letter"d". The letter"b" rotated by three will become the letter at position101, which is the letter"e", and so on.

If the new position of the letter doesn’t go beyond the position of the last letter (alphabet[-1]), then you return the letter at this new position. To do that, you use the built-in functionchr().

chr() is the inverse oford(). It takes an integer representing the Unicode code point of a Unicode character and returns the character at that position. For example,chr(97) will return'a', andchr(98) will return'b':

>>>chr(97)'a'>>>chr(98)'b'

chr() takes an integer that represents the Unicode code point of a character and returns corresponding character.

Finally, if the new rotated position is beyond the position of the last letter (alphabet[-1]), then you need to rotate back to the beginning of the alphabet. To do that, you need to subtract the length of the alphabet from the rotated position (rotated_pos - len(alphabet)) and then return the letter at that new position usingchr().

Withrotate_chr() as your transformation function, you can usemap() to encrypt any text using the Caesar cipher algorithm. Here’s an example that usesstr.join() to concatenate the string:

>>>"".join(map(rotate_chr,"My secret message goes here."))'pb vhfuhw phvvdjh jrhv khuh.'

Strings are also iterables in Python. So, the call tomap() appliesrotate_chr() to every character in the original input string. In this case,"M" becomes"p","y" becomes"b", and so on. Finally, the call tostr.join() concatenates every rotated character in a final encrypted message.

Using Math Operations

A common example of using math operations to transform an iterable of numeric values is to use thepower operator (**). In the following example, you code a transformation function that takes a number and returns the number squared and cubed:

>>>defpowers(x):...returnx**2,x**3...>>>numbers=[1,2,3,4]>>>list(map(powers,numbers))[(1, 1), (4, 8), (9, 27), (16, 64)]

powers() takes a numberx and returns its square and cube. Since Python handlesmultiple return values as tuples, each call topowers() returns a tuple with two values. When you callmap() withpowers() as an argument, you get a list of tuples containing the square and the cube of every number in the input iterable.

There are a lot of math-related transformations that you can perform withmap(). You can add constants to and subtract them from each value. You can also use some functions from themath module likesqrt(),factorial(),sin(),cos(), and so on. Here’s an example usingfactorial():

>>>importmath>>>numbers=[1,2,3,4,5,6,7]>>>list(map(math.factorial,numbers))[1, 2, 6, 24, 120, 720, 5040]

In this case, you transformnumbers into a new list containing the factorial of each number in the original list.

You can perform a wide spectrum of math transformations on an iterable of numbers usingmap(). How far you get into this topic will depend on your needs and your imagination. Give it some thought and code your own examples!

Converting Temperatures

Another use case formap() is to convert betweenunits of measure. Suppose you have a list of temperatures measured in degrees Celsius or Fahrenheit and you need to convert them into the corresponding temperatures in degrees Fahrenheit or Celsius.

You can code two transformation functions to accomplish this task:

defto_fahrenheit(c):return9/5*c+32defto_celsius(f):return(f-32)*5/9

to_fahrenheit() takes a temperature measurement in Celsius and makes the conversion to Fahrenheit. Similarly,to_celsius() takes a temperature in Fahrenheit and converts it to Celsius.

These functions will be your transformation functions. You can use them withmap() to convert an iterable of temperature measurements to Fahrenheit and to Celsius respectively:

>>>celsius_temps=[100,40,80]>>># Convert to Fahrenheit>>>list(map(to_fahrenheit,celsius_temps))[212.0, 104.0, 176.0]>>>fahr_temps=[212,104,176]>>># Convert to Celsius>>>list(map(to_celsius,fahr_temps))[100.0, 40.0, 80.0]

If you callmap() withto_fahrenheit() andcelsius_temps, then you get a list of temperature measures in Fahrenheit. If you callmap() withto_celsius() andfahr_temps, then you get a list of temperature measures in Celsius.

To extend this example and cover any other kind of unit conversion, you just need to code an appropriate transformation function.

Converting Strings to Numbers

When working with numeric data, you’ll likely deal with situations in which all your data are string values. To do any further calculation, you’ll need to convert the string values into numeric values.map() can help with these situations, too.

If you’re sure that your data is clean and doesn’t contain wrong values, then you can usefloat() orint() directly according to your needs. Here are some examples:

>>># Convert to floating-point>>>list(map(float,["12.3","3.3","-15.2"]))[12.3, 3.3, -15.2]>>># Convert to integer>>>list(map(int,["12","3","-15"]))[12, 3, -15]

In the first example, you usefloat() withmap() to convert all the values from string values tofloating-point values. In the second case, you useint() to convert from a string to aninteger. Note that if one of the values is not a valid number, then you’ll get aValueError.

If you’re not sure that your data is clean, then you can use a more elaborate conversion function like the following:

>>>defto_float(number):...try:...returnfloat(number.replace(",","."))...exceptValueError:...returnfloat("nan")...>>>list(map(to_float,["12.3","3,3","-15.2","One"]))[12.3, 3.3, -15.2, nan]

Insideto_float(), you use atry statement that catches aValueError iffloat() fails when convertingnumber. If no error occurs, then your function returnsnumber converted to a valid floating-point number. Otherwise, you get anan (Not a Number) value, which is a specialfloat value that you can use to represent values that aren’t valid numbers, just like"One" in the above example.

You can customizeto_float() according to your needs. For example, you can replace the statementreturn float("nan") with the statementreturn 0.0, and so on.

map() andfilter()

Sometimes you need to process an input iterable and return another iterable that results from filtering out unwanted values in the input iterable. In that case, Python’sfilter() can be a good option for you.filter() is a built-in function that takes two positional arguments:

1. function will be apredicate or Boolean-valued function, a function that returnsTrue orFalse according to the input data.
2. iterable will be any Python iterable.

filter() yields the items of the inputiterable for whichfunction returnsTrue. If you passNone tofunction, thenfilter() uses the identity function. This means thatfilter() will check the truth value of each item initerable and filter out all of the items that arefalsy.

To illustrate how you can usemap() along withfilter(), say you need to calculate thesquare root of all the values in a list. Since your list can contain negative values, you’ll get an error because the square root isn’t defined for negative numbers:

>>>importmath>>>math.sqrt(-16)Traceback (most recent call last):  File"<input>", line1, in<module>math.sqrt(-16)ValueError:math domain error

With a negative number as an argument,math.sqrt() raises aValueError. To avoid this issue, you can usefilter() to filter out all the negative values and then find the square root of the remaining positive values. Check out the following example:

>>>importmath>>>defis_positive(num):...returnnum>=0...>>>defsanitized_sqrt(numbers):...cleaned_iter=map(math.sqrt,filter(is_positive,numbers))...returnlist(cleaned_iter)...>>>sanitized_sqrt([25,9,81,-16,0])[5.0, 3.0, 9.0, 0.0]

is_positive() is a predicate function that takes a number as an argument and returnsTrue if the number is greater than or equal to zero. You can passis_positive() tofilter() to remove all the negative numbers fromnumbers. So, the call tomap() will process only positive numbers andmath.sqrt() won’t give you aValueError.

map() andreduce()

Python’sreduce() is a function that lives in a module calledfunctools in the Python standard library.reduce() is another core functional tool in Python that is useful when you need to apply a function to an iterable and reduce it to a single cumulative value. This kind of operation is commonly known asreduction or folding.reduce() takes two required arguments:

1. function can be any Python callable that accepts two arguments and returns a value.
2. iterable can be any Python iterable.

reduce() will applyfunction to all the items initerable and cumulatively compute a final value.

Here’s an example that combinesmap() andreduce() to calculate the total size of all the files that live in your home directory cumulatively:

>>>importfunctools>>>importoperator>>>importos>>>files=os.listdir(os.path.expanduser("~"))>>>functools.reduce(operator.add,map(os.path.getsize,files))4377381

In this example, you callos.path.expanduser("~") to get the path to your home directory. Then you callos.listdir() on that path to get a list with the paths of all the files that live there.

The call tomap() usesos.path.getsize() to get the size of every file. Finally, you usereduce() withoperator.add() to get the cumulative sum of the size of every single file. The final result is the total size of all the files in your home directory in bytes.

Even though you can usereduce() to solve the problem covered in this section, Python offers other tools that can lead to a more Pythonic and efficient solution. For example, you can use the built-in functionsum() to compute the total size of the files in your home directory:

>>>importos>>>files=os.listdir(os.path.expanduser("~"))>>>sum(map(os.path.getsize,files))4377381

This example is a lot more readable and efficient than the example that you saw before. If you want to dive deeper into how to usereduce() and which alternative tools you can use to replacereduce() in a Pythonic way, then check outPython’s reduce(): From Functional to Pythonic Style.

Using List Comprehensions

There’s a general pattern that you can use to replace a call tomap() with a list comprehension. Here’s how:

# Generating a list with maplist(map(function,iterable))# Generating a list with a list comprehension[function(x)forxiniterable]

Note that the list comprehension almost always reads more clearly than the call tomap(). Since list comprehensions are quite popular among Python developers, it’s common to find them everywhere. So, replacing a call tomap() with a list comprehension will make your code look more familiar to other Python developers.

Here’s an example of how to replacemap() with a list comprehension to build a list of square numbers:

>>># Transformation function>>>defsquare(number):...returnnumber**2>>>numbers=[1,2,3,4,5,6]>>># Using map()>>>list(map(square,numbers))[1, 4, 9, 16, 25, 36]>>># Using a list comprehension>>>[square(x)forxinnumbers][1, 4, 9, 16, 25, 36]

If you compare both solutions, then you might say that the one that uses the list comprehension is more readable because it reads almost like plain English. Additionally, list comprehensions avoid the need to explicitly calllist() onmap() to build the final list.

Using Generator Expressions

map() returns amap object, which is an iterator that yields items on demand. So, the natural replacement formap() is agenerator expression because generator expressions return generator objects, which are also iterators that yield items on demand.

Python iterators are known to be quite efficient in terms of memory consumption. This is the reason whymap() now returns an iterator instead of alist.

There’s a tiny syntactical difference between a list comprehension and a generator expression. The first uses a pair of square brackets ([]) to delimit the expression. The second uses a pair of parentheses (()). So, to turn a list comprehension into a generator expression, you just need to replace the square brackets with parentheses.

You can use generator expressions to write code that reads clearer than code that usesmap(). Check out the following example:

>>># Transformation function>>>defsquare(number):...returnnumber**2>>>numbers=[1,2,3,4,5,6]>>># Using map()>>>map_obj=map(square,numbers)>>>map_obj<map object at 0x7f254d180a60>>>>list(map_obj)[1, 4, 9, 16, 25, 36]>>># Using a generator expression>>>gen_exp=(square(x)forxinnumbers)>>>gen_exp<generator object <genexpr> at 0x7f254e056890>>>>list(gen_exp)[1, 4, 9, 16, 25, 36]

This code has a main difference from the code in the previous section: you change the square brackets to a pair of parentheses to turn the list comprehension into a generator expression.

Generator expressions are commonly used as arguments in function calls. In this case, you don’t need to use parentheses to create the generator expression because the parentheses that you use to call the function also provide the syntax to build the generator. With this idea, you can get the same result as the above example by callinglist() like this:

>>>list(square(x)forxinnumbers)[1, 4, 9, 16, 25, 36]

If you use a generator expression as an argument in a function call, then you don’t need an extra pair of parentheses. The parentheses that you use to call the function provide the syntax to build the generator.

Generator expressions are as efficient asmap() in terms of memory consumption because both of them return iterators that yield items on demand. However, generator expressions will almost always improve your code’s readability. They also make your code more Pythonic in the eyes of other Python developers.