Example 1: Reading Large Files

A common use case of generators is towork with data streams or large files, likeCSV files. These text files separate data into columns by using commas. This format is a common way to share data. Now, what if you want to count the number of rows in a CSV file? The code block below shows one way of counting those rows:

csv_gen=csv_reader("some_csv.txt")row_count=0forrowincsv_gen:row_count+=1print(f"Row count is{row_count}")

Looking at this example, you might expectcsv_gen to be a list. To populate this list,csv_reader() opens a file and loads its contents intocsv_gen. Then, the program iterates over the list and incrementsrow_count for each row.

This is a reasonable explanation, but would this design still work if the file is very large? What if the file is larger than the memory you have available? To answer this question, let’s assume thatcsv_reader() just opens the file and reads it into an array:

defcsv_reader(file_name):file=open(file_name)result=file.read().split("\n")returnresult

This function opens a given file and usesfile.read() along with.split() to add each line as a separate element to a list. If you were to use this version ofcsv_reader() in the row counting code block you saw further up, then you’d get the following output:

Traceback (most recent call last):  File"ex1_naive.py", line22, in<module>main()  File"ex1_naive.py", line13, inmaincsv_gen=csv_reader("file.txt")  File"ex1_naive.py", line6, incsv_readerresult=file.read().split("\n")MemoryError

In this case,open() returns a generator object that you can lazily iterate through line by line. However,file.read().split() loads everything into memory at once, causing theMemoryError.

Before that happens, you’ll probably notice your computer slow to a crawl. You might even need to kill the program with aKeyboardInterrupt. So, how can you handle these huge data files? Take a look at a new definition ofcsv_reader():

defcsv_reader(file_name):forrowinopen(file_name,"r"):yieldrow

In this version, you open the file, iterate through it, and yield a row. This code should produce the following output, with no memory errors:

Row count is 64186394

What’s happening here? Well, you’ve essentially turnedcsv_reader() into a generator function. This version opens a file, loops through each line, and yields each row, instead of returning it.

You can also define agenerator expression (also called agenerator comprehension), which has a very similar syntax tolist comprehensions. In this way, you can use the generator without calling a function:

csv_gen=(rowforrowinopen(file_name))

This is a more succinct way to create the listcsv_gen. You’ll learn more about the Python yield statement soon. For now, just remember this key difference:

• Usingyield will result in a generator object.
• Usingreturn will result in the first line of the fileonly.

Example 2: Generating an Infinite Sequence

Let’s switch gears and look at infinite sequence generation. In Python, to get a finite sequence, you callrange() and evaluate it in a list context:

>>>a=range(5)>>>list(a)[0, 1, 2, 3, 4]

Generating aninfinite sequence, however, will require the use of a generator, since your computer memory is finite:

definfinite_sequence():num=0whileTrue:yieldnumnum+=1

This code block is short and sweet. First, you initialize thevariablenum and start an infinite loop. Then, you immediatelyyield num so that you can capture the initial state. This mimics the action ofrange().

Afteryield, you incrementnum by 1. If you try this with afor loop, then you’ll see that it really does seem infinite:

>>>foriininfinite_sequence():...print(i,end=" ")...0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 2930 31 32 33 34 35 36 37 38 39 40 41 42[...]6157818 6157819 6157820 6157821 6157822 6157823 6157824 6157825 6157826 61578276157828 6157829 6157830 6157831 6157832 6157833 6157834 6157835 6157836 61578376157838 6157839 6157840 6157841 6157842Traceback (most recent call last):  File"<stdin>", line2, in<module>KeyboardInterrupt

The program will continue to execute until you stop it manually.

Instead of using afor loop, you can also callnext() on the generator object directly. This is especially useful for testing a generator in the console:

>>>gen=infinite_sequence()>>>next(gen)0>>>next(gen)1>>>next(gen)2>>>next(gen)3

Here, you have a generator calledgen, which you manually iterate over by repeatedly callingnext(). This works as a great sanity check to make sure your generators are producing the output you expect.

Example 3: Detecting Palindromes

You can use infinite sequences in many ways, but one practical use for them is in building palindrome detectors. Apalindrome detector will locate all sequences of letters ornumbers that are palindromes. These are words or numbers that are read the same forward and backward, like 121. First, define your numeric palindrome detector:

defis_palindrome(num):# Skip single-digit inputsifnum//10==0:returnFalsetemp=numreversed_num=0whiletemp!=0:reversed_num=(reversed_num*10)+(temp%10)temp=temp//10ifnum==reversed_num:returnnumelse:returnFalse

Don’t worry too much about understanding the underlying math in this code. Just note that the function takes an input number, reverses it, and checks to see if the reversed number is the same as the original. Now you can use your infinite sequence generator to get a running list of all numeric palindromes:

>>>foriininfinite_sequence():...pal=is_palindrome(i)...ifpal:...print(i)...112233[...]997999989999999100001101101102201Traceback (most recent call last):  File"<stdin>", line2, in<module>  File"<stdin>", line5, inis_palindromeKeyboardInterrupt

In this case, the only numbers that areprinted to the console are those that are the same forward or backward.

Now that you’ve seen a simple use case for an infinite sequence generator, let’s dive deeper into how generators work.

Building Generators With Generator Expressions

Like list comprehensions, generator expressions allow you to quickly create a generator object in just a few lines of code. They’re also useful in the same cases where list comprehensions are used, with an added benefit: you can create them without building and holding the entire object in memory before iteration. In other words, you’ll have no memory penalty when you use generator expressions. Take this example of squaring some numbers:

>>>nums_squared_lc=[num**2fornuminrange(5)]>>>nums_squared_gc=(num**2fornuminrange(5))

Bothnums_squared_lc andnums_squared_gc look basically the same, but there’s one key difference. Can you spot it? Take a look at what happens when you inspect each of these objects:

>>>nums_squared_lc[0, 1, 4, 9, 16]>>>nums_squared_gc<generator object <genexpr> at 0x107fbbc78>

The first object used brackets to build a list, while the second created a generator expression by using parentheses. The output confirms that you’ve created a generator object and that it is distinct from a list.

Profiling Generator Performance

You learned earlier that generators are a great way to optimize memory. While an infinite sequence generator is an extreme example of this optimization, let’s amp up the number squaring examples you just saw and inspect the size of the resulting objects. You can do this with a call tosys.getsizeof():

>>>importsys>>>nums_squared_lc=[i**2foriinrange(10000)]>>>sys.getsizeof(nums_squared_lc)87624>>>nums_squared_gc=(i**2foriinrange(10000))>>>print(sys.getsizeof(nums_squared_gc))120

In this case, the list you get from the list comprehension is 87,624 bytes, while the generator object is only 120. This means that the list is over 700 times larger than the generator object!

There is one thing to keep in mind, though. If the list is smaller than the running machine’s available memory, then list comprehensions can befaster to evaluate than the equivalent generator expression. To explore this, let’s sum across the results from the two comprehensions above. You can generate a readout withcProfile.run():

>>>importcProfile>>>cProfile.run('sum([i * 2 for i in range(10000)])')         5 function calls in 0.001 seconds   Ordered by: standard name   ncalls  tottime  percall  cumtime  percall filename:lineno(function)        1    0.001    0.001    0.001    0.001 <string>:1(<listcomp>)        1    0.000    0.000    0.001    0.001 <string>:1(<module>)        1    0.000    0.000    0.001    0.001 {built-in method builtins.exec}        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}>>>cProfile.run('sum((i * 2 for i in range(10000)))')         10005 function calls in 0.003 seconds   Ordered by: standard name   ncalls  tottime  percall  cumtime  percall filename:lineno(function)    10001    0.002    0.000    0.002    0.000 <string>:1(<genexpr>)        1    0.000    0.000    0.003    0.003 <string>:1(<module>)        1    0.000    0.000    0.003    0.003 {built-in method builtins.exec}        1    0.001    0.001    0.003    0.003 {built-in method builtins.sum}        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Here, you can see that summing across all values in the list comprehension took about a third of the time as summing across the generator. If speed is an issue and memory isn’t, then a list comprehension is likely a better tool for the job.

Remember, list comprehensions return full lists, while generator expressions return generators. Generators work the same whether they’re built from a function or an expression. Using an expression just allows you to define simple generators in a single line, with an assumedyield at the end of each inner iteration.

The Python yield statement is certainly the linchpin on which all of the functionality of generators rests, so let’s dive into howyield works in Python.

How to Use.send()

For this next section, you’re going to build a program that makes use of all three methods. This program will print numeric palindromes like before, but with a few tweaks. Upon encountering a palindrome, your new program will add a digit and start a search for the next one from there. You’ll also handle exceptions with.throw() and stop the generator after a given amount of digits with.close(). First, let’s recall the code for your palindrome detector:

defis_palindrome(num):# Skip single-digit inputsifnum//10==0:returnFalsetemp=numreversed_num=0whiletemp!=0:reversed_num=(reversed_num*10)+(temp%10)temp=temp//10ifnum==reversed_num:returnTrueelse:returnFalse

This is the same code you saw earlier, except that now the program returns strictlyTrue orFalse. You’ll also need to modify your original infinite sequence generator, like so:

 1definfinite_palindromes(): 2num=0 3whileTrue: 4ifis_palindrome(num): 5i=(yieldnum) 6ifiisnotNone: 7num=i 8num+=1

There are a lot of changes here! The first one you’ll see is in line 5, wherei = (yield num). Though you learned earlier thatyield is a statement, that isn’t quite the whole story.

As of Python 2.5 (the same release that introduced the methods you are learning about now),yield is anexpression, rather than a statement. Of course, you can still use it as a statement. But now, you can also use it as you see in the code block above, wherei takes the value that is yielded. This allows you to manipulate the yielded value. More importantly, it allows you to.send() a value back to the generator. When execution picks up afteryield,i will take the value that is sent.

You’ll also checkif i is not None, which could happen ifnext() is called on the generator object. (This can also happen when you iterate with afor loop.) Ifi has a value, then you updatenum with the new value. But regardless of whether or noti holds a value, you’ll then incrementnum and start the loop again.

Now, take a look at the main function code, which sends the lowest number with another digit back to the generator. For example, if the palindrome is 121, then it will.send() 1000:

pal_gen=infinite_palindromes()foriinpal_gen:digits=len(str(i))pal_gen.send(10**(digits))

With this code, you create the generator object and iterate through it. The program only yields a value once a palindrome is found. It useslen() to determine the number of digits in that palindrome. Then, it sends10 ** digits to the generator. This brings execution back into the generator logic and assigns10 ** digits toi. Sincei now has a value, the program updatesnum, increments, and checks for palindromes again.

Once your code finds and yields another palindrome, you’ll iterate via thefor loop. This is the same as iterating withnext(). The generator also picks up at line 5 withi = (yield num). However, nowi isNone, because you didn’t explicitly send a value.

What you’ve created here is acoroutine, or a generator function into which you can pass data. These are useful for constructing data pipelines, but as you’ll see soon, they aren’t necessary for building them. (If you’re looking to dive deeper, thenthis course on coroutines and concurrency is one of the most comprehensive treatments available.)

Now that you’ve learned about.send(), let’s take a look at.throw().

How to Use.throw()

.throw() allows you to throw exceptions with the generator. In the below example, you raise the exception in line 6. This code will throw aValueError oncedigits reaches 5:

 1pal_gen=infinite_palindromes() 2foriinpal_gen: 3print(i) 4digits=len(str(i)) 5ifdigits==5: 6pal_gen.throw(ValueError("We don't like large palindromes")) 7pal_gen.send(10**(digits))

This is the same as the previous code, but now you’ll check ifdigits is equal to 5. If so, then you’ll.throw() aValueError. To confirm that this works as expected, take a look at the code’s output:

11111111110101Traceback (most recent call last):  File"advanced_gen.py", line47, in<module>main()  File"advanced_gen.py", line41, inmainpal_gen.throw(ValueError("We don't like large palindromes"))  File"advanced_gen.py", line26, ininfinite_palindromesi=(yieldnum)ValueError:We don't like large palindromes

.throw() is useful in any areas where you might need to catch anexception. In this example, you used.throw() to control when you stopped iterating through the generator. You can do this more elegantly with.close().

How to Use.close()

As its name implies,.close() allows you to stop a generator. This can be especially handy when controlling an infinite sequence generator. Let’s update the code above by changing.throw() to.close() to stop the iteration:

 1pal_gen=infinite_palindromes() 2foriinpal_gen: 3print(i) 4digits=len(str(i)) 5ifdigits==5: 6pal_gen.close() 7pal_gen.send(10**(digits))

Instead of calling.throw(), you use.close() in line 6. The advantage of using.close() is that it raisesStopIteration, an exception used to signal the end of a finite iterator:

11111111110101Traceback (most recent call last):  File"advanced_gen.py", line46, in<module>main()  File"advanced_gen.py", line42, inmainpal_gen.send(10**(digits))StopIteration

Now that you’ve learned more about the special methods that come with generators, let’s talk about using generators to build data pipelines.