A lot of the articles in this series take advantage of a feature of Python which allows us to performance test our code, and I finally wanted to get around to explaining how it works and how to use it.

In this article, I cover three main techniques: brute force,timeit, andcProfile. Personally, I performance test my code withtimeit because I find it easy to understand, but you may find the various profiling tools helpful. At the end, I’ll ask you to demonstrate your newfound skills with a challenge.

Table of Contents

• Problem Introduction
• Solutions
  • Performance Testing by Brute Force
  • Performance Testing Using the timeit Library
  • Performance Testing Using the cProfile Library
  • Performance Testing With External Libraries
• Challenge
• A Little Recap

Problem Introduction

In order to start talking about how to performance test Python code, we need to define it. At a high level, a performance test is anything that verifies the speed, reliability, scalability, and/or stability of software. For our purposes, we’ll be looking at speed. In particular, we’ll be looking at different ways to compare the speed of two programs using their relative execution times.

When it comes to performance testing software, the process isn’t always easy or obvious. In particular, there are a lot of pitfalls. For example, performance tests can be influenced by all sorts of factors like background processes (i.e. Spotify, Eclipse, GitHub Desktop, etc.).

In addition, performance tests aren’t always written in a way that fairly accounts for differences in implementation. For example, I might have two snippets of code that have the same behavior except one requires a library to be imported. When I run my test, I don’t want the import to affect the test outcome. As a result, I should write my tests such that I don’t start timing until after the library is imported.

On top of all that, it’s important to take into account different types of scenarios when performance testing. For instance, if we have two similar snippets, one might have better performance for larger data sets. It’s important to test a range of data sets for that reason.

At any rate, the goal of this article is to look at a few different ways we can performance test code in Python. Let’s dig in!

Solutions

As always, I like to share a few ways to accomplish our task. Of course, if you’ve been following along in this series, you know that I prefer to use thetimeit library to test snippets. Luckily, there are more options iftimeit isn’t for you.

Performance Testing by Brute Force

If you’ve never done any performance testing before, you probably have a gist of how to get started. Typically, we want to take a timestamp before and after we run our code snippet. Then, we can calculate the difference between those times and use the result in our comparison with other snippets.

To do this in Python, we can take advantage of thedatetime library:

import datetimestart_time = datetime.datetime.now()# insert code snippet hereend_time = datetime.datetime.now()print(end_time - start_time)

Of course, this solution leaves a lot to be desired. For example, it only gives us a single data point. Ideally, we’d want to run this a few times to collect an average or at least a lower bound, but this can do in a pinch.

Performance Testing Using the timeit Library

If you’d prefer to have all this timestamp garbage abstracted away with the addition of a few perks, check out thetimeit library. With thetimeit library, there are basically two main ways to test code: command line or inline. For our purposes, we’ll take a look at the inline version since that’s what I use for all my testing.

To test code using thetimeit library, you’ll need to call either thetimeit function or therepeat function. Either one is fine, but therepeat function gives a bit more control.

As an example, we’ll test the following code snippet froman earlier article on list comprehensions:

[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]

In this snippet, we’re generating a list of pairs from two tuples. To test it, we could use thetimeit function:

import timeittimeit.timeit("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]")

If done correctly, this will run the snippet a million times and return an average execution time as a result. Of course, you’re welcome to change the number of iterations using thenumber keyword argument:

import timeittimeit.timeit("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]", number=1000)

Naturally, we can take this test a step further by running it multiple times using the repeat function:

import timeittimeit.repeat("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]")

Instead of returning an execution time, this function returns a list of execution times. In this case, the list will contain three separate execution times. Of course, we don’t need all those times. Instead, we can return the smallest execution time, so we can get an idea of the lower bound of the snippet:

import timeitmin(timeit.repeat("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]"))

If you’ve been around a bit, you’ve probably seen this exact syntax in my performance tests in other articles or videos. Of course, I go the extra mile and increase the number of repetitions, but it’s probably overkill. In any case, this is a great way of performance testing Python snippets.

Performance Testing Using the cProfile Library

Outside oftimeit and outright brute force, you can always leverage other profiling tools likecProfile. Liketimeit, we can leveragecProfile to get runtime statistics from a section of code. Of course, cProfile is quite a bit more detailed. For example, we can run the same list comprehension from above as follows:

import cProfilecProfile.run("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]")

As a result, you get a nice report that looks like this:

4 function calls in 0.000 seconds   Ordered by: standard name   ncalls  tottime  percall  cumtime  percall filename:lineno(function)        1    0.000    0.000    0.000    0.000 <string>:1(<listcomp>)        1    0.000    0.000    0.000    0.000 <string>:1(<module>)        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Here, we get a nice table which includes a lot of helpful information. Specifically, each row indicates a function that was executed, and each column breaks down a different runtime segment. For example, the<listcomp> function was called once (ncalls) and took 0.000 seconds (tottime) excluding calls to subfunctions. To understand everything else in this table, check out the following breakdown of all six columns:

• ncalls: the number of times that particular function was called
  • This number may actually be written as a fraction (e.g.3/1) where the first value is the number of total calls and the second value is the number of primitive calls (not recursive).
• tottime: the total amount of time the function spent executingnot including calls to subfunctions
• percall (first): the ratio of tottime to ncalls (i.e. the average amount of time spent in this function excluding subfunctions)
• cumtime: the total amount of time the function spent executing including calls to subfunctions
• percall (second): the ratio of cumtime to primitive calls (i.e. the average amount of time spent in this function)
• filename:lineno(function): the filename, line number, and function in question

As you can see,cProfile helps you peek at the internal workings of a code snippet. Of course, you don’t get fine grained timings, so this works better as a compliment totimeit rather than a replacement. That said, I thinkcProfile would be excellent for profiling large scripts. That way, you can determine which functions need optimization.

Performance Testing With External Libraries

While Python provides plenty of ways to benchmark your own code, there are also other libraries we can leverage as well. For instance:

• line_profiler
• memory_profiler
• profilehooks

Personally, I’ve never used any of these tools, but I felt I should share them for the sake of completeness. Feel free to follow those links to learn more.

Challenge

At this point, I’d usually share some performance metrics for each of the solutions above, but that doesn’t really make sense in this context. Instead, it’s time to jump straight to the challenge!

Pick one of the articles in this series and run your own performance metrics on each of the solutions. Since I typically runtimeit, maybe you could try using one of the other tools from this article. For example, try running cProfile on allthe string formatting solutions.

When you’re done, share the best results in the comments. I’m interested to see what you learn! While you’re at it, check my work. I’d love to know if there are other solutions I’m missing.

A Little Recap

As always, I like to finish things out with a list of options. Keep in mind that each solution leverages an example code snippet. In this case, I chose a list comprehension, but you can use any snippet:

# Brute force solutionimport datetimestart_time = datetime.datetime.now()[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)] # example snippetend_time = datetime.datetime.now()print(end_time - start_time)# timeit solutionimport timeitmin(timeit.repeat("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]"))# cProfile solutionimport cProfilecProfile.run("[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]")

Well, that’s all I’ve got! If you have any performance tools of your own to add to the list, feel free to share them in the comments.

In the meantime, I have plenty of How to Python articles you might be interested in:

• How to Sort a List of Strings in Python
• How to Get the Last Item of a List in Python
• How to Check If a File Exists in Python

If you prefer visual media,I have a YouTube channel which is currently focused on explaining content from this series. Head on over there and throw me a subscribe to help me build up my channel.

Also, feel to leverage some of the following related resources provided by Amazon:

• Serious Python: Black-Belt Advice on Deployment, Scalability, Testing, and More
• Testing Python: Applying Unit Testing, TDD, BDD and Acceptance Testing 1st Edition

Finally, you can always get the latest The Renegade Coder content sent to your inbox throughthe email list. If you want to go the extra mile,toss me a couple bucks on Patreon. You won’t regret it!

At any rate, until next time!

Recent Code Posts