Python’sin Operator

To better understand thein operator, you’ll start by writing some small demonstrative examples that determine if a given valueis in a list:

>>>5in[2,3,5,9,7]True>>>8in[2,3,5,9,7]False

The first expression returnsTrue because5 appears inside your list of numbers. The second expression returnsFalse because8 isn’t present in the list.

According to thein operatordocumentation, an expression likevalue in collection is equivalent to the following code:

any(valueisitemorvalue==itemforitemincollection)

Thegenerator expression wrapped in the call toany() builds a list of the Boolean values that result from checking if the targetvalue has the same identity or is equal to the currentitem incollection. The call toany() checks if any one of the resulting Boolean values isTrue, in which case the function returnsTrue. If all the values areFalse, thenany() returnsFalse.

Python’snot in Operator

Thenot in membership operator does exactly the opposite. With this operator, you can check if a given valueis not in a collection of values:

>>>5notin[2,3,5,9,7]False>>>8notin[2,3,5,9,7]True

In the first example, you getFalse because5 is in[2, 3, 5, 9, 7]. In the second example, you getTrue because8 isn’t in the list of values. This negative logic may seem like a tongue twister. To avoid confusion, remember that you’re trying to determine if the value isnot part of a given collection of values.

With this quick overview of how membership operators work, you’re ready to go to the next level and learn howin andnot in work with different built-in data types.

Lists, Tuples, and Ranges

So far, you’ve coded a few examples of using thein andnot in operators to determine if a given value is present in an existing list of values. For these examples, you’ve explicitly usedlist objects. So, you’re already familiar with how membership tests work with lists.

With tuples, the membership operators work the same as they would with lists:

>>>5in(2,3,5,9,7)True>>>5notin(2,3,5,9,7)False

There are no surprises here. Both examples work the same as the list-focused examples. In the first example, thein operator returnsTrue because the target value,5, is in the tuple. In the second example,not in returns the opposite result.

For lists and tuples, the membership operators use asearch algorithm that iterates over the items in the underlying collection. Therefore, as your iterable gets longer, the search time increases in direct proportion. UsingBig O notation, you’d say that membership operations on these data types have atime complexity ofO(n).

If you use thein andnot in operators withrange objects, then you get a similar result:

>>>5inrange(10)True>>>5notinrange(10)False>>>5inrange(0,10,2)False>>>5notinrange(0,10,2)True

When it comes torange objects, using membership tests may seem unnecessary at first glance. Most of the time, you’ll know the values in the resulting range beforehand. But what if you’re usingrange() with arguments that are determined at runtime?

Consider the following examples, which userandom numbers to determine arguments at runtime:

>>>fromrandomimportrandint>>>50inrange(0,100,randint(1,10))False>>>50inrange(0,100,randint(1,10))False>>>50inrange(0,100,randint(1,10))True>>>50inrange(0,100,randint(1,10))True

On your machine, you might get different results because you’re working with random ranges. In these specific examples,step is the only value that varies. In real code, you could have varying values forstart andstop as well.

Forrange objects, the algorithm behind the membership tests computes the presence of a given value using the expression(value - start) % step) == 0, which depends on the arguments used to create the range at hand. This makes membership testsvery efficient when they operate onrange objects. In this case, you’d say that their time complexity isO(1).

>>>(2,3,5,9,7).index(8)Traceback (most recent call last):...ValueError:tuple.index(x): x not in tuple

Remember that the target value in a membership test can be of any type. The test will check if that value is or isn’t in the target collection. For example, say that you have a hypothetical app where the users authenticate with a username and a password. You can have something like this:

username=input("Username: ")password=input("Password: ")users=[("john","secret"),("jane","secret"),("linda","secret")]if(username,password)inusers:print(f"Hi{username}, you're logged in!")else:print("Wrong username or password")

This is a naive example. It’s unlikely that anyone would handle their users and passwords like this. But the example shows that the target value can be of any data type. In this case, you use a tuple of strings representing the username and the password of a given user.

Here’s how the code works in practice:

$pythonusers.pyUsername: johnPassword: secretHi john, you're logged in!$pythonusers.pyUsername: tinaPassword: secretWrong username or password

In the first example, the username and password are correct because they’re in theusers list. In the second example, the username doesn’t belong to any registered user, so the authentication fails.

In these examples, it’s important to note that the order in which the data is stored in the login tuple is critical because something like("john", "secret") isn’t equal to("secret", "john") in tuple comparison even if they have the same items.

In this section, you’ve explored examples that showcase the core behavior of membership operators with common Python built-in sequences. However, there’s a built-in sequence left. Yes, strings! In the next section, you’ll learn how membership operators work with this data type in Python.

Strings

Python strings are a fundamental tool in every Python developer’s tool kit. Like tuples, lists, and ranges, strings are also sequences because their items or characters are sequentially stored in memory.

You can use thein andnot in operators with strings when you need to figure out if a given character is present in the target string. For example, say that you’re using strings to set and manage user permissions for a given resource:

>>>classUser:...def__init__(self,username,permissions):...self.username=username...self.permissions=permissions...>>>admin=User("admin","wrx")>>>john=User("john","rx")>>>defhas_permission(user,permission):...returnpermissioninuser.permissions...>>>has_permission(admin,"w")True>>>has_permission(john,"w")False

TheUser class takes two arguments, a username and a set of permissions. To provide the permissions, you use a string in whichw means that the user haswrite permission,r means that the user hasread permission, andx impliesexecution permissions. Note that these letters are the same ones that you’d find in the Unix-stylefile-system permissions.

The membership test insidehas_permission() checks whether the currentuser has a givenpermission or not, returningTrue orFalse accordingly. To do this, thein operator searches the permissions string to find a single character. In this example, you want to know if the users havewrite permission.

However, your permission system has a hidden issue. What would happen if you called the function with an empty string? Here’s your answer:

>>>has_permission(john,"")True

Because an empty string is always considered a substring of any other string, an expression like"" in user.permissions will returnTrue. Depending on who has access to your users’ permissions, this behavior of membership tests may imply a security breach in your system.

You can also use the membership operators to determine if astring contains a substring:

>>>greeting="Hi, welcome to Real Python!">>>"Hi"ingreetingTrue>>>"Hi"notingreetingFalse>>>"Hello"ingreetingFalse>>>"Hello"notingreetingTrue

For the string data type, an expression likesubstring in string isTrue ifsubstring is part ofstring. Otherwise, the expression isFalse.

>>>greeting.find("Python")20>>>greeting.find("Hello")-1

An important point to remember when using membership tests on strings is that string comparisons are case-sensitive:

>>>"PYTHON"ingreetingFalse

This membership test returnsFalse because strings comparisons are case-sensitive, and"PYTHON" in uppercase isn’t present ingreeting. To work around this case sensitivity, you can normalize all your strings using either the.upper() or.lower() method:

>>>"PYTHON".lower()ingreeting.lower()True

In this example, you use.lower() to convert the target substring and the original string into lowercase letters. This conversion tricks the case sensitivity in the implicit string comparison.

Generators

Generator functions andgenerator expressions create memory-efficientiterators known asgenerator iterators. To be memory efficient, these iterators yield items on demand without keeping a complete series of values in memory.

In practice, a generator function is afunction that uses theyield statement in its body. For example, say that you need a generator function that takes a list of numbers and returns an iterator that yields square values from the original data. In this case, you can do something like this:

>>>defsquares_of(values):...forvalueinvalues:...yieldvalue**2...>>>squares=squares_of([1,2,3,4])>>>next(squares)1>>>next(squares)4>>>next(squares)9>>>next(squares)16>>>next(squares)Traceback (most recent call last):...StopIteration

This function returns a generator iterator that yields square numbers on demand. You can use the built-innext() function to retrieve consecutive values from the iterator. When the generator iterator is completely consumed, it raises aStopIteration exception to communicate that no more values are left.

You can use the membership operators on a generator function likesquares_of():

>>>4insquares_of([1,2,3,4])True>>>9insquares_of([1,2,3,4])True>>>5insquares_of([1,2,3,4])False

Thein operator works as expected when you use it with generator iterators, returningTrue if the value is present in the iterator andFalse otherwise.

However, there’s something you need to be aware of when checking for membership on generators. A generator iterator will yield each item only once. If you consume all the items, then the iterator will be exhausted, and you won’t be able to iterate over it again. If you consume only some items from a generator iterator, then you can iterate over the remaining items only.

When you usein ornot in on a generator iterator, the operator will consume it while searching for the target value. If the value is present, then the operator will consume all the values up to the target value. The rest of the values will still be available in the generator iterator:

>>>squares=squares_of([1,2,3,4])>>>4insquaresTrue>>>next(squares)9>>>next(squares)16>>>next(squares)Traceback (most recent call last):...StopIteration

In this example,4 is in the generator iterator because it’s the square of2. Therefore,in returnsTrue. When you usenext() to retrieve a value fromsquare, you get9, which is the square of3. This result confirms that you no longer have access to the first two values. You can continue callingnext() until you get aStopIteration exception when the generator iterator is exhausted.

Likewise, if the value isn’t present in the generator iterator, then the operator will consume the iterator completely, and you won’t have access to any of its values:

>>>squares=squares_of([1,2,3,4])>>>5insquaresFalse>>>next(squares)Traceback (most recent call last):...StopIteration

In this example, thein operator consumessquares completely, returningFalse because the target value isn’t in the input data. Because the generator iterator is now exhausted, a call tonext() withsquares as an argument raisesStopIteration.

You can also create generator iterators using generator expressions. These expressions use the same syntax aslist comprehensions but replace the square brackets ([]) with round brackets (()). You can use thein andnot in operators with the result of a generator expression:

>>>squares=(value**2forvaluein[1,2,3,4])>>>squares<generator object <genexpr> at 0x1056f20a0>>>>4insquaresTrue>>>next(squares)9>>>next(squares)16>>>next(squares)Traceback (most recent call last):...StopIteration

Thesquaresvariable now holds the iterator that results from the generator expression. This iterator yields square values from the input list of numbers. Generator iterators from generator expressions work the same as generator iterators from generator functions. So, the same rules apply when you use them in membership tests.

Another critical issue can arise when you use thein andnot in operators with generator iterators. This issue can appear when you’re working with infinite iterators. The function below returns an iterator that yields infinite integers:

>>>definfinite_integers():...number=0...whileTrue:...yieldnumber...number+=1...>>>integers=infinite_integers()>>>integers<generator object infinite_integers at 0x1057e8c80>>>>next(integers)0>>>next(integers)1>>>next(integers)2>>>next(integers)3>>>next(integers)

Theinfinite_integers() function returns a generator iterator, which is stored inintegers. This iterator yields values on demand, but remember, there will be infinite values. Because of this, it won’t be a good idea to use the membership operators with this iterator. Why? Well, if the target value isn’t in the generator iterator, then you’ll run into an infinite loop that’ll make your executionhang.

Dictionaries and Sets

Python’s membership operators also work with dictionaries and sets. If you use thein ornot in operators directly on a dictionary, then it’ll check whether the dictionary has a given key or not. You can also do this check using the.keys() method, which is more explicit about your intentions.

You can also check if a given value or key-value pair is in a dictionary. To do these checks, you can use the.values() and.items() methods, respectively:

>>>likes={"color":"blue","fruit":"apple","pet":"dog"}>>>"fruit"inlikesTrue>>>"hobby"inlikesFalse>>>"blue"inlikesFalse>>>"fruit"inlikes.keys()True>>>"hobby"inlikes.keys()False>>>"blue"inlikes.keys()False>>>"dog"inlikes.values()True>>>"drawing"inlikes.values()False>>>("color","blue")inlikes.items()True>>>("hobby","drawing")inlikes.items()False

In these examples, you use thein operator directly on yourlikes dictionary to check whether the"fruit","hobby", and"blue" keys are in the dictionary or not. Note that even though"blue" is a value inlikes, the test returnsFalse because it only considers the keys.

Next up, you use the.keys() method to get the same results. In this case, the explicit method name makes your intentions much clearer to other programmers reading your code.

To check if a value like"dog" or"drawing" is present inlikes, you use the.values() method, which returns aview object with the values in the underlying dictionary. Similarly, to check if a key-value pair is contained inlikes, you use.items(). Note that the target key-value pairs must be two-item tuples with the key and value in that order.

If you’re using sets, then the membership operators work as they would with lists or tuples:

>>>fruits={"apple","banana","cherry","orange"}>>>"banana"infruitsTrue>>>"banana"notinfruitsFalse>>>"grape"infruitsFalse>>>"grape"notinfruitsTrue

These examples show that you can also check whether a given value is contained in a set by using the membership operatorsin andnot in.

Now that you know how thein andnot in operators work with different built-in data types, it’s time to put these operators into action with a couple of examples.

Replacing Chainedor Operators

Using a membership test to replace a compound Boolean expression with severalor operators is a useful technique that allows you to simplify your code and make it more readable.

To see this technique in action, say that you need to write a function that takes a color name as a string and determines whether it’s a primary color. To figure this out, you’ll use theRGB (red, green, and blue) color model:

>>>defis_primary_color(color):...color=color.lower()...returncolor=="red"orcolor=="green"orcolor=="blue"...>>>is_primary_color("yellow")False>>>is_primary_color("green")True

Inis_primary_color(), you use a compound Boolean expression that uses theor operator to check if the input color is either red, green, or blue. Even though this function works as expected, the condition may be confusing and difficult to read and understand.

The good news is that you can replace the above condition with a compact and readable membership test:

>>>defis_primary_color(color):...primary_colors={"red","green","blue"}...returncolor.lower()inprimary_colors...>>>is_primary_color("yellow")False>>>is_primary_color("green")True

Now your function uses thein operator to check whether the input color is red, green, or blue. Assigning the set of primary colors to a properly named variable likeprimary_colors also helps to make your code more readable. The final check is pretty clear now. Anyone reading your code will immediately understand that you’re trying to determine if the input color is a primary color according to the RGB color model.

If you look at the example again, then you’ll notice that the primary colors have been stored in a set. Why? You’ll find your answer in the following section.

Writing Efficient Membership Tests

Python uses adata structure called ahash table to implement dictionaries and sets. Hash tables have a remarkable property: looking for any given value in the data structure takes about the same time, no matter how many values the table has. Using Big O notation, you’ll say that value lookups in hash tables have a time complexity ofO(1), which makes them super fast.

Now, what does this feature of hash tables have to do with membership tests on dictionaries and sets? Well, it turns out that thein andnot in operators work very quickly when they operate on these types. This detail allows you to optimize your code’s performance by favoring dictionaries and sets over lists and other sequences in membership tests.

To have an idea of how much more efficient than a list a set can be, go ahead and create the following script:

fromtimeitimporttimeita_list=list(range(100_000))a_set=set(range(100_000))list_time=timeit("-1 in a_list",number=1,globals=globals())set_time=timeit("-1 in a_set",number=1,globals=globals())print(f"Sets are{(list_time/set_time):.2f} times faster than Lists")

This script creates a list of integer numbers with one hundred thousand values and a set with the same number of elements. Then the script computes the time that it takes to determine if the number-1 is in the list and the set. You know up front that-1 doesn’t appear in the list or set. So, the membership operator will have to check all the values before getting a final result.

As you already know, when thein operator searches for a value in a list, it uses an algorithm with a time complexity ofO(n). On the other hand, when thein operator searches for a value in a set, it uses the hash table lookup algorithm, which has a time complexity ofO(1). This fact can make a big difference in terms of performance.

Go ahead andrun your script from the command line using the following command:

$pythonperformance.pySets are 1563.33 times faster than Lists

Although your command’s output may be slightly different, it’ll still show a significant performance difference when you use a set instead of a list in this specific membership test. With a list, the processing time will be proportional to the number of values. With a set, the time will be pretty much the same for any number of values.

This performance test shows that when your code is doing membership checks on large collections of values, you should use sets instead of lists whenever possible. You’ll also benefit from sets when your code performs several membership tests during its execution.

However, note that it’s not a good idea to convert an existing list into a set just to perform a few membership tests. Remember that converting a list into a set is an operation withO(n) time complexity.