1.1.1Getting ready

There are three general cases for expressions that involve numbers beyond integers, which are:

1. Currency: Dollars, cents, euros, and so on. Currency generally has a fixed number of decimal places and rounding rules to properly quantize results.

2. Rational Numbers or Fractions: When we scale a recipe that serves eight, for example, down to five people, we’re doing fractional math using a scaling factor of.

3. Floating Point: This includes all other kinds of calculations. This also includes irrational numbers, likeπ, root extraction, and logarithms.

When we have one of the first two cases, we should avoid floating-point numbers.

1.1.2How to do it...

We’ll look at each of the three cases separately.

>>> import decimal  >>> total_amount.quantize(penny, decimal.ROUND_UP)  Decimal(’3.17’)

1.1.3How it works...

For these numeric types, Python offers a variety of operators:+,-,*,/,//,%, and**. These are for addition, subtraction, multiplication, true division, truncated division, modulo, and raising to a power, respectively. We’ll look at the two division operators,/ and//, in theChoosing between true division and floordivision recipe.

Python will do some conversions between the various numeric types. We can mixint andfloat values; the integers will be promoted to floating-point to provide the most accurate answer possible. Similarly, we can mixint andFraction as well as mixingint andDecimal. Note that we cannot casually mixDecimal with float orFraction; an explicit conversion function will be required.

It’s important to note that float values are approximations. The Python syntax allows us to write floating-point values using base 10 digits; however, that’s not how values are represented internally.

We can write the value 8.066×10⁶⁷ like this in Python:

>>> 8.066e+67  8.066e+67

The actual value used internally will involve a binary approximation of the decimal value we wrote. The internal value for this example is this:

>>> (6737037547376141/(2**53))*(2**226)  8.066e+67

The numerator is a big number,6737037547376141. The denominator is always 2⁵³. This is why values can get truncated.

We can use themath.frexp() function to see these internal details of a number:

>>> import math  >>> math.frexp(8.066E+67)  (0.7479614202861186, 226)

The two parts are called themantissa(orsignificand) and theexponent. If we multiply the mantissa by 2⁵³, we always get a whole number, which is the numerator of the binary fraction.

Unlike the built-in float, aFraction is an exact ratio of two integer values. We can create ratios that involve integers with a very large number of digits. We’re not limited by a fixed denominator.

ADecimal value, similarly, is based on a very large integer value, as well as a scaling factor to determine where the decimal place goes. These numbers can be huge and won’t suffer from peculiar representation issues.

1.1.4There’s more...

The Pythonmath module contains several specialized functions for working with floating-point values. This module includes common elementary functions such as square root, logarithms, and various trigonometry functions. It also has some other functions such as gamma, factorial, and the Gaussian error function.

Themath module includes several functions that can help us do more accurate floating-point calculations. For example, themath.fsum() function will compute a floating-point sum more carefully than the built-insum() function. It’s less susceptible to approximation issues.

We can also make use of themath.isclose() function to compare two floating-point values, an expression, and a literal1.0, to see if they’re nearly equal:

>>> (19/155)*(155/19) == 1.0  False  >>> math.isclose((19/155)*(155/19), 1.0)  True

This function provides us with a way to compare two floating-point numbers meaningfully for near-equality.

Python also offerscomplexnumbers. A complex number has a real and an imaginary part. In Python, we write3.14+2.78j to represent the complex number 3.14 + 2.78. Python will comfortably convert between float and complex. We have the usual group of operators available for complex numbers.

To support complex numbers, there’s thecmath package. Thecmath.sqrt() function, for example, will return a complex value rather than raise an exception when extracting the square root of a negative number. Here’s an example:

>>> math.sqrt(-2)  Traceback (most recent call last):  ...  ValueError: math domain error  >>> import cmath  >>> cmath.sqrt(-2)  1.4142135623730951j

This module is helpful when working with complex numbers.

1.1.5See also

• We’ll talk more about floating-point numbers and fractions in theChoosing between true division and floor division recipe.

• Seehttps://en.wikipedia.org/wiki/IEEE_floating_point.

1.2.1Getting ready

There are several general cases for division:

• Adiv-modpair: We want both parts – the quotient and the remainder. The name refers to the division and modulo operations combined together. We can summarize the quotient and remainder asq,r= (⌊⌋,a modb).

  We often use this when converting values from one base into another. When we convert seconds into hours, minutes, and seconds, we’ll be doing adiv-modkind of division. We don’t want the exact number of hours; we want a truncated number of hours, and the remainder will be converted into minutes and seconds.

• Thetruevalue: This is a typical floating-point value; it will be a good approximation to the quotient. For example, if we’re computing an average of several measurements, we usually expect the result to be floating-point, even if the input values are all integers.

• Arational fractionvalue: This is often necessary when working in American units of feet, inches, and cups. For this, we should be using theFraction class. When we divideFraction objects, we always get exact answers.

We need to decide which of these cases apply, so we know which division operator to use.

1.2.2How to do it...

We’ll look at these three cases separately.

>>> total_seconds = 7385  >>> hours = total_seconds / 3600  >>> round(hours, 4)  2.0514

1.2.3How it works...

Python has two division operators:

• The/ true division operator produces a true, floating-point result. It does this even when the two operands are integers. This is an unusual operator in this respect. All other operators preserve the type of the data. The true division operation – when applied to integers – produces a float result.

• The// truncated division operator always produces a truncated result. For two integer operands, this is the truncated quotient. When floating-point operands are used, this is a truncated floating-point result:

  >>> 7358.0 // 3600.0  2.0

1.2.4See also

• For more on the choice between floating-point and fractions, see theChoosing between float, decimal, and fraction recipe.

• SeePEP-238.

1.3.1Getting ready

The easiest way to decompose a complex string is by generalizing the string into a pattern and then writing a regular expression that describes that pattern.

There are limits to the patterns that regular expressions can describe. When we’re confronted with deeply nested documents in a language like HTML, XML, or JSON, we often run into problems and be prohibited from using regular expressions.

There module contains all of the various classes and functions we need to create and use regular expressions.

Let’s say that we want to decompose text from a recipe website. Each line looks like this:

>>> ingredient = "Kumquat: 2 cups"

We want to separate the ingredient from the measurements.

1.3.2How to do it...

To write and use regular expressions, we often do this:

1. Generalize the example. In our case, we have something that we can generalize as:

   (ingredient words): (amount digits) (unit words)

2. We’ve replaced literal text with a two-part summary: what it means and how it’s represented. For example,ingredient is represented as words, whileamount is represented as digits. Import there module:

   >>> import re

3. Rewrite the pattern intoregular expression(RE) notation:

   >>> pattern_text = r’([\w\s]+):\s+(\d+)\s+(\w+)’

   We’ve replaced representation hints such asingredient words, a mixture of letters and spaces, with[\w\s]+. We’ve replacedamount digits with\d+. And we’ve replacedsingle spaceswith\s+ to allow one or more spaces to be used as punctuation. We’ve left the colon in place because, in regular expression notation, a colon matches itself.

   For each of the fields of data, we’ve used() to capture the data matching the pattern. We didn’t capture the colon or the spaces because we don’t need the punctuation characters.

   REs typically use a lot of\ characters. To make this work out nicely in Python, we almost always userawstrings. Ther’ tells Python not to look at the\ characters and not to replace them with special characters that aren’t on our keyboards.

4. Compile thepattern:

   >>> pattern = re.compile(pattern_text)

5. Match the pattern against the input text. If the input matches the pattern, we’ll get amatch object that shows details of the substring that matched:

   >>> match = pattern.match(ingredient)  >>> match is None  False  >>> match.groups()  (’Kumquat’, ’2’, ’cups’)

6. Extract the named groups of characters from thematch object:

   >>> match.group(1)  ’Kumquat’  >>> match.group(2)  ’2’  >>> match.group(3)  ’cups’

Each group is identified by the order of the capture() portions of the regular expression. This gives us a tuple of the different fields captured from the string. We’ll return to the use of the tuple data structure in theUsing tuples of items recipe. This can be confusing in more complex regular expressions; there is a way to provide a name, instead of the numeric position, to identify a capture group.

1.3.3How it works...

There are a lot of different kinds of string patterns that we can describe with regular expressions.

We’ve shown a number of character classes:

• \w matches any alphanumeric character (a to z, A to Z, 0 to 9).

• \d matches any decimal digit.

• \s matches any space or tab character.

These classes also have inverses:

• \W matches any character that’s not a letter or a digit.

• \D matches any character that’s not a digit.

• \S matches any character that’s not some kind of space or tab.

Many characters match themselves. Some characters, however, have a special meaning, and we have to use\ to escape from that special meaning:

• We saw that+ as a suffix means to match one or more of the preceding patterns.\d+ matches one or more digits. To match an ordinary+, we need to use\+.

• We also have* as a suffix, which matches zero or more of the preceding patterns.\w* matches zero or more characters. To match a*, we need to use\*.

• We have? as a suffix, which matches zero or one of the preceding expressions. This character is used in other places, and has a different meaning in the other context. We’ll see it used in?P<name>...)|, where it is inside \verb|)— to define special properties for the grouping.

• The. character matches any single character. To match a. specifically, we need to use\..

We can create our own unique sets of characters using[] to enclose the elements of the set. We might have something like this:

(?P<name>\w+)\s*[=:]\s*(?P<value>.*)

This has a\w+ to match any number of alphanumeric characters. This will be collected into a group calledname. It uses\s* to match an optional sequence of spaces. It matches any character in the set[=:]. Exactly one of the two characters in this set must be present. It uses\s* again to match an optional sequence of spaces. Finally, it uses.* to match everything else in the string. This is collected into a group namedvalue.

We can use this to parse strings, like this:

size = 12  weight: 14

By being flexible with the punctuation, we can make a program easier to use. We’ll tolerate any number of spaces, and either an= or a: as a separator.

1.3.4There’s more...

A long regular expression can be awkward to read. We have a clever Pythonic trick for presenting an expression in a way that’s much easier to read:

>>> ingredient_pattern = re.compile(  ... r’(?P<ingredient>[\w\s]+):\s+’ # name of the ingredient up to the ":"  ... r’(?P<amount>\d+)\s+’ # amount, all digits up to a space  ... r’(?P<unit>\w+)’ # units, alphanumeric characters  ... )

This leverages three syntax rules:

• A statement isn’t finished until the() characters match.

• Adjacent string literals are silently concatenated into a single long string.

• Anything between# and the end of the line is a comment, and is ignored.

We’ve put Python comments after the important clauses in our regular expression. This can help us understand what we did, and perhaps help us diagnose problems later.

We can also use the regular expression’s ”verbose” mode to add gratuitous whitespace and comments inside a regular expression string. To do this, we must usere.X as an option when compiling a regular expression to make whitespace and comments possible. This revised syntax looks like this:

>>> ingredient_pattern_x = re.compile(r’’’  ... (?P<ingredient>[\w\s]+):\s+ # name of the ingredient up to the ":"  ... (?P<amount>\d+)\s+ # amount, all digits up to a space  ... (?P<unit>\w+) # units, alphanumeric characters  ... ’’’, re.X)

We can either break the pattern up into separate string components, or make use of extended syntax to make the regular expression more readable. The benefit of providing names shows up when we use thegroupdict() method of thematch object to extract parsed values by the name associated with the pattern being captured.

1.3.5See also

• TheDecoding bytes – how to get proper characters from some bytes recipe.

• There are many books on regular expressions and Python regular expressions in particular, likeMastering Python Regular Expressionshttps://www.packtpub.com/application-development/mastering-python-regular-expressions.

1.4.1Getting ready

Let’s say we have pieces of data that we need to turn into a nicely formatted message. We might have data that includes the following:

>>> id = "IAD"  >>> location = "Dulles Intl Airport"  >>> max_temp = 32  >>> min_temp = 13  >>> precipitation = 0.4

And we’d like a line that looks like this:

IAD : Dulles Intl Airport : 32 / 13 / 0.40

1.4.2How to do it...

1. Create an f-string for the result, replacing all of the data items with placeholders. Inside each placeholder, put a variable name (or an expression.) Note that the string uses the prefix off’. This prefix creates a sophisticated string object where values are interpolated into the template when the string is used:

   f’{id} : {location} : {max_temp} / {min_temp} / {precipitation}’

2. For each name or expression, an optional data type can be appended to the names in the template string. The basic data type codes are:

   • s for string

   • d for decimal number

   • f for floating-point number

   It would look like this:

   f’{id:s} : {location:s} : {max_temp:d} / {min_temp:d} / {precipitation:f}’

   Because the book’s margins are narrow, the string has been broken to

   fit on the page. It’s a single (very wide) line of code.

3. Add length information where required. Length is not always required, and in some cases, it’s not even desirable. In this example, though, the length information ensures that each message has a consistent format. For strings and decimal numbers, prefix the format with the length like this:19s or3d. For floating-point numbers, use a two-part prefix like5.2f to specify the total length of five characters, with two to the right of the decimal point. Here’s the whole format:

   >>> f’{id:3s} : {location:19s} : {max_temp:3d} / {min_temp:3d} / {precipitation:5.2f}’  ’IAD : Dulles Intl Airport :  32 /  13 /  0.40’

1.4.3How it works...

F-strings can do a lot of relatively sophisticated string assembly by interpolating data into a template. There are a number of conversions available.

We’ve seen three of the formatting conversions—s,d,f—but there are many others. Details can be found in theFormatted string literalssection of thePython Standard Library:https://docs.python.org/3/reference/lexical_analysis.html\#formatted-string-literals.

Here are some of the format conversions we might use:

• b is for binary, base 2.

• c is for Unicode character. The value must be a number, which is converted into a character. Often, we use hexadecimal numbers for these characters, so you might want to try values such as 0x2661 through 0x2666 to see interesting Unicode glyphs.

• d is for decimal numbers.

• E ande are for scientific notations. 6.626E-34 or 6.626e-34, depending on whichE ore character is used.

• F andf are for floating-point. Fornot a number, thef format shows lowercasenan; theF format shows uppercaseNAN.

• G andg are for general use. This switches automatically betweenE andF (ore andf) to keep the output in the given sized field. For a format of20.5G, up to 20-digit numbers will be displayed usingF formatting. Larger numbers will useE formatting.

• n is for locale-specific decimal numbers. This will insert , or . characters, depending on the current locale settings. The default locale may not have 1,000 separators defined. For more information, see the locale module.

• o is for octal, base 8.

• s is for string.

• X andx are for hexadecimal, base 16. The digits include uppercase A-F and lowercase a-f, depending on whichX orx format character is used.

• % is for percentage. The number is multiplied by 100 and the output includes a% character.

We have a number of prefixes we can use for these different types. The most common one is the length. We might use{name:5d} to put in a 5-digit number. There are several prefixes for the preceding types:

• Fill and alignment: We can specify a specific filler character (space is the default) and an alignment. Numbers are generally aligned to the right and strings to the left. We can change that using<,>, or^. This forces left alignment, right alignment, or centering, respectively. There’s a peculiar= alignment that’s used to put padding after a leading sign.

• Sign: The default rule is a leading negative sign where needed. We can use+ to put a sign on all numbers,- to put a sign only on negative numbers, and a space to use a space instead of a plus for positive numbers. In scientific output, we often use{value:5.3f}. The space makes sure that room is left for the sign, ensuring that all the decimal points line up nicely.

• Alternate form: We can use the# to get an alternate form. We might have something like{0:#x},{0:#o}, or{0:#b} to get a prefix on hexadecimal, octal, or binary values. With a prefix, the numbers will look like0xnnn,0onnn, or0bnnn. The default is to omit the two-character prefix.

• Leading zero: We can include0 to get leading zeros to fill in the front of a number. Something like{code:08x} will produce a hexadecimal value with leading zeroes to pad it out to eight characters.

• Width and precision: For integer values and strings, we only provide the width. For floating-point values, we often providewidth.precision.

There are some times when we won’t use a{name:format} specification. Sometimes, we’ll need to use a{name!conversion} specification. There are only three conversions available:

• {name!r} shows the representation that would be produced byrepr(name).

• {name!s} shows the string value that would be produced bystr(name); this is the default behavior if you don’t specify any conversion. Using!s explicitly lets you add string-type format specifiers.

• {name!a} shows the ASCII value that would be produced byascii(name).

• Additionally, there’s a handy debugging format specifier available. We can include a trailing equals sign,=, to get a handy dump of a variable or expression. The following example uses both forms:

  >>> value = 2**12-1  >>> f’{value=} {2**7+1=}’  ’value=4095 2**7+1=129’

The f-string showed the value of the variable namedvalue and the result of an expression,2**7+1.

InChapter 7, we’ll leverage the idea of the{name!r} format specification to simplify displaying information about related objects.

1.4.4There’s more...

The f-string processing relies on the stringformat() method. We can leverage this method and the relatedformat_map() method for cases where we have more complex data structures.

Looking forward toChapter 5, we might have a dictionary where the keys are simple strings that fit with theformat_map() rules:

>>> data = dict(  ... id=id, location=location, max_temp=max_temp,  ... min_temp=min_temp, precipitation=precipitation  ... )  >>> ’{id:3s} : {location:19s} : {max_temp:3d} / {min_temp:3d} / {precipitation:5.2f}’.format_map(data)  ’IAD : Dulles Intl Airport :  32 /  13 /  0.40’

We’ve created a dictionary object, data, that contains a number of values with keys that are valid Python identifiers:id,location,max_temp,min_temp, andprecipitation. We can then use this dictionary with theformat_map() method to extract values from the dictionary using the keys.

Note that the formatting template here isnotan f-string. It doesn’t have thef" prefix. Instead of using the automatic formatting features of an f-string, we’ve done the interpolation ”the hard way” using theformat_map() method of an f-string.

1.4.5See also

• More details can be found in theFormatted string literalssection of thePython Standard Library:https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals.

1.5.1Getting ready

Here’s a string that we’d like to rearrange:

>>> title = "Recipe 5: Rewriting an Immutable String"

We’d like to do two transformations:

• Remove the part before:.

• Replace the punctuation with_ and make all the characters lowercase.

We’ll make use of thestring module:

>>> from string import whitespace, punctuation

This has two important constants:

• string.whitespace lists all of the whitespace characters that are also part of ASCII, including space and tab.

• string.punctuation lists punctuation marks that are also part of ASCII. Unicode has a large domain of punctuation marks. This is a widely used subset.

1.5.2How to do it...

We can work with a string exploded into a list. We’ll look at lists in more depth inChapter 4:

1. Explode the string into alist object:

   >>> title_list = list(title)

2. Find the partition character. Theindex() method for a list has the same semantics as theindex() method has for a string. It locates the position with the given value:

   >>> colon_position = title_list.index(’:’)

3. Delete the characters that are no longer needed. Thedel statement can remove items from a list. Unlike strings, lists are mutable data structures:

   >>> del title_list[:colon_position+1]

4. Replace punctuation by stepping through each position. In this case, we’ll use afor statement to visit every index in the string:

   >>> for position in range(len(title_list)):  ...     if title_list[position] in whitespace+punctuation:  ...         title_list[position]= ’_’

5. The expressionrange(len(title_list)) generates all of the values between 0 andlen(title_list)-1. This assures us that the value of position will be each value index in the list. Join the list of characters to create a new string. It seems a little odd to use a zero-length string,’’, as a separator when concatenating strings together. However, it works perfectly:

   >>> title = ’’.join(title_list)  >>> title  ’_Rewriting_an_Immutable_String’

We assigned the resulting string back to the original variable. The original string object, which had been referred to by that variable, is no longer needed: it’s automatically removed from memory (this is known asgarbage collection). The new string object replaces the value of the variable.

1.5.3How it works...

This is a change in representation trick. Since a string is immutable, we can’t update it. We can, however, convert it into a mutable form; in this case, a list. We can make whatever changes are required to the mutable list object. When we’re done, we can change the representation from a list back to a string and replace the original value of the variable.

Lists provide some features that strings don’t have. Conversely, strings provide a number of features lists don’t have. As an example, we can’t convert a list into lowercase the way we can convert a string.

There’s an important trade-off here:

• Strings are immutable, which makes them very fast. Strings are focused on Unicode characters. When we look at mappings and sets, we can use strings as keys for mappings and items in sets because the value is immutable.

• Lists are mutable. Operations are slower. Lists can hold any kind of item. We can’t use a list as a key for a mapping or an item in a set because the list value could change.

Strings and lists are both specialized kinds of sequences. Consequently, they have a number of common features. The basic item indexing and slicing features are shared. Similarly, a list uses the same kind of negative index values that a string does: the expressionlist[-1] is the last item in a list object.

We’ll return to mutable data structures inChapter 4.

1.5.4See also

• Sometimes, we need to build a string, and then convert it into bytes. See theEncoding strings – creating ASCII and UTF-8 bytes recipe for how we can do this.

• Other times, we’ll need to convert bytes into a string. See theDecodingbytes – how to get proper characters from some bytes recipe for more information.

1.6.1Getting ready

Python works in Unicode. There are thousands of individual Unicode characters available.

We can see all the available characters athttps://en.wikipedia.org/wiki/List_of_Unicode_characters, as well as athttp://www.unicode.org/charts/.

We’ll need the Unicode character number. We may also want the Unicode character name.

A given font on our computer may not be designed to provide glyphs for all of those characters. In particular, Windows computer fonts may have trouble displaying some of these characters. Using the following Windows command to change to code page 65001 is sometimes necessary:

chcp 65001

Linux and macOS rarely have problems with Unicode characters.

1.6.2How to do it...

Python usesescape sequencesto extend the ordinary characters we can type to cover the vast space of Unicode characters. Each escape sequence starts with a\ character. The next character tells us exactly which of the Unicode characters to create. Locate the character that’s needed. Get the name or the number. The numbers are always given as hexadecimal, base 16. Websites describing Unicode often write the character asU+2680. The name might beDIE FACE-1. Use\unnnn with up to a four-digit number,nnnn. Or, use\N{name} with the spelled-out name. If the number is more than four digits, use\Unnnnnnnn with the number padded out to exactly eight digits:

>>> ’You Rolled \u2680’  ’You Rolled ’  >>> ’You drew \U0001F000’  ’You drew ’  >>> ’Discard \N{MAHJONG TILE RED DRAGON}’  ’Discard ’

Yes, we can include a wide variety of characters in Python output. To place a\ in the string without the following characters being part of an escape sequence, we need to use\\. For example, we might need this for Windows file paths.

1.6.3How it works...

Python uses Unicode internally. The 128 or so characters we can type directly using the keyboard all have handy internal Unicode numbers.

When we write:

’HELLO’

Python treats it as shorthand for this:

’\u0048\u0045\u004c\u004c\u004f’

Once we get beyond the characters on our keyboards, the remaining thousands of characters are identified only by their number.

When the string is being compiled by Python,\uxxxx,\Uxxxxxxxx, and\N{name} are all replaced by the proper Unicode character. If we have something syntactically wrong—for example,\N{name with no closing}—we’ll get an immediate error from Python’s internal syntax checking.

Regular expressions use a lot of\ characters and that we specifically do not want Python’s normal compiler to touch them; we used ther’ prefix on a regular expression string to prevent\ from being treated as an escape and possibly converted into something else. To use the full domain of Unicode characters, we cannot avoid using\ as an escape.

What if we need to use Unicode in a regular expression? We’ll need to use\\ all over the place in the regular expression. We might see something like this:’\\w+[\u2680\u2681\u2682\u2683\u2684\u2685]\\d+’.

We couldn’t use ther’ prefix on the string because we needed to have the Unicode escapes processed. This forced us to use\\ for elements of the regular expression. We used\uxxxx for the Unicode characters that are part of the pattern. Python’s internal compiler will replace\uxxxx with Unicode characters and\\w will become the required\w internally.

When we look at a string at the>>> prompt, Python will display the string in its canonical form. Python prefers to display strings with’ as a delimiter, using" when the string contains a’. We can use either’ or" for a string delimiter when writing code. Python doesn’t generally display raw strings; instead, it puts all of the necessary escape sequences back into the string:

>>> r"\w+"  ’\\w+’

We provided a string in raw form. Python displayed it in canonical form.

1.6.4See also

• In theEncoding strings – creating ASCII and UTF-8 bytes and theDecoding bytes – how to get proper characters from some bytes recipes, we’ll look at how Unicode characters are converted into sequences of bytes so we can write them to a file. We’ll look at how bytes from a file (or downloaded from a website) are turned into Unicode characters so they can be processed.

• If you’re interested in history, you can read up on ASCII and EBCDIC and other old-fashioned character codes here:http://www.unicode.org/charts/.

1.7.1Getting ready

Historically, a character occupied 1 byte. Python leverages the old ASCII encoding scheme for bytes; this sometimes leads to confusion between bytes and text strings of Unicode characters.

Unicode characters are encoded into sequences of bytes. There are a number of standardized encodings and a number of non-standard encodings.

Plus, there also are some encodings that only work for a small subset of Unicode characters. We try to avoid these, but there are some situations where we’ll need to use a subset encoding scheme.

Unless we have a really good reason not to, we almost always use UTF-8 encoding for Unicode characters. Its main advantage is that it’s a compact representation of the Latin alphabet, which is used for English and a number of European languages.

Sometimes, an internet protocol requires ASCII characters. This is a special case that requires some care because the ASCII encoding can only handle a small subset of Unicode characters.

1.7.2How to do it...

Python will generally use our OS’s default encoding for files and internet traffic. The details are unique to each OS:

1. We can make a general setting using thePYTHONIOENCODING environment variable. We set this outside of Python to ensure that a particular encoding is used everywhere. When using Linux or macOS, use the shell’sexport statement to set the environment variable. For Windows, use theset command, or the PowerShellSet-Item cmdlet. For Linux, it looks like this:

   (cookbook3) % export PYTHONIOENCODING=UTF-8

2. Run Python:

   (cookbook3) % python

3. We sometimes need to make specific settings when we open a file inside our script. We’ll return to this topic inChapter 11. Open the file with a given encoding. Read or write Unicode characters to the file:

   >>> with open(’some_file.txt’, ’w’, encoding=’utf-8’) as output:  ...     print(’You drew \U0001F000’, file=output)  >>> with open(’some_file.txt’, ’r’, encoding=’utf-8’) as input:  ...     text = input.read()  >>> text  ’You drew ’

We can also manually encode characters, in the rare case that we need to open a file in bytes mode; if we use a mode ofwb, we’ll also need to use manual encoding of each string:

>>> string_bytes = ’You drew \U0001F000’.encode(’utf-8’)  >>> string_bytes  b’You drew \xf0\x9f\x80\x80’

We can see that a sequence of bytes (\xf0\x9f\x80\x80) was used to encode a single Unicode character,U+1F000,.

1.7.3How it works...

Unicode defines a number of encoding schemes. While UTF-8 is the most popular, there is also UTF-16 and UTF-32. The number is the typical number of bits per character. A file with 1,000 characters encoded in UTF-32 would be 4,000 8-bit bytes. A file with 1,000 characters encoded in UTF-8 could be as few as 1,000 bytes, depending on the exact mix of characters. In UTF-8 encoding, characters with Unicode numbers above U+007F require multiple bytes.

Various OSes have their own coding schemes. macOS files can be encoded in Mac Roman or Latin-1. Windows files might use CP1252 encoding.

The point with all of these schemes is to have a sequence of bytes that can be mapped to a Unicode character and—going the other way—a way to map each Unicode character to one or more bytes. Ideally, all of the Unicode characters are accounted for. Pragmatically, some of these coding schemes are incomplete.

The historical form of ASCII encoding can only represent about 100 of the Unicode characters as bytes. It’s easy to create a string that cannot be encoded using the ASCII scheme.

Here’s what the error looks like:

>>> ’You drew \U0001F000’.encode(’ascii’)  Traceback (most recent call last):  ...  UnicodeEncodeError: ’ascii’ codec can’t encode character ’\U0001f000’ in position 9: ordinal not in range(128

We may see this kind of error when we accidentally open a file with an encoding that’s not the widely used standard of UTF-8. When we see this kind of error, we’ll need to change our processing to select the encoding actually used to create the file. It’s almost impossible to guess what encoding was used, so some research may be required to locate metadata about the file that states the encoding.

Bytes are often displayed using printable characters. We’ll seeb’hello’ as shorthand for a five-byte value. The letters are chosen using the old ASCII encoding scheme, where byte values from 0x20 to 0x7F will be shown as characters, and outside this range, more complex-looking escapes will be used.

This use of characters to represent byte values can be confusing. The prefix ofb’ is our hint that we’re looking at bytes, not proper Unicode characters.

1.7.4See also

• There are a number of ways to build strings of data. See theBuildingcomplicated strings with f-strings and theBuilding complicated stringsfrom lists of strings recipes for examples of creating complex strings. The idea is that we might have an application that builds a complex string, and then we encode it into bytes.

• For more information on UTF-8 encoding, seehttps://en.wikipedia.org/wiki/UTF-8.

• For general information on Unicode encodings, seehttp://unicode.org/faq/utf_bom.html.

1.8.1Getting ready

Let’s say we’re interested in offshore marine weather forecasts. Perhaps this is because we are departing theChesapeake Bayfor theCaribbean.

Are there any special warnings coming from theNational Weather Services office in Wakefield, Virginia?

Here’s the link:https://forecast.weather.gov/product.php?site=AKQ&product=SMW&issuedby=AKQ.

We can download this with Python’surllib module:

>>> import urllib.request  >>> warnings_uri = (  ...     ’https://forecast.weather.gov/’  ...     ’product.php?site=AKQ&product=SMW&issuedby=AKQ’  ... )  >>> with urllib.request.urlopen(warnings_uri) as source:  ...     forecast_text = source.read()

Note that we’ve enclosed the URI string in() and broken it into two separate string literals. Python will concatenate these two adjacent literals into a single string. We’ll look at this in some depth inChapter 2.

As an alterative, we can use programs likecurl orwget to get this. At the OS Terminal prompt, we might run the following (long) command:

(cookbook3) % curl ’https://forecast.weather.gov/product.php?site=AKQ&product=SMW&issuedby=AKQ’ -o AKQ.html

Typesetting this book tends to break the command onto many lines. It’s really one very long line.

The code repository includes a sample file,ch01/Text Products for SMW Issuedby AKQ.html.

Theforecast_text value is a stream of bytes. It’s not a proper string. We can tell because it starts like this:

>>> forecast_text[:80]  b’<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/x’

The data goes on for a while, providing details from the web page. Because the displayed value starts withb’, it’s bytes, not proper Unicode characters. It was probably encoded with UTF-8, which means some characters could have weird-looking\xnn escape sequences instead of proper characters. We want to have the proper characters.

While this data has many easy-to-read characters, theb’ prefix shows that it’s a collection of byte values, not proper text. Generally, a bytes object behaves somewhat like a string object. Sometimes, we can work with bytes directly. Most of the time, we’ll want to decode the bytes and create proper Unicode characters from them.

1.8.2How to do it...

1. Determine the coding scheme if possible. In order to decode bytes to create proper Unicode characters, we need to know what encoding scheme was used. When we read XML documents, there’s a big hint provided within the document:

   <?xml version="1.0" encoding="UTF-8"?>

   When browsing web pages, there’s often a header containing this information:

   Content-Type: text/html; charset=ISO-8859-4

   Sometimes, an HTML page may include this as part of the header:

   <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

   In other cases, we’re left to guess. In the case of US weather data, a good first guess is UTF-8. Another good guess is ISO-8859-1. In some cases, the guess will depend on the language.

2. Thecodecs — Codec registry and base classessection of thePythonStandard Librarylists the standard encodings available. Decode the data:

   >>> document = forecast_text.decode("UTF-8")  >>> document[:80]  ’<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/x’

   Theb’ prefix is no longer used to show that these are bytes. We’ve created a proper string of Unicode characters from the stream of bytes.

3. If this step fails with an exception, we guessed wrong about the encoding. We need to try another encoding in order to parse the resulting document.

Since this is an HTML document, we should useBeautiful Soupto extract the data. Seehttp://www.crummy.com/software/BeautifulSoup/.

We can, however, extract one nugget of information from this document without completely parsing the HTML:

>>> import re  >>> content_pattern = re.compile(r"// CONTENT STARTS(.*?)// CONTENT ENDS", re.MULTILINE | re.DOTALL)  >>> content_pattern.search(document)  <re.Match object; span=(8530, 9113), match=’// CONTENT STARTS HERE -->\n\n<span0.4.8.3">
1.8.3How it works...
See theEncoding strings – creating ASCII and UTF-8 bytes recipe for more information on Unicode and the different ways that Unicode characters can be encoded into streams of bytes.
At the foundation of the OS, files and network connections are built up from bytes. It’s our software that decodes the bytes to discover the content. It might be characters, images, or sounds. In some cases, the default assumptions are wrong and we need to do our own decoding.

1.8.4See also

• Once we’ve recovered the string data, we have a number of ways of parsing or rewriting it. See theString parsing with regular expressions recipe for examples of parsing a complex string.

• For more information on encodings, seehttps://en.wikipedia.org/wiki/UTF-8 andhttp://unicode.org/faq/utf_bom.html.

1.9.1Getting ready

In theString parsing with regular expressions recipe, we skipped over an interesting data structure.

We had data that looked like this:

>>> ingredient = "Kumquat: 2 cups"

We parsed this into meaningful data using a regular expression, like this:

>>> import re  >>> ingredient_pattern = re.compile(r’(?P<ingredient>\w+):\s+(?P<amount>\d+)\s+(?P<unit>\w+)’)  >>> match = ingredient_pattern.match(ingredient)  >>> match.groups()  (’Kumquat’, ’2’, ’cups’)

The result is a tuple object with three pieces of data. There are lots of places where this kind of grouped data can come in handy.

1.9.2How to do it...

We’ll look at two aspects to this: putting things into tuples and getting things out of tuples.

>>> one_tuple = (’item’, )  >>> len(one_tuple)  1

>>> 355,  (355,)

1.9.3How it works...

Tuples are one example of the more generalSequence class. We can do a few things with sequences.

Here’s an example tuple that we can work with:

>>> t = (’Kumquat’, ’2’, ’cups’)

Here are some operations we can perform on this tuple:

• How many items int?

  >>> len(t)  3

• How many times does a particular value appear int?

  >>> t.count(’2’)  1

• Which position has a particular value?

  >>> t.index(’cups’)  2  >>> t[2]  ’cups’

• When an item doesn’t exist, we’ll get an exception:

  >>> t.index(’Rice’)  Traceback (most recent call last):  ...  ValueError: tuple.index(x): x not in tuple

• Does a particular value exist?

  >>> ’Rice’ in t  False

1.9.4There’s more...

A tuple, like a string, is a sequence of items. In the case of a string, it’s a sequence of characters. In the case of a tuple, it’s a sequence of many things. Because they’re both sequences, they have some common features. We’ve noted that we can pluck out individual items by their index position. We can use theindex() method to locate the position of an item.

The similarities end there. A string has many methods it can use to create a new string that’s a transformation of a string, plus methods to parse strings, plus methods to determine the content of the strings. A tuple doesn’t have any of these bonus features. It’s—perhaps—the simplest possible data structure.

1.9.5See also

• We looked at one other sequence, the list, in theBuilding complicatedstrings from lists of strings recipe.

• We’ll also look at sequences inChapter 4.

1.10.1Getting ready

Let’s continue looking at items in recipes. The regular expression for parsing the string had three attributes:ingredient,amount, andunit. We used the following pattern with names for the various substrings:

r’(?P<ingredient>\w+):\s+(?P<amount>\d+)\s+(?P<unit>\w+)’)

The resulting data tuple looked like this:

>>> item = match.groups()  >>> item  (’Kumquat’, ’2’, ’cups’)

While the matching betweeningredient,amount, andunit is pretty clear, using something like the following isn’t ideal. What does1 mean? Is it really the quantity?

>>> from fractions import Fraction  >>> Fraction(item[1])  Fraction(2, 1)

We want to define tuples with names, as well as positions.

1.10.2How to do it...

1. We’ll use theNamedTuple class definition from thetyping package:

   >>> from typing import NamedTuple

2. With this base class definition, we can define our own unique tuples, with names for the items:

   >>> class Ingredient(NamedTuple):  ...     ingredient: str  ...     amount: str  ...     unit: str

3. Now, we can create an instance of this unique kind of tuple by using the classname:

   >>> item_2 = Ingredient(’Kumquat’, ’2’, ’cups’)

4. When we want a value from the tuple, we can use a name instead of the position:

   >>> Fraction(item_2.amount)  Fraction(2, 1)  >>> f"Use {item_2.amount} {item_2.unit} fresh {item_2.ingredient}"  ’Use 2 cups fresh Kumquat’

1.10.3How it works...

TheNamedTuple class definition introduces a core concept fromChapter 7. We’ve extended the base class definition to add unique features for our application. In this case, we’ve named the three attributes eachIngredient tuple must contain.

Because a subclass ofNamedTuple class is a tuple, the order of the attribute names is fixed. We can use a reference like the expressionitem_2[0] as well as the expressionitem_2.ingredient. Both names refer to the item in index 0 of the tuple,item_2.

The core tuple types can be called ”anonymous tuples” or maybe ”index-only tuples.” This can help to distinguish them from the more sophisticated ”named tuples” introduced through thetyping module.

Tuples are very useful as tiny containers of closely related data. Using theNamedTuple class definition makes them even easier to work with.

1.10.4There’s more...

We can have a mixed collection of values in a tuple or a named tuple. We need to perform conversion before we can build the tuple. It’s important to remember that a tuple cannot ever be changed. It’s an immutable object, similar in many ways to the way strings and numbers are immutable.

For example, we might want to work with amounts that are exact fractions. Here’s a more sophisticated definition:

>>> from typing import NamedTuple  >>> from fractions import Fraction  >>> class IngredientF(NamedTuple):  ...     ingredient: str  ...     amount: Fraction  ...     unit: str

These objects require some care to create. If we’re using a bunch of strings, we can’t simply build this object from three string values; we need to convert the amount into aFraction instance. Here’s an example of creating an item using aFraction conversion:

>>> item_3 = IngredientF(’Kumquat’, Fraction(’2’), ’cups’)

This tuple has a more useful value for the amount of each ingredient. We can now do mathematical operations on the amounts:

>>> f’{item_3.ingredient} doubled: {item_3.amount * 2}’  ’Kumquat doubled: 4’

It’s very handy to explicitly state the data type within theNamedTuple class definition. It turns out Python doesn’t use the type information directly. Other tools, for example,mypy, can check the type hints in aNamedTuple against the operations in the rest of the code to be sure they agree.

1.10.5See also

• We’ll look at class definitions inChapter 7.

Join our community Discord space

Join our Python Discord workspace to discuss and find out more about the book:https://packt.link/dHrHU