Arrays in Computer Science

To understand arrays better, it helps to zoom out a bit and look at them through the lens of theory. This will clarify some baseline terminology, including:

• Abstract data types
• Data structures
• Data types

Computer science models collections of data asabstract data types (ADTs) that support certain operations like insertion or deletion of elements. These operations must satisfy additional constraints that describe the abstract data type’s unique behaviors.

The wordabstract in this context means these data types leave the implementation details up to you, only defining the expected semantics or the set of available operations that an ADT must support. As a result, you can often represent one abstract data type using a few alternativedata structures, which are concrete implementations of the same conceptual approach to organizing data.

Programming languages usually provide a few data structures in the form of built-indata types as a convenience so that you don’t have to implement them yourself. This means you can focus on solving more abstract problems instead of starting from scratch every time. For example, the Pythondict data type is ahash table data structure that implements thedictionary abstract data type.

To reiterate the meaning of these terms,abstract data types define the desired semantics,data structures implement them, anddata types represent data structures in programming languages as built-in syntactic constructs.

Some of the most common examples of abstract data types include these:

• Dictionaries
• Graphs
• Lists
• Queues
• Sets
• Stacks
• Trees

In some cases, you can build more specific kinds of abstract data types on top of existing ADTs by incorporating additional constraints. For instance, you canbuild a stack by modifying the queue or the other way around.

As you can see, the list of ADTs doesn’t include arrays. That’s because the array is a specificdata structure representing the list abstract data type. The list ADT dictates what operations the array must support and which behaviors it should exhibit. If you’ve worked with the Pythonlist, then you should already have a pretty good idea of what the list in computer science is all about.

Thelist abstract data type is a linear collection of values forming an ordered sequence of elements. These elements follow a specific arrangement, meaning that each element has a position relative to the others, identified by a numeric index that usually starts at zero. The list has a variable but finite length. It may or may not contain values of different types, as well as duplicates.

The interface of the list abstract data type resembles Python’slist, typically including the following operations:

Now that you understand where the array data structure fits into the bigger picture, it’s time to take a closer look at it.

The Array as a Data Structure

To visualize the traditional array data structure graphically, imagine a series of identically sized rectangular boxes lined up side by side, each holding a distinct value of a common type. For example, here’s what an array of integers could look like:

This array has seven elements, indexed from zero to six, all of which are integer numbers. Note that some of these values occur more than once.

At first glance, this looks just like the standardlist in Python. After all, both represent the list abstract data type covered earlier. What makes the array data structure special is a few unique features. In particular, an array must be:

• Homogeneous: All elements in an array share a common data type, allowing them to have a uniform size. This is different from Python lists, which are allowed to contain elements of mixed types.
• Contiguous: Array elements are adjacent to each other in the computer memory, occupying acontiguous block of memory space. No other data is allowed between the beginning and the end of an array. Such a layout corresponds to the way your computer memory is logically organized.
• Fixed-Size: Because other data may follow immediately after the block of memory occupied by your array, expanding the block would overwrite that data, potentially causing severe errors or malfunctions. Therefore, when you allocate memory for an array, you must know its size up front without the possibility to change it later.

The first two traits make the array arandom-access data structure, letting you instantly retrieve an element at any given index without having to traverse the whole array. When you know the memory address of the first element in the array, you can calculate the address of the desired element by applying a straightforward math formula:

To find the memory addressA_i of an element located at indexi in the array, you can take the address of your array, which is the same as the address of the first elementA₀ at index zero, and then add an offset. Because all elements have equal sizes, you can obtain the memory address offset by multiplying the target indexi by the single element’s size in bytes.

This formula works as long as all elements have an identical size and appear next to each other. The downside of such a memory layout is the array’sfixed size, which becomes apparent once you wish to append, insert, or remove elements. However, you can only simulate resizing by making another array, copying existing elements to it, and disregarding your old array.

As a result, you end up frequently asking the operating system for an ever larger block of memory, and you waste time moving data from one place to another. It’s what makes insertions and deletions so inefficient with pure arrays.

One way to mitigate this limitation is to build an abstraction on top of your array that eagerly allocates more memory than necessary and keeps track of the current number of elements against the array’s capacity. When the underlying array becomes full, you can reallocate and copy the data to a larger array. That’s roughly what the Pythonlist does under the surface to cope with dynamic resizing for you.

What about elements of the same type butvariable sizes, such as strings?

Strings can have several representations in the computer memory. For example, thelength-prefixed representation allocates an extra byte to encode the number of characters that follow. Another popular representation is thenull-terminated string, which is essentially an array of bytes terminated by a specialsentinel value called thenull character. With such a representation, you can create anarray of pointers to strings:

The array depicted above consists ofpointers, or numeric addresses, each indicating a specific memory area where the given string begins. Even though the individual elements can have variable sizes and be scattered all over your computer’s memory, their corresponding pointers are plain integers of uniform size that fit into an array.

As it turns out, the array data structure isn’t the only way to implement the list abstract data type. Next up, you’ll learn about a popular alternative—the linked list.

Arrays vs Linked Lists

At this point, you know that a list in computer science is an abstract data type and an array is one of its specific implementations. But why would you want to have more than one implementation, and how do you know which one to use?

The choice of a given data structure is vital forperformance andmemory efficiency. Most data structures offer a trade-off between the two, and they usually provide bettertime-space complexity for certain operations at the expense of others. Therefore, you should always pick a suitable data structure based on the requirements of the specific problem that you’re trying to solve.

For example, you can implement the list abstract data type using either anarray or alinked list. Both are sequences of elements, but arrays will be more suitable in scenarios involving frequent lookups, while linked lists will work better when you anticipate a greater number of insertions or deletions.

Arrays always offer instant access to arbitrary elements but require extra time to add a new element. Conversely, linked lists allow for quick insertions but may take longer to find an element by index. That trade-off is the result of how both data structures organize data in memory.

If you’re interested in how the linked list achieves efficient insertions and deletions, then check out therelevant section of the corresponding article on Wikipedia to learn more. Now, the time has finally come to reveal whether Python has arrays or not.

Arrays in Python

Although a Pythonlist represents the list abstract data type, its implementation isn’t the typical array or linked list. When you look at theCPython source code, you’ll find that the underlyingPyListObject is adynamic array ofpointers to objects, allowing Python’slist to combine the benefits of both data structures. It’s cleverly optimized, ensuring quick lookups as well as insertions under most circumstances.

Python takes the idea of keeping an array of pointers to character strings, which was outlined before, a step further. It does so by retaining extra information about the data type of a value being pointed to from an internal array.

This makes Python’slist capable of storing heterogeneous elements of arbitrary types. On the other hand, this and the over-allocation make the Pythonlist less memory-efficient than the classic array that you’d typically find in lower-level programming languages like C.

To sum up, Python internally relies on the array data structure but doesn’t directly expose it to you. The closest you can get to an array in Python is with the standard library’sarray orNumPy’sndarray data types. However, neither one of them provides a true array, as they can only holdnumeric values.

On the Real Python website, you’ll find tons ofresources on NumPy if you’d like to learn more about that library. And up next, you’ll read about thearray module.

Choose the Type of Your Array Elements

The first step before creating an instance of thearray class from thearray module is to decide on the common numeric type for its elements. Note that all array elements must have the same fixed numeric type. Once you choose from a few well-known numeric types, you can’t change your mind, so choose wisely. Because the numeric type determines the range of possible values, you won’t be able to store numbers outside that range in your array.

Unlike a Pythonlist, which accepts arbitrary kinds of objects, a Pythonarray can only accommodate numbers that fit a precise numeric type defined at the time of its creation. To use a real-life analogy, you can think of a Pythonlist as a conveyor belt at a grocery store checkout counter. In contrast, a Pythonarray is more like a carton specifically designed to hold eggs of a certain size and nothing else.

Python provides comparatively fewnumeric types through its syntax. They are:

• Integers
• Floating-point numbers
• Complex numbers

Integers in Python havearbitrary precision, making them theoretically unbounded. Floating-point numbers and, by extension, complex numbers have a hard limit defined in thesys.float_info constant.

The Python standard library also ships with more specific types, includingfractions anddecimal numbers, which are built on top of the primitive types listed above. Beyond that, you’re on your own. While the numeric types available in Python cover the majority of high-level use cases, you might face some challenges when you start dealing with low-level binary data.

To address these challenges, thearray module gives you much more granular control over the range of numbers. It exposes numeric types from the C programming language that are unavailable in standard Python. For example, you can use this module to declare an array oflong integers orsingle-precision floats. Additionally, you can specify whether these numbers should besigned or unsigned, whereas all numbers in pure Python are always signed.

There are thirteen numeric types in thearray module, each represented by a single-lettertype code, which you can reveal by accessing thetypecodes variable:

>>>importarray>>>array.typecodes'bBuhHiIlLqQfd'

Uppercase letters correspond to unsigned numbers, while lowercase letters represent signed ones. Their specific meanings are as follows:

As you can see, there are five pairs of integers consisting of the signed and unsigned flavor, two consistently signed floating-point types, and onewide character type that’s been deprecated since Python 3.3.

The more bytes you allocate, the greater the range of numbers you can represent. For example, asigned short requires two bytes or sixteen bits in memory, which gives you a range of numbers between -32,768 and 32,767. With its unsigned counterpart, you lose the ability to represent negative numbers, but you gain positive numbers up to 65,535. In other words, the range of representable values shifts upward.

Generally, knowing the number of bytes and whether the numeric type is signed lets you find the minimum and maximum values like so:

>>>defget_range(num_bytes,signed=True):...num_bits=num_bytes*8...ifsigned:...return-2**(num_bits-1),2**(num_bits-1)-1...else:...return0,(2**num_bits)-1...>>>get_range(num_bytes=2,signed=False)(0, 65535)>>>get_range(num_bytes=2,signed=True)(-32768, 32767)>>>get_range(num_bytes=4,signed=False)(0, 4294967295)>>>get_range(num_bytes=4,signed=True)(-2147483648, 2147483647)>>>get_range(num_bytes=8,signed=False)(0, 18446744073709551615)>>>get_range(num_bytes=8,signed=True)(-9223372036854775808, 9223372036854775807)

The resulting tuples represent the minimum and maximum values included in each range of numbers.

Note that the table you saw earlier indicates theminimum number of bytes required to store a value of each data type. In practice, the actual number of bytes may be higher, depending on your system architecture and how the Python interpreter was compiled. For example, theint data type will occupy four bytes instead of just two on most modern computers:

>>>fromarrayimportarray>>>array("i").itemsize4>>>array("i").typecode'i'

You can check a concrete array’s.itemsize attribute to find out exactly how many bytes one of its elements will consume in memory. Similarly, you can read the array’s.typecode attribute to show the letter that you used to create that array. Both attributes are read-only.

If you’re loading data from an external source into an array, then you should pick the type that exactly matches the corresponding format. Sometimes, it may not be possible to map a nonstandard numeric type onto one of the type codes in thearray module. For example,OpenEXR files often use16-bit floating-point numbers, while someWAV files encode data using24-bit signed integers. Later, you’ll learn how to cope with such situations.

Otherwise, if you’re writing data of your own, then use the smallest type that can fit all possible values to optimize the memory use. Say you had an array of people’s ages ranging from 0 to 200. Because age can’t be negative, you may choose a type code corresponding to an unsigned byte, which can hold values from 0 to 255, giving you ample space.

By the way, if anyone you know lives to be older than two hundred years, then stop worrying about data types and ask them about their secret!

Once you determine which numeric type best suits your needs, you can finally create an array.

Create an Empty Array to Populate Later

The array’sconstructor expects only one required argument, which must specify the desired type of elements. When you omit the optional second argument, then you’ll define anempty array of numbers. For example, the following code creates an empty array of unsigned shorts, which are 2-byte integers without a sign:

>>>fromarrayimportarray>>>empty=array("H")>>>len(empty)0

Unlike the classic array data structure with a fixed length, Python’sarray data type isdynamically sized. Therefore, you can add elements to an empty array later, and it’ll automatically resize itself accordingly.

Before learning how to populate an existing array with numbers, you’ll explore other ways of creating array instances in Python.

Initialize a New Array Using Another Iterable

Instead of creating an empty array, you can alternatively populate your new array with elements from aninitializer value passed as the second argument to the constructor. The initializer must be either abytes instance or aniterable, provided that it consists of one of the following types:

• Integers
• Floats
• Characters

Unlike some other programming languages, Python has no dedicated type for representing individual characters. An iterable of characters means a Python string or any other collection comprising strings of length one. Also, note that Python arrays don’t support complex numbers, even though they’re another numeric type in the language.

Below are a few examples demonstrating how you can use the Python array to interpret the samebyte sequence in different ways. This lets you control how to read binary data from a file, for example. To test this out, generate sample bytes by encoding thesnake emoji with UTF-8:

>>>snake_emoji="\N{snake}".encode("utf-8")>>>snake_emojib'\xf0\x9f\x90\x8d'>>>len(snake_emoji)4>>>forbyteinsnake_emoji:...print(format(byte,"b"))...11110000100111111001000010001101

The resulting byte sequence consists of four bytes, whose bits are depicted in the code block above. You can pass this byte sequence to a new array as the initializer value. Depending on the type code of the array, you’ll get wildly different results:

>>>array("b",snake_emoji)array('b', [-16, -97, -112, -115])>>>array("B",snake_emoji)array('B', [240, 159, 144, 141])>>>array("h",snake_emoji)array('h', [-24592, -29296])>>>array("H",snake_emoji)array('H', [40944, 36240])>>>array("i",snake_emoji)array('i', [-1919901712])>>>array("I",snake_emoji)array('I', [2375065584])>>>array("f",snake_emoji)array('f', [-8.913188736296437e-31])

When you declare an array of bytes using either theb orB type code, you get a new array of exactly four numbers corresponding to the individual byte values of your initializer. With wider types, such as the 2-byte short or the 4-byte integer and float, you get fewer elements in the array, but their magnitudes become larger.

Additionally, the arrays of signed types denoted with lowercase letter codes make Python interpret the numbers as negative. In this case, all four bytes of your initializer parameter have theirmost significant bit set to one, which becomes thesign bit.

If you already have another iterable container of integers, floats, or characters, then you can pass it to the array’s constructor to initialize the array with those values:

>>>array("i",[2**iforiinrange(8)])array('i', [1, 2, 4, 8, 16, 32, 64, 128])>>>array("d",(1.4,2.8,4,5.6))array('d', [1.4, 2.8, 4.0, 5.6])>>>array("u","café")array('u', 'café')>>>array("u",["c","a","f","é"])array('u', 'café')

The first array contains eight integers, which are powers of two that you calculate in alist comprehension. The second array has four double-precision floating-point numbers derived from atuple of numbers. The last two arrays represent the same characters obtained from a string and a list of individual single-character strings, respectively.

Internally, each Python array is a linear collection of numbers. The textual representation of a non-empty array depicts those numbers, except for the sequence ofUnicode characters, which are rendered as a Python string instead. Nevertheless, characters are encoded as numericalcode points under the hood, even though you don’t immediately see them.

You can initialize Python arrays in various ways depending on your needs. Moreover, you can clone an existing array, allowing you to work with a copy of the data without affecting your original array.

Use an Existing Array as a Prototype

As a special case, you can provide another Python array as the initializer value to copy its elements. After all, an array is just an iterable of numbers that you can pass to the constructor:

>>>original=array("i",[1,2,3])>>>cloned=array("i",original)>>>cloned==originalTrue>>>clonedisoriginalFalse

First, you create an array of three integers, and then you feed it to a new array. The cloned array consists of the same values as the original prototype but exists as a separate entity in the memory. Both arrays compare as equal while being distinct objects, so you can modify them independently of each other.

Note that both arrays contain elements of the same type. As long as the target array declares a superset type of the input array, or more generally, if your numbers can comfortably fit on the available range of values, then you can initialize the array:

>>>array("B",array("Q",range(256)))array('B', [0, 1, 2, (...), 253, 254, 255])

Here, you initialize an array of unsigned bytes with another array whose elements use at least eight times more memory per number! However, because the actual values are small enough to fit on the target data type, there are no issues.

Something interesting happens when you feed an array of integers into a new array of floating-point numbers:

>>>array("f",array("i",[1,2,3]))array('f', [1.0, 2.0, 3.0])

The single- and double-precision floats are supersets of the input type, so Python automaticallypromotes your numbers to the broader type. As a side effect, you can freely mix integers and floats when initializing an array of floating-point numbers, as Python will eventually convert them to an appropriate representation:

>>>array("d",(1,2.8,4,5.6))array('d', [1.0, 2.8, 4.0, 5.6])

Your input tuple consists of Python integers and floats, all of which eventually get converted to the double-precision floating-point numbers.

However, this type conversion won’t work the other way around because it would result in a potential information loss about the fractional part of the floating-point numbers:

>>>array("i",array("f",[3.14,2.72]))Traceback (most recent call last):...TypeError:'float' object cannot be interpreted as an integer

Trying to initialize an array of integers with floating-point numbers fails. There are many more things that can go wrong when you misuse the Python array, so you’ll next learn how to cope with them.

Avoid Common Pitfalls in Creating Arrays

Remember that all elements that you stuff into an array, either through your initializer parameter or the interface that you’ll soon learn, must be compatible with the array’s declared type code. Otherwise, you’ll expose yourself to a host of problems.

First of all, when you supply an initializer containing a value of an unsupported data type, such as an iterable of complex numbers or Python strings instead of characters, then you’ll get aTypeError:

>>>array("d",[3+2j,3-2j])Traceback (most recent call last):...TypeError:must be real number, not complex>>>array("u",["café","tea"])Traceback (most recent call last):...TypeError:array item must be unicode character

In the first case, you declared an array of double-precision floating-point numbers but passed a list comprising complex numbers, which aren’t among the numeric types that arrays support. Then, you tried to initialize an array of Unicode characters using a list of multicharacter Python strings.

By the way, you can only provide an iterable of characters, such as a whole string or a list of its individual characters, when you use the"u" type code. For all other type codes, Python will raise aTypeError again:

>>>array("i","café")Traceback (most recent call last):...TypeError:cannot use a str to initialize an array with typecode 'i'>>>array("i",["c","a","f","é"])Traceback (most recent call last):...TypeError:'str' object cannot be interpreted as an integer

Arguably, the most common error is theinteger overflow, which occurs when at least one of the numbers falls outside the supported range of values:

>>>array("B",[-273,0,100])Traceback (most recent call last):...OverflowError:unsigned byte integer is less than minimum>>>array("b",[55,89,144])Traceback (most recent call last):...OverflowError:signed char is greater than maximum

The first call to the array’s constructor causes anunderflow because unsigned bytes can’t be negative. Conversely, the second call results in anoverflow, as the maximum value of a signed byte is 127.

A related problem might arise when you stumble on a number that exceeds the maximum value representable on the array’s data type:

>>>array("i",[10**309])Traceback (most recent call last):...OverflowError:Python int too large to convert to C long>>>array("Q",[10**309])Traceback (most recent call last):...OverflowError:int too big to convert>>>array("d",[10**309])Traceback (most recent call last):...OverflowError:int too large to convert to float>>>array("d",[1e309])array('d', [inf])

In the first case, Python’s integer 10³⁰⁹ exceeds the maximum value representable as a C long data type. In the second case, no corresponding C data type exists to accommodate such a huge number. Next, you try to initialize an array of floating-point numbers with a Pythonlist containing your big integer.

Notice the difference between calculating an integer value with the exponentiation operator (**) and specifying the corresponding floating-point literal,1e309, to represent the same value. Because floating-point numbers are bounded, Python automatically rounds the literal to aspecial value representing infinity. On the other hand, it raises theOverflowError during an attempt to convert an extremely largeint tofloat.

At this point, you know that you can mix integers and floating-point numbers when initializing a Python array of an appropriate type. However, watch out for theloss of precision that may occur when you define an array of single-precision floats:

>>>array("f",(1.4,2.8,4,5.6))array('f', [1.399999976158142, 2.799999952316284, 4.0, 5.599999904632568])

With fewer bits at your disposal, you can’t represent some of these values exactly. Because of thefloating-point representation error, they may get rounded to the nearest representable number.

Furthermore, when you move to multibyte numbers, then you must ensure that a byte sequence passed as the initializer value is a multiple of the given element size:

>>>array("H",b"\xff\x00\x80")Traceback (most recent call last):...ValueError:bytes length not a multiple of item size

In this case, you tried initializing an array of unsigned shorts, which are two bytes long each, with a sequence of only three bytes. Because one byte is missing from your initializer, Python raises aValueError with an appropriate message.

You’re now equipped with knowledge that will prevent you from falling into these common traps when you work with arrays in Python.

Manipulate Arrays as Mutable Sequences

Although thearray class is fairly low level and used rather infrequently in Python, it’s a full-fledgedsequence type alongside the more widespreadlist andtuple data types. As a result, you can use an array whenever asequence or aniterable is expected. For example, you may pass an array ofFibonacci numbers to a relevant function:

>>>fromarrayimportarray>>>fibonacci_numbers=array("I",[1,1,2,3,5,8,13,21,34,55])>>>len(fibonacci_numbers)10>>>sum(fibonacci_numbers)143

The built-inlen() function expects a sequence as an argument, whereassum() takes an iterable of numbers, which an array certainly qualifies as. Thanks to Python’sduck typing, the exact data type doesn’t matter as long as the object at hand behaves as expected in the given context.

Naturally, being a finite sequence makes the array iterable. Therefore, you can loop over its elements or request the array to return an iterator object for manual traversal if you prefer:

>>>fori,numberinenumerate(fibonacci_numbers):...print(f"[{i}] ={number:>2}")...[0] =  1[1] =  1[2] =  2[3] =  3[4] =  5[5] =  8[6] = 13[7] = 21[8] = 34[9] = 55>>>it=iter(fibonacci_numbers)>>>next(it)1

In the example above, you first use thefor loop along with theenumerate() function to visit each element of the Fibonacci sequence and then print the corresponding index and value. Next, you create a temporary iterator object and advance it once using the built-innext() function.

To a large extent, the interface of a Python array resembles alist, so it should make you feel right at home:

>>>fibonacci_numbers[-1]55>>>fibonacci_numbers[2:9:3]array('I', [2, 8, 34])>>>fibonacci_numbers+array("I",[89,144])array('I', [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144])>>>3*array("I",[89,144])array('I', [89, 144, 89, 144, 89, 144])>>>42infibonacci_numbersFalse

You can use the square bracket syntax forindexing andslicing an array, as well as the familiaroperators, including theconcatenation operator (+), repetition operator (*), andmembership test operators (in,not in). Additionally, you’ll find most of Python’slistmethods, such as.count() and.index(), in the array:

>>>fibonacci_numbers.count(1)2>>>fibonacci_numbers.index(13)6

At the same time, high-level methods like.copy(),.clear(), and.sort() are missing from the array, but you can simulate them using alternative techniques.

Unlike the immutable tuples and strings, arrays in Python areunhashable andmutable sequences, meaning that you can modify their contents at will:

>>>fibonacci_numbers.append(89)>>>fibonacci_numbers.extend([233,377])>>>fibonacci_numbers.insert(-2,144)>>>fibonacci_numbers.reverse()>>>fibonacci_numbers.remove(144)>>>fibonacci_numbers.pop(0)377>>>fibonacci_numbers.pop()1

These methods modify your arrayin place, which explains why most of them don’treturn any value. It means that you can change an existing array’s contents without creating a new array instance.

Thesubscription syntax lets you set or delete the value of the individual numbers at specified indices, as well as replace or remove entire arrayslices:

>>>fibonacci_numbers[0]=144>>>delfibonacci_numbers[-1]>>>fibonacci_numbers[2:9:3]=array("I",[3,13,55])>>>delfibonacci_numbers[2:9:3]

Note that you can only assign an array slice declared with anidentical type code. In addition to this, be careful when you perform theextended slice assignment, which includes the third step parameter:

>>>fibonacci_numbers[2:9:3]=array("I",[3,13,55,233])Traceback (most recent call last):...ValueError:attempt to assign array of size 4 to extended slice of size 3

As with any sequence, you must ensure that the replacement slice has the same length as the one that you’re replacing when you use the extended syntax. That’s because the extra elements would produce gaps in your sequence, which must remain contiguous. On the other hand, regular slices with a step equal to one don’t have such a limitation.

Okay, you now know that an array shares an interface with other mutable sequence types in Python. Next up, you’ll learn about the unique features that set it apart.

Convert Between Arrays and Other Types

Apart from the.itemsize and.typecode attributes that you saw earlier, a Python array provides additional methods that are specific to it. In particular, it has several methods for converting between arrays and other data types:

The methods listed in the column on the left append elements to an array without overwriting any existing values. So, you can call them multiple times on the same instance to feed more numbers into your array. In fact, when you pass a list or a string as an initializer parameter to the array’s constructor, it’ll delegate the execution to some of these methods behind the curtain.

Conversely, the methods on the right allow you to represent the array’s content with the corresponding type. This may be useful when you want to process your data in pure Python or export it to a file.

The.fromfile() and.tofile() methods are the only ones that take a parameter, which must be afile object open in binary mode for reading or writing, respectively. The.fromfile() method also takes a mandatory second parameter with the number of elements to read. So, you must either know up front how many elements the file stores or retain this information separately:

>>>fromarrayimportarray>>>fromstructimportpack,unpack>>>defsave(filename,numbers):...withopen(filename,mode="wb")asfile:...file.write(numbers.typecode.encode("ascii"))...file.write(pack("<I",len(numbers)))...numbers.tofile(file)...>>>defload(filename):...withopen(filename,mode="rb")asfile:...typecode=file.read(1).decode("ascii")...(length,)=unpack("<I",file.read(4))...numbers=array(typecode)...numbers.fromfile(file,length)...returnnumbers...>>>save("binary.data",array("H",[12,42,7,15,42,38,21]))>>>load("binary.data")array('H', [12, 42, 7, 15, 42, 38, 21])

In this code snippet, your customsave() function usesstruct.pack() to encode the number of array elements in the file on a 4-byte unsigned integer (I). Note thatstruct usesformat characters similar to the array’s type codes. Additionally, the function stores your array’s type code as anASCII byte before dumping the actual array elements with.tofile().

Theload() function reverses these steps by first reading the type code and the number of elements in the file. It then creates an empty array of the correct numeric type, calls.fromfile() to populate it with the corresponding numbers from the file, and returns the filled array.

When you share binary files between operating systems or computer architectures, you must consider the potential differences in theirendianness or byte ordering. There are two major types of endianness:

1. Big-Endian: The most significant byte comes first.
2. Little-Endian: The least significant byte comes first.

If you don’t take extra steps to handle these differences, then you’ll face issues when transferring binary data between two systems that use different byte order. More specifically, you’ll misinterpret and corrupt your data by reading the individual bytes in the wrong order. So, when one system writes a number like 42, it may appear as 704,643,072 on the other end:

>>>unpack("<I",pack("<I",42))# Consistent byte order(42,)>>>unpack(">I",pack("<I",42))# Inconsistent byte order(704643072,)

While thestruct module gives you control over thebyte order and alignment through the angle bracket syntax (< and>), Python arrays remain agnostic about the interpretation of their underlying bytes. However, they do let you swap the byte order within each array element if you know that it requires adjustment:

>>>data=b"\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02">>>data_points=array("L",data)>>>data_pointsarray('L', [72057594037927936, 144115188075855872])>>>data_points.byteswap()>>>data_pointsarray('L', [1, 2])

If you read raw data from a file into a Python array knowing that the file format uses a different byte order than your system, then you can bring it to the correct endianness by reversing the bytes. Note that.byteswap() has no effect on arrays of single bytes (b andB) because the order only applies to multibyte numbers, such as short or long integers.

You’ve already delved deep into the complexities of binary data handling in Python with the help of arrays. Still, you can go even deeper by engaging a custom C library for processing your array elements, which you’ll explore next.

Treat Arrays as Bytes-Like Objects

Last but not least, the Python array supports thebuffer protocol, making it abytes-like object that you can use in contexts requiring direct access to the low-level memory region in CPython. This lets you integrate withC and C++ libraries seamlessly while avoiding unnecessary data copying.

To get the details about the underlying buffer, call your array’s.buffer_info() method:

>>>fromarrayimportarray>>>data_points=array("L",[72057594037927936,144115188075855872])>>>data_points.buffer_info()(140039807835648, 2)>>>id(data_points)140039810496960

It returns a tuple comprising the memory address of the buffer and the total number of elements that it currently holds. Notice that the memory address of the buffer is slightly different than the address of the Python array object that wraps it.

You can calculate the total number of bytes allocated for the array’s buffer by using the following code:

>>>data_points.itemsize*data_points.buffer_info()[1]16

Because there are two elements in thedata_points array and each element is eight bytes long, the total size amounts to sixteen bytes. You may need this information when accessing the buffer from aC extension module.

Writing a Python extension module in C or another compiled programming language is tedious. Fortunately, there’s a more straightforward way, which involves the standard-libraryctypes module. It allows for calling native functions from ashared library loaded dynamically into your Python script. Because arrays in Python implement the buffer protocol, you can obtain a pointer to the corresponding memory region.

Remember that you can download the supporting materials by clicking the link below, which may help if you get stuck or don’t want to type the code by hand:

First, define a bare-bones function in C that’ll take an array of numbers and modify its elements in place by incrementing them:

voidincrement(int*numbers,unsignedintlength){for(inti=0;i<length;i++){numbers[i]++;}}

The function expects a pointer to an array of signed integers as well as the length of the array. The second argument is necessary because C doesn’t keep track of the number of elements in arrays. Then, the function loops over array indices and increments each element by one.

If you’re on a Linux distribution and have theGNU C compiler installed, then you can issue the following command to turn this code into a shared library:

$gcc-shared-fPIC-O3-ocruncher.socruncher.c

The-fPIC flag makes the compiler generateposition-independent code, which is required for loading a dynamic-link library into another process, whereas-O3 enables optimizations.

You can now return to Python, load the compiled library, and call your custom C function against a non-empty Python array:

>>>fromarrayimportarray>>>fromctypesimportPOINTER,c_int,c_uint,cdll>>>cruncher=cdll.LoadLibrary("./cruncher.so")>>>cruncher.increment.argtypes=(POINTER(c_int),c_uint)>>>cruncher.increment.restype=None>>>python_array=array("i",[-21,13,-8,5,-3,2,-1,1,0,1])>>>c_array=(c_int*len(python_array)).from_buffer(python_array)>>>cruncher.increment(c_array,len(c_array))>>>python_arrayarray('i', [-20, 14, -7, 6, -2, 3, 0, 2, 1, 2])

You start by specifying the native function’s signature, which consists of the argument types and the return value type. In this case, your function takes a pointer to signed integers and an unsigned integer without returning anything. It’s vital that you define this signature, or else Python won’t be able to call the C function correctly.

Next, you create a Python array filled with a few positive and negative Fibonacci numbers using the"i" type code, which corresponds to the signed integer type in C. You then wrap your Python array in a lightweight adapter, which you later pass to the nativeincrement() function.

What’s important, calling.from_buffer() on the array adapter doesn’t allocate new memory:

>>>id(memoryview(python_array))140664913614336>>>id(memoryview(c_array))140664913614336

As you can see, thememory views of both the Python and C arrays have identical addresses, which proves that they point to the same buffer under the surface. In other words, your C function directly operates on the Python array without needing to copy the data, significantly reducing memory use and improving performance.

If this wasn’t enough evidence that the C function indeed modifies your Python array, notice how its elements change after you callincrement(). They all increase by one, which is exactly what the compiled function is supposed to do.

Combining Python with C through the buffer protocol can be incredibly efficient, especially for large numeric arrays, offering considerable performance gains over pure Python code. It’s one of the secrets behind NumPy arrays being as fast and efficient as they are.

Now, it’s time to determine precisely how much Python’s arrays outperform conventional lists.

Understand Common Numeric Types

High-level programming languages provide convenient abstractions over the binary representations of numbers to let you focus on solving bigger problems. For example, Python hasint,float, andcomplex, whileJavaScript has only oneNumber type to rule them. Slightly lower-level languages like C and C++ come with a broader spectrum of numeric types to offer greater control over data storage, which is required for working with binary data.

TheIEEE 754 specification governs the representation and behavior ofreal numbers in most modern computers, regardless of the programming language. Whether you use a Pythonfloat, a JavaScriptNumber, or a Cdouble, they should work the same. Note that in C and C++, among many other languages, you can choose between single-precision or double-precision floating-point numbers. With higher-level languages, you often only have the latter.

When storinginteger numbers in memory, computers typically use thetwo’s complement representation, which can handle both positive and negative values. In some languages, you can decide whether to use signed or unsigned integers. With unsigned numbers, you can effectively double the representable range of values. Python’s integers are always signed but use clever internal representations to give them arbitrary precision at a small cost.

As a Python developer, you might’ve never thought of the intricacies of numeric types and their representations. However, once you read binary data from an external source, you must know how to correctly interpret the corresponding byte sequences. At the very least, you should know the answers to the following questions:

• How many consecutive bytes make up a number?
• Is it a signed or an unsigned number?
• What byte order does it follow?
• Is therepadding involved?

Due to the architectural design of modern computers, which are optimized for working withoctets, as well as for historical reasons, standard numeric types have bit lengths that arepowers of two. For example, common bit lengths include eight, sixteen, thirty-two, and sixty-four bits. Programming languages provide built-in data types that correspond to these bit lengths, such aschar,short,int, andlong in C.

Thearray module in Python exposes these numeric types so that you can use them in your arrays. Unfortunately, there’ll be times when the binary data that you need to work with doesn’t exactly match any of these standard numeric types. Therefore, you’ll need to find a work-around to process your data properly.

Interpret Signed Integers of Any Size

While 8-bit bytes and their multiples defined by powers of two are the standard units of digital information, not all data types follow this pattern. Legacy systems or specialized applications may employ nonstandard numeric types.

For instance, manymusic streaming services offer content in a format based on 24-bit signed integers per sample to strike a balance between good audio quality and the amount of data to transfer. By using a nonstandardaudio bit depth, they achieve a noticeably higher quality than the standard16-bit CDs but still maintain a reasonably small size.

Unfortunately, this creates a problem for you because Python arrays don’t provide a type code for 24-bit integers. You’ll need to allocate the next larger data type per audio sample in your array, which happens to be a 32-bit integer. Then, you’ll need to manually reinterpret the corresponding bytes, which you can do by callingint.from_bytes() on them :

>>>audio_sample=b"\x7f\xfc>">>>int.from_bytes(audio_sample)8387646

The audio sample in the example above consists of three bytes or twenty-four bits. After calling theint.from_bytes()class method, you convert raw bytes expressed as aPythonbytes() object into an integer. This numeric value represents the relative amplitude of the sound wave that enters your speakers.

Check out the following code snippet, which combines this technique with a Python array:

>>>importwave>>>withwave.open("PCM_24_bit_signed.wav",mode="rb")aswave_file:...ifwave_file.getsampwidth()==3:...raw_bytes=wave_file.readframes(wave_file.getnframes())...samples=array(..."i",...(...int.from_bytes(...raw_bytes[i:i+3],...byteorder="little",...signed=True,...)...foriinrange(0,len(raw_bytes),3)...),...)...

Using the Pythonwave module, you load stereo frames, each consisting of the left and right audio channels, from a 24-bitPCM-encoded WAV file. Next, you create a new array of 32-bit signed integers and initialize it using agenerator expression that iterates over the raw bytes three elements at a time. For each slice, you interpret its bytes, explicitly setting the expected byte order and sign.

To verify that you’ve loaded your audio samples correctly, you can do some math:

>>>len(raw_bytes)40600914>>>len(samples)13533638>>>wave_file.getnframes()6766819

The raw bytes represent the content of the audio file. In this case, that amounts to just over forty megabytes, or 40,600,914 bytes of data. When you divide this number by the three bytes per sample, you get a total of 13,533,638 stereo samples, or 6,766,819 samples per channel, which is also the number of frames.

Because you declared your array with the"i" type code, which stands for 32-bit signed integers, the total amount of memory that it consumes surpasses the file’s original size:

>>>samples.itemsize4>>>samples.itemsize*len(samples)54134552

Instead of roughly forty megabytes, your array clocks in at fifty-four, which is one-third more. That makes sense, as each array element uses four instead of three bytes.

Unfortunately, there’s no way to avoid this extra cost with nonstandard numeric types, even if you declare your array in the C programming language. Anyhow, you now know how to emulate these nonstandard numeric types using the available tools.