Therandom Module

Probably the most widely known tool for generating random data in Python is itsrandom module, which uses theMersenne Twister PRNG algorithm as its core generator.

Earlier, you touched briefly onrandom.seed(), and now is a good time to see how it works. First, let’s build some random data without seeding. Therandom.random() function returns a random float in the interval [0.0, 1.0). The result will always be less than the right-hand endpoint (1.0). This is also known as a semi-open range:

>>># Don't call `random.seed()` yet>>>importrandom>>>random.random()0.35553263284394376>>>random.random()0.6101992345575074

If you run this code yourself, I’ll bet my life savings that the numbers returned on your machine will be different. Thedefault when you don’t seed the generator is to use your current system time or a “randomness source” from your OS if one is available.

Withrandom.seed(), you can make results reproducible, and the chain of calls afterrandom.seed() will produce the same trail of data:

>>>random.seed(444)>>>random.random()0.3088946587429545>>>random.random()0.01323751590501987>>>random.seed(444)# Re-seed>>>random.random()0.3088946587429545>>>random.random()0.01323751590501987

Notice the repetition of “random” numbers. The sequence of random numbers becomes deterministic, or completely determined by the seed value, 444.

Let’s take a look at some more basic functionality ofrandom. Above, you generated a random float. You can generate a random integer between two endpoints in Python with therandom.randint() function. This spans the full [x, y] interval and may include both endpoints:

>>>random.randint(0,10)7>>>random.randint(500,50000)18601

Withrandom.randrange(), you can exclude the right-hand side of the interval, meaning the generated number always lies within [x, y) and will always be smaller than the right endpoint:

>>>random.randrange(1,10)5

If you need to generate random floats that lie within a specific [x, y] interval, you can userandom.uniform(), which plucks from thecontinuous uniform distribution:

>>>random.uniform(20,30)27.42639687016509>>>random.uniform(30,40)36.33865802745107

To pick a random element from a non-empty sequence (like alist or a tuple), you can userandom.choice(). There is alsorandom.choices() for choosing multiple elements from a sequence with replacement (duplicates are possible):

>>>items=['one','two','three','four','five']>>>random.choice(items)'four'>>>random.choices(items,k=2)['three','three']>>>random.choices(items,k=3)['three','five','four']

To mimic sampling without replacement, userandom.sample():

>>>random.sample(items,4)['one', 'five', 'four', 'three']

You can randomize a sequence in-place usingrandom.shuffle(). This will modify the sequence object and randomize the order of elements:

>>>random.shuffle(items)>>>items['four', 'three', 'two', 'one', 'five']

If you’d rather not mutate the original list, you’ll need tomake a copy first and then shuffle the copy. You can create copies of Python lists with thecopy module, or justx[:] orx.copy(), wherex is the list.

Before moving on to generating random data with NumPy, let’s look at one more slightly involved application: generating a sequence of unique random strings of uniform length.

It can help to think about the design of the function first. You need to choose from a “pool” of characters such as letters, numbers, and/or punctuation, combine these into a single string, and then check that this string has not already been generated. A Pythonset works well for this type of membership testing:

importstringdefunique_strings(k:int,ntokens:int,pool:str=string.ascii_letters)->set:"""Generate a set of unique string tokens.    k: Length of each token    ntokens: Number of tokens    pool: Iterable of characters to choose from    For a highly optimized version:    https://stackoverflow.com/a/48421303/7954504    """seen=set()# An optimization for tightly-bound loops:# Bind these methods outside of a loopjoin=''.joinadd=seen.addwhilelen(seen)<ntokens:token=join(random.choices(pool,k=k))add(token)returnseen

''.join() joins the letters fromrandom.choices() into a single Pythonstr of lengthk. This token is added to the set, which can’t contain duplicates, and thewhile loop executes until the set has the number of elements that you specify.

Let’s try this function out:

>>>unique_strings(k=4,ntokens=5){'AsMk', 'Cvmi', 'GIxv', 'HGsZ', 'eurU'}>>>unique_strings(5,4,string.printable){"'O*1!", '9Ien%', 'W=m7<', 'mUD|z'}

For a fine-tuned version of this function,this Stack Overflow answer uses generator functions, name binding, and some other advanced tricks to make a faster, cryptographically secure version ofunique_strings() above.

PRNGs for Arrays:numpy.random

One thing you might have noticed is that a majority of the functions fromrandom return a scalar value (a singleint,float, or other object). If you wanted to generate a sequence of random numbers, one way to achieve that would be with a Pythonlist comprehension:

>>>[random.random()for_inrange(5)][0.021655420657909374, 0.4031628347066195, 0.6609991871223335, 0.5854998250783767, 0.42886606317322706]

But there is another option that is specifically designed for this. You can think of NumPy’s ownnumpy.random package as being like the standard library’srandom, but forNumPy arrays. (It also comes loaded with the ability to draw from a lot more statistical distributions.)

Take note thatnumpy.random uses its own PRNG that is separate from plain oldrandom. You won’t produce deterministically random NumPy arrays with a call to Python’s ownrandom.seed():

>>>importnumpyasnp>>>np.random.seed(444)>>>np.set_printoptions(precision=2)# Output decimal fmt.

Without further ado, here are a few examples to whet your appetite:

>>># Return samples from the standard normal distribution>>>np.random.randn(5)array([ 0.36,  0.38,  1.38,  1.18, -0.94])>>>np.random.randn(3,4)array([[-1.14, -0.54, -0.55,  0.21],       [ 0.21,  1.27, -0.81, -3.3 ],       [-0.81, -0.36, -0.88,  0.15]])>>># `p` is the probability of choosing each element>>>np.random.choice([0,1],p=[0.6,0.4],size=(5,4))array([[0, 0, 1, 0],       [0, 1, 1, 1],       [1, 1, 1, 0],       [0, 0, 0, 1],       [0, 1, 0, 1]])

In the syntax forrandn(d0, d1, ..., dn), the parametersd0, d1, ..., dn are optional and indicate the shape of the final object. Here,np.random.randn(3, 4) creates a 2d array with 3 rows and 4 columns. The data will bei.i.d., meaning that each data point is drawn independent of the others.

Another common operation is to create a sequence of randomBoolean values,True orFalse. One way to do this would be withnp.random.choice([True, False]). However, it’s actually about 4x faster to choose from(0, 1) and thenview-cast these integers to their corresponding Boolean values:

>>># NumPy's `randint` is [inclusive, exclusive), unlike `random.randint()`>>>np.random.randint(0,2,size=25,dtype=np.uint8).view(bool)array([ True, False,  True,  True, False,  True, False, False, False,       False, False,  True,  True, False, False, False,  True, False,        True, False,  True,  True,  True, False,  True])

What about generatingcorrelated data? Let’s say you want to simulate two correlated time series. One way of going about this is with NumPy’smultivariate_normal() function, which takes a covariance matrix into account. In other words, to draw from a single normally distributed random variable, you need to specify its mean and variance (or standard deviation).

To sample from themultivariate normal distribution, you specify the means and covariance matrix, and you end up with multiple, correlated series of data that are each approximately normally distributed.

However, rather than covariance,correlation is a measure that is more familiar and intuitive to most. It’s the covariance normalized by the product of standard deviations, and so you can also define covariance in terms of correlation and standard deviation:

So, could you draw random samples from a multivariate normal distribution by specifying a correlation matrix and standard deviations? Yes, but you’ll need to get the aboveinto matrix form first. Here,S is a vector of the standard deviations,P is their correlation matrix, andC is the resulting (square) covariance matrix:

This can be expressed in NumPy as follows:

defcorr2cov(p:np.ndarray,s:np.ndarray)->np.ndarray:"""Covariance matrix from correlation & standard deviations"""d=np.diag(s)returnd@p@d

Now, you can generate two time series that are correlated but still random:

>>># Start with a correlation matrix and standard deviations.>>># -0.40 is the correlation between A and B, and the correlation>>># of a variable with itself is 1.0.>>>corr=np.array([[1.,-0.40],...[-0.40,1.]])>>># Standard deviations/means of A and B, respectively>>>stdev=np.array([6.,1.])>>>mean=np.array([2.,0.5])>>>cov=corr2cov(corr,stdev)>>># `size` is the length of time series for 2d data>>># (500 months, days, and so on).>>>data=np.random.multivariate_normal(mean=mean,cov=cov,size=500)>>>data[:10]array([[ 0.58,  1.87],       [-7.31,  0.74],       [-6.24,  0.33],       [-0.77,  1.19],       [ 1.71,  0.7 ],       [-3.33,  1.57],       [-1.13,  1.23],       [-6.58,  1.81],       [-0.82, -0.34],       [-2.32,  1.1 ]])>>>data.shape(500, 2)

You can think ofdata as 500 pairs of inversely correlated data points. Here’s a sanity check that you can back into the original inputs, which approximatecorr,stdev, andmean from above:

>>>np.corrcoef(data,rowvar=False)array([[ 1.  , -0.39],       [-0.39,  1.  ]])>>>data.std(axis=0)array([5.96, 1.01])>>>data.mean(axis=0)array([2.13, 0.49])

Before we move on to CSPRNGs, it might be helpful to summarize somerandom functions and theirnumpy.random counterparts:

Now that you’ve covered two fundamental options for PRNGs, let’s move onto a few more secure adaptations.

os.urandom(): About as Random as It Gets

Python’sos.urandom() function is used by bothsecrets anduuid (both of which you’ll see here in a moment). Without getting into too much detail,os.urandom() generates operating-system-dependent random bytes that can safely be called cryptographically secure:

• On Unix operating systems, it reads random bytes from the special file/dev/urandom, which in turn “allow access to environmental noise collected from device drivers and other sources.” (Thank you,Wikipedia.) This is garbled information that is particular to your hardware and system state at an instance in time but at the same time sufficiently random.

• On Windows, the C++ functionCryptGenRandom() is used. This function is still technically pseudorandom, but it works by generating a seed value from variables such as the process ID, memory status, and so on.

Withos.urandom(), there is no concept of manually seeding. While still technically pseudorandom, this function better aligns with how we think of randomness. The only argument is the number ofbytes to return:

>>>os.urandom(3)b'\xa2\xe8\x02'>>>x=os.urandom(6)>>>xb'\xce\x11\xe7"!\x84'>>>type(x),len(x)(bytes, 6)

Before we go any further, this might be a good time to delve into a mini-lesson oncharacter encoding. Many people, including myself, have some type of allergic reaction when they seebytes objects and a long line of\x characters. However, it’s useful to know how sequences such asx above eventually get turned into strings or numbers.

os.urandom() returns a sequence of single bytes:

>>>xb'\xce\x11\xe7"!\x84'

But how does this eventually get turned into a Pythonstr or sequence of numbers?

First, recall one of the fundamental concepts of computing, which is that a byte is made up of 8 bits. You can think of a bit as a single digit that is either 0 or 1. A byte effectively chooses between 0 and 1 eight times, so both01101100 and11110000 could represent bytes. Try this, which makes use of Pythonf-strings introduced in Python 3.6, in your interpreter:

>>>binary=[f'{i:0>8b}'foriinrange(256)]>>>binary[:16]['00000000', '00000001', '00000010', '00000011', '00000100', '00000101', '00000110', '00000111', '00001000', '00001001', '00001010', '00001011', '00001100', '00001101', '00001110', '00001111']

This is equivalent to[bin(i) for i in range(256)], with some special formatting.bin() converts an integer to its binary representation as a string.

Where does that leave us? Usingrange(256) above is not a random choice. (No pun intended.) Given that we are allowed 8 bits, each with 2 choices, there are2 ** 8 == 256 possible bytes “combinations.”

This means that each byte maps to an integer between 0 and 255. In other words, we would need more than 8 bits to express the integer 256. You can verify this by checking thatlen(f'{256:0>8b}') is now 9, not 8.

Okay, now let’s get back to thebytes data type that you saw above, by constructing a sequence of the bytes that correspond to integers 0 through 255:

>>>bites=bytes(range(256))

If you calllist(bites), you’ll get back to a Python list that runs from 0 to 255. But if you just printbites, you get an ugly looking sequence littered with backslashes:

>>>bitesb'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15' '\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJK' 'LMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86' '\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b' # ...

These backslashes are escape sequences, and\xhhrepresents the character with hex valuehh. Some of the elements ofbites are displayed literally (printable characters such as letters, numbers, and punctuation). Most are expressed with escapes.\x08 represents a keyboard’s backspace, while\x13 is acarriage return (part of a new line, on Windows systems).

If you need a refresher on hexadecimal, Charles Petzold’sCode: The Hidden Language is a great place for that. Hex is a base-16 numbering system that, instead of using 0 through 9, uses 0 through 9 anda throughf as its basic digits.

Finally, let’s get back to where you started, with the sequence of random bytesx. Hopefully this makes a little more sense now. Calling.hex() on abytes object gives astr of hexadecimal numbers, with each corresponding to a decimal number from 0 through 255:

>>>xb'\xce\x11\xe7"!\x84'>>>list(x)[206, 17, 231, 34, 33, 132]>>>x.hex()'ce11e7222184'>>>len(x.hex())12

One last question: how isb.hex() 12 characters long above, even thoughx is only 6 bytes? This is because two hexadecimal digits correspond precisely to a single byte. Thestr version ofbytes will always be twice as long as far as our eyes are concerned.

Even if the byte (such as\x01) does not need a full 8 bits to be represented,b.hex() will always use two hex digits per byte, so the number 1 will be represented as01 rather than just1. Mathematically, though, both of these are the same size.

With that under your belt, let’s touch on a recently introduced module,secrets, which makes generating secure tokens much more user-friendly.

Python’s Best Keptsecrets

Introduced in Python 3.6 byone of the more colorful PEPs out there, thesecrets module is intended to be the de facto Python module for generating cryptographically secure random bytes and strings.

You can check out thesource code for the module, which is short and sweet at about 25 lines of code.secrets is basically a wrapper aroundos.urandom(). It exports just a handful of functions for generating random numbers, bytes, and strings. Most of these examples should be fairly self-explanatory:

>>>n=16>>># Generate secure tokens>>>secrets.token_bytes(n)b'A\x8cz\xe1o\xf9!;\x8b\xf2\x80pJ\x8b\xd4\xd3'>>>secrets.token_hex(n)'9cb190491e01230ec4239cae643f286f'>>>secrets.token_urlsafe(n)'MJoi7CknFu3YN41m88SEgQ'>>># Secure version of `random.choice()`>>>secrets.choice('rain')'a'

Now, how about a concrete example? You’ve probably used URL shortener services liketinyurl.com orbit.ly that turn an unwieldy URL into something likehttps://bit.ly/2IcCp9u. Most shorteners don’t do any complicated hashing from input to output; they just generate a random string, make sure that string has not already been generated previously, and then tie that back to the input URL.

Let’s say that after taking a look at theRoot Zone Database, you’ve registered the siteshort.ly. Here’s a function to get you started with your service:

# shortly.pyfromsecretsimporttoken_urlsafeDATABASE={}defshorten(url:str,nbytes:int=5)->str:ext=token_urlsafe(nbytes=nbytes)ifextinDATABASE:returnshorten(url,nbytes=nbytes)else:DATABASE.update({ext:url})returnf'short.ly/{ext}

Is this a full-fledged real illustration? No. I would wager that bit.ly does things in a slightly more advanced way than storing its gold mine in a global Python dictionary that is not persistent between sessions.

However, it’s roughly accurate conceptually:

>>>urls=(...'https://realpython.com/',...'https://docs.python.org/3/howto/regex.html'...)>>>foruinurls:...print(shorten(u))short.ly/p_Z4fLIshort.ly/fuxSyNY>>>DATABASE{'p_Z4fLI': 'https://realpython.com/', 'fuxSyNY': 'https://docs.python.org/3/howto/regex.html'}

The bottom line here is that, whilesecrets is really just a wrapper around existing Python functions, it can be your go-to when security is your foremost concern.