Changingos.urandom() on platforms with the getrandom() system call

This PEP proposes that in Python 3.6+,os.urandom() be updated to callthegetrandom() syscall in non-blocking mode if available and raiseBlockingIOError:systemrandomnumbergeneratorisnotready;seesecrets.token_bytes()if the kernel reports that the call would block.

This behaviour will then propagate through to the existingrandom.SystemRandom, which provides a relatively thin wrapper aroundos.urandom() that matches therandom.Random() API.

However, the newsecrets module introduced byPEP 506 will be updated tocatch the new exception and implicitly wait for the system random numbergenerator if the exception is ever encountered.

In all cases, as soon as a call to one of these security sensitive APIssucceeds, all future calls to these APIs in that process will succeedwithout blocking (once the operating system random number generator is readyafter system boot, it remains ready).

On Linux and NetBSD, this will replace the previous behaviour of returningpotentially predictable results read from/dev/urandom.

On FreeBSD, Solaris, and Illumos, this will replace the previous behaviour ofimplicitly blocking until the system random number generator is ready. However,it is not clear if these operating systems actually allow userspace code (andhence Python) to run before the system random number generator is ready.

Note that in all cases, if calling the underlyinggetrandom() API reportsENOSYS rather than returning a successful response or reportingEAGAIN,CPython will continue to fall back to reading from/dev/urandom directly.

Addingsecrets.wait_for_system_rng()

A new exception shouldn’t be added without a straightforward recommendationfor how to resolve that error when encountered (however rare encounteringthe new error is expected to be in practice). For security sensitive code thatactually does need to use the lower level interfaces to the system randomnumber generator (rather than the newsecrets module), and does receivelive bug reports indicating this is a real problem for the userbase of thatparticular application rather than a theoretical one, this PEP’s recommendationwill be to add the following snippet (directly or indirectly) to the__main__ module:

importsecretssecrets.wait_for_system_rng()

Or, if compatibility with versions prior to Python 3.6 is needed:

try:importsecretsexceptImportError:passelse:secrets.wait_for_system_rng()

Within thesecrets module itself, this will then be used intoken_bytes() to block implicitly if the new exception is encountered:

deftoken_bytes(nbytes=None):ifnbytesisNone:nbytes=DEFAULT_ENTROPYtry:result=os.urandom(nbytes)exceptBlockingIOError:wait_for_system_rng()result=os.urandom(nbytes)returnresult

Other parts of the module will then be updated to usetoken_bytes() astheir basic random number generation building block, rather than callingos.urandom() directly.

Application frameworks covering use cases where access to the system randomnumber generator is almost certain to be needed (e.g. web frameworks) maychoose to incorporate a call tosecrets.wait_for_system_rng() implicitlyinto the commands that start the application such that existing calls toos.urandom() will be guaranteed to never raise the new exception when usingthose frameworks.

For cases where the error is encountered for an application which cannot bemodified directly, then the following command can be used to wait for thesystem random number generator to initialize before starting that application:

python3-c"import secrets; secrets.wait_for_system_rng()"

For example, this snippet could be added to a shell script or a systemdExecStartPre hook (and may prove useful in reliably waiting for thesystem random number generator to be ready, even if the subsequent commandis not itself an application running under Python 3.6)

Given the changes proposed toos.urandom() above, and the inclusion ofanos.getrandom() API on systems that support it, the suggestedimplementation of this function would be:

ifhasattr(os,"getrandom"):# os.getrandom() always blocks waiting for the system RNG by defaultdefwait_for_system_rng():"""Block waiting for system random number generator to be ready"""os.getrandom(1)returnelse:# As far as we know, other platforms will never get BlockingIOError# below but the implementation makes pessimistic assumptionsdefwait_for_system_rng():"""Block waiting for system random number generator to be ready"""# If the system RNG is already seeded, don't wait at alltry:os.urandom(1)returnexceptBlockingIOError:pass# Avoid the below busy loop if possibletry:block_on_system_rng=open("/dev/random","rb")exceptFileNotFoundError:passelse:withblock_on_system_rng:block_on_system_rng.read(1)# Busy loop until the system RNG is readywhileTrue:try:os.urandom(1)breakexceptBlockingIOError:# Only check once per millisecondtime.sleep(0.001)

On systems where it is possible to wait for the system RNG to be ready, thisfunction will do so without a busy loop ifos.getrandom() is defined,os.urandom() itself implicitly blocks, or the/dev/random device isavailable. If the system random number generator is ready, this call isguaranteed to never block, even if the system’s/dev/random device usesa design that permits it to block intermittently during normal system operation.

Limitations on scope

No changes are proposed for Windows or Mac OS X systems, as neither of thoseplatforms provides any mechanism to run Python code before the operatingsystem random number generator has been initialized. Mac OS X goes so far asto kernel panic and abort the boot process if it can’t properly initialize therandom number generator (although Apple’s restrictions on the supportedhardware platforms make that exceedingly unlikely in practice).

Similarly, no changes are proposed for other *nix systems that do not offerthegetrandom() syscall. On these systems,os.urandom() will continueto block waiting for the system random number generator to be initialized.

While other *nix systems that offer a non-blocking API (other thangetrandom()) for requesting random numbers suitable for use in securitysensitive applications could potentially receive a similar update to the oneproposed forgetrandom() in this PEP, such changes are out of scope forthis particular proposal.

Python’s behaviour on older versions of affected platforms that do not offerthe newgetrandom() syscall will also remain unchanged.

Ensuring thesecrets module implicitly blocks when needed

This is done to help encourage the meme that arises for folks that want thesimplest possible answer to the right way to generate security sensitive randomnumbers to be “Use the secrets module when available or your application mightcrash unexpectedly”, rather than the more boilerplate heavy “Always callsecrets.wait_for_system_rng() when available or your application might crashunexpectedly”.

It’s also done due to the BDFL having a higher tolerance for APIs that mightblock unexpectedly than he does for APIs that might throw an unexpectedexception[11].

RaisingBlockingIOError inos.urandom() on Linux

For several years now, the security community’s guidance has been to useos.urandom() (or therandom.SystemRandom() wrapper) when implementingsecurity sensitive operations in Python.

To help improve API discoverability and make it clearer that secrecy andsimulation are not the same problem (even though they both involverandom numbers),PEP 506 collected several of the one line recipes basedon the lower levelos.urandom() API into a newsecrets module.

However, this guidance has also come with a longstanding caveat: developerswriting security sensitive software at least for Linux, and potentially forsome other *BSD systems, may need to wait until the operating system’srandom number generator is ready before relying on it for security sensitiveoperations. This generally only occurs ifos.urandom() is read veryearly in the system initialization process, or on systems with few sources ofavailable entropy (e.g. some kinds of virtualized or embedded systems), butunfortunately the exact conditions that trigger this are difficult to predict,and when it occurs then there is no direct way for userspace to tell it hashappened without querying operating system specific interfaces.

On *BSD systems (if the particular *BSD variant allows the problem to occurat all) and potentially also Solaris and Illumos, encountering this situationmeansos.urandom() will either block waiting for the system random numbergenerator to be ready (the associated symptom would be for the affected scriptto pause unexpectedly on the first call toos.urandom()) or else willbehave the same way as it does on Linux.

On Linux, in Python versions up to and including Python 3.4, and inPython 3.5 maintenance versions following Python 3.5.2, there’s no clearindicator to developers that their software may not be working as expectedwhen run early in the Linux boot process, or on hardware without goodsources of entropy to seed the operating system’s random number generator: dueto the behaviour of the underlying/dev/urandom device,os.urandom()on Linux returns a result either way, and it takes extensive statisticalanalysis to show that a security vulnerability exists.

By contrast, ifBlockingIOError is raised in those situations, thendevelopers using Python 3.6+ can easily choose their desired behaviour:

1. Wait for the system RNG at or before application startup (security sensitive)
2. Switch to using the random module (non-security sensitive)

Makingsecrets.wait_for_system_rng() public

Earlier versions of this PEP proposed a number of recipes for wrappingos.urandom() to make it suitable for use in security sensitive use cases.

Discussion of the proposal on the security-sig mailing list prompted therealization[9] that the core assumption driving the API design in this PEPwas that choosing between letting the exception cause the application to fail,blocking waiting for the system RNG to be ready and switching to using therandom module instead ofos.urandom is an application and use-casespecific decision that should take into account application and use-casespecific details.

There is no way for the interpreter runtime or support libraries to determinewhether a particular use case is security sensitive or not, and while it’sstraightforward for application developer to decide how to handle an exceptionthrown by a particular API, they can’t readily workaround an API blocking whenthey expected it to be non-blocking.

Accordingly, the PEP was updated to addsecrets.wait_for_system_rng() asan API for applications, scripts and frameworks to use to indicate that theywanted to ensure the system RNG was available before continuing, while librarydevelopers could continue to callos.urandom() without worrying that itmight unexpectedly start blocking waiting for the system RNG to be available.

Unaffected Applications

The following kinds of applications would be entirely unaffected by the change,regardless of whether or not they perform security sensitive operations:

• applications that don’t support Linux
• applications that are only run on desktops or conventional servers
• applications that are only run after the system RNG is ready (includingthose where an application framework callssecrets.wait_for_system_rng()on their behalf)

Applications in this category simply won’t encounter the new exception, so itwill be reasonable for developers to wait and see if they receivePython 3.6 compatibility bugs related to the new runtime behaviour, rather thanattempting to pre-emptively determine whether or not they’re affected.

Affected security sensitive applications

Security sensitive applications would need to either change their systemconfiguration so the application is only started after the operating systemrandom number generator is ready for security sensitive operations, change theapplication startup code to invokesecrets.wait_for_system_rng(), orelse switch to using the newsecrets.token_bytes() API.

As an example for components started via a systemd unit file, the followingsnippet would delay activation until the system RNG was ready:

ExecStartPre=python3 -c “import secrets; secrets.wait_for_system_rng()”

Alternatively, the following snippet will usesecrets.token_bytes() ifavailable, and fall back toos.urandom() otherwise:

try:

import secrets.token_bytes as _get_random_bytes

except ImportError:

import os.urandom as _get_random_bytes

Affected non-security sensitive applications

Non-security sensitive applications should be updated to use therandommodule rather thanos.urandom:

defpseudorandom_bytes(num_bytes):returnrandom.getrandbits(num_bytes*8).to_bytes(num_bytes,"little")

Depending on the details of the application, the random module may offerother APIs that can be used directly, rather than needing to emulate theraw byte sequence produced by theos.urandom() API.

Why propose this now?

The main reason is because the Python 3.5.0 release switched to using the newLinuxgetrandom() syscall when available in order to avoid consuming afile descriptor[1], and this had the side effect of making the followingoperations block waiting for the system random number generator to be ready:

• os.urandom (and APIs that depend on it)
• importing therandom module
• initializing the randomized hash algorithm used by some builtin types

While the first of those behaviours is arguably desirable (and consistent withthe existing behaviour ofos.urandom on other operating systems), thelatter two behaviours are unnecessary and undesirable, and the last one is nowknown to cause a system level deadlock when attempting to run Python scriptsduring the Linux init process with Python 3.5.0 or 3.5.1[2], while the secondone can cause problems when using virtual machines without robust entropysources configured[3].

Since decoupling these behaviours in CPython will involve a number ofimplementation changes more appropriate for a feature release than a maintenancerelease, the relatively simple resolution applied in Python 3.5.2 was to revertall three of them to a behaviour similar to that of previous Python versions:if the new Linux syscall indicates it will block, then Python 3.5.2 willimplicitly fall back on reading/dev/urandom directly[4].

However, this bug reportalso resulted in a range of proposals to addnewAPIs likeos.getrandom()[5],os.urandom_block()[6],os.pseudorandom() andos.cryptorandom()[7], or adding new optionalparameters toos.urandom() itself[8], and then attempting to educateusers on when they should call those APIs instead of just using a plainos.urandom() call.

These proposals arguably represent overreactions, as the question of reliablyobtaining random numbers suitable for security sensitive work on Linux is arelatively obscure problem of interest mainly to operating system developersand embedded systems programmers, that may not justify expanding thePython standard library’s cross-platform APIs with new Linux-specific concerns.This is especially so with thesecrets module already being added as the“use this and don’t worry about the low level details” option for developerswriting security sensitive software that for some reason can’t rely on evenhigher level domain specific APIs (like web frameworks) and also don’t need toworry about Python versions prior to Python 3.6.

That said, it’s also the case that low cost ARM devices are becomingincreasingly prevalent, with a lot of them running Linux, and a lot of folkswriting Python applications that run on those devices. That creates anopportunity to take an obscure security problem that currently requires a lotof knowledge about Linux boot processes and provably unpredictable randomnumber generation to diagnose and resolve, and instead turn it into arelatively mundane and easy-to-find-in-an-internet-search runtime exception.

The cross-platform behaviour ofos.urandom()

On operating systems other than Linux and NetBSD,os.urandom() may alreadyblock waiting for the operating system’s random number generator to be ready.This will happen at most once in the lifetime of the process, and the call issubsequently guaranteed to be non-blocking.

Linux and NetBSD are outliers in that, even when the operating system’s randomnumber generator doesn’t consider itself ready for use in security sensitiveoperations, reading from the/dev/urandom device will return random valuesbased on the entropy it has available.

This behaviour is potentially problematic, so Linux 3.17 added a newgetrandom() syscall that (amongst other benefits) allows callers toeither block waiting for the random number generator to be ready, orelse request an error return if the random number generator is not ready.Notably, the new API doesnot support the old behaviour of returningdata that is not suitable for security sensitive use cases.

Versions of Python prior up to and including Python 3.4 access theLinux/dev/urandom device directly.

Python 3.5.0 and 3.5.1 (when build on a system that offered the new syscall)calledgetrandom() in blocking mode in order to avoid the use of a filedescriptor to access/dev/urandom. While there were no specific problemsreported due toos.urandom() blocking in user code, therewere problemsdue to CPython implicitly invoking the blocking behaviour during interpreterstartup and when importing therandom module.

Rather than trying to decouple SipHash initialization from theos.urandom() implementation, Python 3.5.2 switched to callinggetrandom() in non-blocking mode, and falling back to reading from/dev/urandom if the syscall indicates it will block.

As a result of the above,os.urandom() in all Python versions up to andincluding Python 3.5 propagate the behaviour of the underling/dev/urandomdevice to Python code.

Problems with the behaviour of/dev/urandom on Linux

The Pythonos module has largely co-evolved with Linux APIs, so havingos module functions closely follow the behaviour of their Linux operatingsystem level counterparts when running on Linux is typically considered to bea desirable feature.

However,/dev/urandom represents a case where the current behaviour isacknowledged to be problematic, but fixing it unilaterally at the kernel levelhas been shown to prevent some Linux distributions from booting (at least inpart due to components like Python currently using it fornon-security-sensitive purposes early in the system initialization process).

As an analogy, consider the following two functions:

defgenerate_example_password():"""Generates passwords solely for use in code examples"""returngenerate_unpredictable_password()defgenerate_actual_password():"""Generates actual passwords for use in real applications"""returngenerate_unpredictable_password()

If you think of an operating system’s random number generator as a method forgenerating unpredictable, secret passwords, then you can think of Linux’s/dev/urandom as being implemented like:

# Oversimplified artist's conception of the kernel code# implementing /dev/urandomdefgenerate_unpredictable_password():ifsystem_rng_is_ready:returnuse_system_rng_to_generate_password()else:# we can't make an unpredictable password; silently return a# potentially predictable one instead:return"p4ssw0rd"

In this scenario, the author ofgenerate_example_password is fine - even if"p4ssw0rd" shows up a bit more often than they expect, it’s only used inexamples anyway. However, the author ofgenerate_actual_password has aproblem - how do they prove that their calls togenerate_unpredictable_password never follow the path that returns apredictable answer?

In real life it’s slightly more complicated than this, because theremight be some level of system entropy available – so the fallback mightbe more likereturnrandom.choice(["p4ssword","passw0rd","p4ssw0rd"]) or something even more variable and hence only statisticallypredictable with better odds than the author ofgenerate_actual_passwordwas expecting. This doesn’t really make things more provably secure, though;mostly it just means that if you try to catch the problem in the obvious way –ifreturned_password=="p4ssw0rd":raiseUhOh – then it doesn’t work,becausereturned_password might instead bep4ssword or evenpa55word, or just an arbitrary 64 bit sequence selected from fewer than2**64 possibilities. So this rough sketch does give the right general idea ofthe consequences of the “more predictable than expected” fallback behaviour,even though it’s thoroughly unfair to the Linux kernel team’s efforts tomitigate the practical consequences of this problem without resorting tobreaking backwards compatibility.

This design is generally agreed to be a bad idea. As far as we cantell, there are no use cases whatsoever in which this is the behavioryou actually want. It has led to the use of insecuressh keys onreal systems, and many *nix-like systems (including at least Mac OSX, OpenBSD, and FreeBSD) have modified their/dev/urandomimplementations so that they never return predictable outputs, eitherby making reads block in this case, or by simply refusing to run anyuserspace programs until the system RNG has beeninitialized. Unfortunately, Linux has so far been unable to followsuit, because it’s been empirically determined that enabling theblocking behavior causes some currently extant distributions tofail to boot.

Instead, the newgetrandom() syscall was introduced, makingitpossible for userspace applications to access the system random numbergenerator safely, without introducing hard to debug deadlock problems intothe system initialization processes of existing Linux distros.

Consequences ofgetrandom() availability for Python

Prior to the introduction of thegetrandom() syscall, it simply wasn’tfeasible to access the Linux system random number generator in a provablysafe way, so we were forced to settle for reading from/dev/urandom as thebest available option. However, withgetrandom() insisting on raising anerror or blocking rather than returning predictable data, as well as havingother advantages, it is now the recommended method for accessing the kernelRNG on Linux, with reading/dev/urandom directly relegated to “legacy”status. This moves Linux into the same category as other operating systemslike Windows, which doesn’t provide a/dev/urandom device at all: thebest available option for implementingos.urandom() is no longer simplyreading bytes from the/dev/urandom device.

This means that what used to be somebody else’s problem (the Linux kerneldevelopment team’s) is now Python’s problem – given a way to detect that thesystem RNG is not initialized, we have to choose how to handle thissituation whenever we try to use the system RNG.

It could simply block, as was somewhat inadvertently implemented in 3.5.0,and as is proposed in Victor Stinner’s competing PEP:

# artist's impression of the CPython 3.5.0-3.5.1 behaviordefgenerate_unpredictable_bytes_or_block(num_bytes):whilenotsystem_rng_is_ready:waitreturnunpredictable_bytes(num_bytes)

Or it could raise an error, as this PEP proposes (insome cases):

# artist's impression of the behavior proposed in this PEPdefgenerate_unpredictable_bytes_or_raise(num_bytes):ifsystem_rng_is_ready:returnunpredictable_bytes(num_bytes)else:raiseBlockingIOError

Or it could explicitly emulate the/dev/urandom fallback behavior,as was implemented in 3.5.2rc1 and is expected to remain for the restof the 3.5.x cycle:

# artist's impression of the CPython 3.5.2rc1+ behaviordefgenerate_unpredictable_bytes_or_maybe_not(num_bytes):ifsystem_rng_is_ready:returnunpredictable_bytes(num_bytes)else:return(b"p4ssw0rd"*(num_bytes//8+1))[:num_bytes]

(And the same caveats apply to this sketch as applied to thegenerate_unpredictable_password sketch of/dev/urandom above.)

There are five places where CPython and the standard library attempt to use theoperating system’s random number generator, and thus five places where thisdecision has to be made:

• initializing the SipHash used to protectstr.__hash__ andfriends against DoS attacks (called unconditionally at startup)
• initializing therandom module (called whenrandom isimported)
• servicing user calls to theos.urandom public API
• the higher levelrandom.SystemRandom public API
• the newsecrets module public API added byPEP 506

Previously, these five places all used the same underlying code, andthus made this decision in the same way.

This whole problem was first noticed because 3.5.0 switched thatunderlying code to thegenerate_unpredictable_bytes_or_block behavior,and it turns out that there are some rare cases where Linux bootscripts attempted to run a Python program as part of system initialization, thePython startup sequence blocked while trying to initialize SipHash,and then this triggered a deadlock because the system stopped doinganything – including gathering new entropy – until the Python scriptwas forcibly terminated by an external timer. This is particularly unfortunatesince the scripts in question never processed untrusted input, so there was noneed for SipHash to be initialized with provably unpredictable random data inthe first place. This motivated the change in 3.5.2rc1 to emulate the old/dev/urandom behavior in all cases (by callinggetrandom() innon-blocking mode, and then falling back to reading/dev/urandomif the syscall indicates that the/dev/urandom pool is not yetfully initialized.)

We don’t know whether such problems may also exist in the Fedora/RHEL/CentOSecosystem, as the build systems for those distributions use chroots on serversrunning an older operating system kernel that doesn’t offer thegetrandom()syscall, which means CPython’s current build configuration compiles out theruntime check for that syscall[10].

A similar problem was found due to therandom module callingos.urandom as a side-effect of import in order to seed the defaultglobalrandom.Random() instance.

We have not received any specific complaints regarding direct calls toos.urandom() orrandom.SystemRandom() blocking with 3.5.0 or 3.5.1 -only problem reports due to the implicit blocking on interpreter startup andas a side-effect of importing the random module.

Independently of this PEP, the first two cases have already been updated tonever block, regardless of the behaviour ofos.urandom().

WherePEP 524 proposes to make all 3 of the latter cases block implicitly,this PEP proposes that approach only for the last case (thesecrets)module, withos.urandom() andrandom.SystemRandom() instead raisingan exception when they detect that the underlying operating system callwould block.