Instrumenting CPython with DTrace and SystemTap

author:

David Malcolm

author:

Łukasz Langa

DTrace and SystemTap are monitoring tools, each providing a way to inspectwhat the processes on a computer system are doing. They both usedomain-specific languages allowing a user to write scripts which:

  • filter which processes are to be observed

  • gather data from the processes of interest

  • generate reports on the data

As of Python 3.6, CPython can be built with embedded “markers”, alsoknown as “probes”, that can be observed by a DTrace or SystemTap script,making it easier to monitor what the CPython processes on a system aredoing.

CPython implementation detail: DTrace markers are implementation details of the CPython interpreter.No guarantees are made about probe compatibility between versions ofCPython. DTrace scripts can stop working or work incorrectly withoutwarning when changing CPython versions.

Enabling the static markers

macOS comes with built-in support for DTrace. On Linux, in order tobuild CPython with the embedded markers for SystemTap, the SystemTapdevelopment tools must be installed.

On a Linux machine, this can be done via:

$yuminstallsystemtap-sdt-devel

or:

$sudoapt-getinstallsystemtap-sdt-dev

CPython must then beconfiguredwiththe--with-dtraceoption:

checking for --with-dtrace... yes

On macOS, you can list available DTrace probes by running a Pythonprocess in the background and listing all probes made available by thePython provider:

$python3.6-q&$sudodtrace-l-Ppython$!# or: dtrace -l -m python3.6   ID   PROVIDER            MODULE                          FUNCTION NAME29564 python18035        python3.6          _PyEval_EvalFrameDefault function-entry29565 python18035        python3.6             dtrace_function_entry function-entry29566 python18035        python3.6          _PyEval_EvalFrameDefault function-return29567 python18035        python3.6            dtrace_function_return function-return29568 python18035        python3.6                           collect gc-done29569 python18035        python3.6                           collect gc-start29570 python18035        python3.6          _PyEval_EvalFrameDefault line29571 python18035        python3.6                 maybe_dtrace_line line

On Linux, you can verify if the SystemTap static markers are present inthe built binary by seeing if it contains a “.note.stapsdt” section.

$readelf-S./python|grep.note.stapsdt[30] .note.stapsdt        NOTE         0000000000000000 00308d78

If you’ve built Python as a shared library(with the--enable-shared configure option), youneed to look instead within the shared library. For example:

$readelf-Slibpython3.3dm.so.1.0|grep.note.stapsdt[29] .note.stapsdt        NOTE         0000000000000000 00365b68

Sufficiently modern readelf can print the metadata:

$readelf-n./pythonDisplaying notes found at file offset 0x00000254 with length 0x00000020:    Owner                 Data size          Description    GNU                  0x00000010          NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 2.6.32Displaying notes found at file offset 0x00000274 with length 0x00000024:    Owner                 Data size          Description    GNU                  0x00000014          NT_GNU_BUILD_ID (unique build ID bitstring)        Build ID: df924a2b08a7e89f6e11251d4602022977af2670Displaying notes found at file offset 0x002d6c30 with length 0x00000144:    Owner                 Data size          Description    stapsdt              0x00000031          NT_STAPSDT (SystemTap probe descriptors)        Provider: python        Name: gc__start        Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6        Arguments: -4@%ebx    stapsdt              0x00000030          NT_STAPSDT (SystemTap probe descriptors)        Provider: python        Name: gc__done        Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8        Arguments: -8@%rax    stapsdt              0x00000045          NT_STAPSDT (SystemTap probe descriptors)        Provider: python        Name: function__entry        Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8        Arguments: 8@%rbp 8@%r12 -4@%eax    stapsdt              0x00000046          NT_STAPSDT (SystemTap probe descriptors)        Provider: python        Name: function__return        Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea        Arguments: 8@%rbp 8@%r12 -4@%eax

The above metadata contains information for SystemTap describing how itcan patch strategically placed machine code instructions to enable thetracing hooks used by a SystemTap script.

Static DTrace probes

The following example DTrace script can be used to show the call/returnhierarchy of a Python script, only tracing within the invocation ofa function called “start”. In other words, import-time functioninvocations are not going to be listed:

self int indent;python$target:::function-entry/copyinstr(arg1) == "start"/{        self->trace = 1;}python$target:::function-entry/self->trace/{        printf("%d\t%*s:", timestamp, 15, probename);        printf("%*s", self->indent, "");        printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);        self->indent++;}python$target:::function-return/self->trace/{        self->indent--;        printf("%d\t%*s:", timestamp, 15, probename);        printf("%*s", self->indent, "");        printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);}python$target:::function-return/copyinstr(arg1) == "start"/{        self->trace = 0;}

It can be invoked like this:

$sudodtrace-q-scall_stack.d-c"python3.6 script.py"

The output looks like this:

156641360502280  function-entry:call_stack.py:start:23156641360518804  function-entry: call_stack.py:function_1:1156641360532797  function-entry:  call_stack.py:function_3:9156641360546807 function-return:  call_stack.py:function_3:10156641360563367 function-return: call_stack.py:function_1:2156641360578365  function-entry: call_stack.py:function_2:5156641360591757  function-entry:  call_stack.py:function_1:1156641360605556  function-entry:   call_stack.py:function_3:9156641360617482 function-return:   call_stack.py:function_3:10156641360629814 function-return:  call_stack.py:function_1:2156641360642285 function-return: call_stack.py:function_2:6156641360656770  function-entry: call_stack.py:function_3:9156641360669707 function-return: call_stack.py:function_3:10156641360687853  function-entry: call_stack.py:function_4:13156641360700719 function-return: call_stack.py:function_4:14156641360719640  function-entry: call_stack.py:function_5:18156641360732567 function-return: call_stack.py:function_5:21156641360747370 function-return:call_stack.py:start:28

Static SystemTap markers

The low-level way to use the SystemTap integration is to use the staticmarkers directly. This requires you to explicitly state the binary filecontaining them.

For example, this SystemTap script can be used to show the call/returnhierarchy of a Python script:

probe process("python").mark("function__entry") {     filename = user_string($arg1);     funcname = user_string($arg2);     lineno = $arg3;     printf("%s => %s in %s:%d\\n",            thread_indent(1), funcname, filename, lineno);}probe process("python").mark("function__return") {    filename = user_string($arg1);    funcname = user_string($arg2);    lineno = $arg3;    printf("%s <= %s in %s:%d\\n",           thread_indent(-1), funcname, filename, lineno);}

It can be invoked like this:

$stap\show-call-hierarchy.stp\-c"./python test.py"

The output looks like this:

11408 python(8274):        => __contains__ in Lib/_abcoll.py:36211414 python(8274):         => __getitem__ in Lib/os.py:42511418 python(8274):          => encode in Lib/os.py:49011424 python(8274):          <= encode in Lib/os.py:49311428 python(8274):         <= __getitem__ in Lib/os.py:42611433 python(8274):        <= __contains__ in Lib/_abcoll.py:366

where the columns are:

  • time in microseconds since start of script

  • name of executable

  • PID of process

and the remainder indicates the call/return hierarchy as the script executes.

For a--enable-shared build of CPython, the markers are contained within thelibpython shared library, and the probe’s dotted path needs to reflect this. Forexample, this line from the above example:

probe process("python").mark("function__entry") {

should instead read:

probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") {

(assuming adebug build of CPython 3.6)

Available static markers

function__entry(strfilename,strfuncname,intlineno)

This marker indicates that execution of a Python function has begun.It is only triggered for pure-Python (bytecode) functions.

The filename, function name, and line number are provided back to thetracing script as positional arguments, which must be accessed using$arg1,$arg2,$arg3:

  • $arg1 :(constchar*) filename, accessible usinguser_string($arg1)

  • $arg2 :(constchar*) function name, accessible usinguser_string($arg2)

  • $arg3 :int line number

function__return(strfilename,strfuncname,intlineno)

This marker is the converse offunction__entry(), and indicates thatexecution of a Python function has ended (either viareturn, or via anexception). It is only triggered for pure-Python (bytecode) functions.

The arguments are the same as forfunction__entry()

line(strfilename,strfuncname,intlineno)

This marker indicates a Python line is about to be executed. It isthe equivalent of line-by-line tracing with a Python profiler. It isnot triggered within C functions.

The arguments are the same as forfunction__entry().

gc__start(intgeneration)

Fires when the Python interpreter starts a garbage collection cycle.arg0 is the generation to scan, likegc.collect().

gc__done(longcollected)

Fires when the Python interpreter finishes a garbage collectioncycle.arg0 is the number of collected objects.

import__find__load__start(strmodulename)

Fires beforeimportlib attempts to find and load the module.arg0 is the module name.

Added in version 3.7.

import__find__load__done(strmodulename,intfound)

Fires afterimportlib’s find_and_load function is called.arg0 is the module name,arg1 indicates if module wassuccessfully loaded.

Added in version 3.7.

audit(strevent,void*tuple)

Fires whensys.audit() orPySys_Audit() is called.arg0 is the event name as C string,arg1 is aPyObjectpointer to a tuple object.

Added in version 3.8.

SystemTap Tapsets

The higher-level way to use the SystemTap integration is to use a “tapset”:SystemTap’s equivalent of a library, which hides some of the lower-leveldetails of the static markers.

Here is a tapset file, based on a non-shared build of CPython:

/*   Provide a higher-level wrapping around the function__entry and   function__return markers: \*/probe python.function.entry = process("python").mark("function__entry"){    filename = user_string($arg1);    funcname = user_string($arg2);    lineno = $arg3;    frameptr = $arg4}probe python.function.return = process("python").mark("function__return"){    filename = user_string($arg1);    funcname = user_string($arg2);    lineno = $arg3;    frameptr = $arg4}

If this file is installed in SystemTap’s tapset directory (e.g./usr/share/systemtap/tapset), then these additional probepoints becomeavailable:

python.function.entry(strfilename,strfuncname,intlineno,frameptr)

This probe point indicates that execution of a Python function has begun.It is only triggered for pure-Python (bytecode) functions.

python.function.return(strfilename,strfuncname,intlineno,frameptr)

This probe point is the converse ofpython.function.return, andindicates that execution of a Python function has ended (either viareturn, or via an exception). It is only triggered for pure-Python(bytecode) functions.

Examples

This SystemTap script uses the tapset above to more cleanly implement theexample given above of tracing the Python function-call hierarchy, withoutneeding to directly name the static markers:

probe python.function.entry{  printf("%s => %s in %s:%d\n",         thread_indent(1), funcname, filename, lineno);}probe python.function.return{  printf("%s <= %s in %s:%d\n",         thread_indent(-1), funcname, filename, lineno);}

The following script uses the tapset above to provide a top-like view of allrunning CPython code, showing the top 20 most frequently entered bytecodeframes, each second, across the whole system:

global fn_calls;probe python.function.entry{    fn_calls[pid(), filename, funcname, lineno] += 1;}probe timer.ms(1000) {    printf("\033[2J\033[1;1H") /* clear screen \*/    printf("%6s %80s %6s %30s %6s\n",           "PID", "FILENAME", "LINE", "FUNCTION", "CALLS")    foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) {        printf("%6d %80s %6d %30s %6d\n",            pid, filename, lineno, funcname,            fn_calls[pid, filename, funcname, lineno]);    }    delete fn_calls;}