2.1 Via the ‘set_event’ interface¶

The events which are available for tracing can be found in the file/sys/kernel/debug/tracing/available_events.

To enable a particular event, such as ‘sched_wakeup’, simply echo itto /sys/kernel/debug/tracing/set_event. For example:

# echo sched_wakeup >> /sys/kernel/debug/tracing/set_event

To disable an event, echo the event name to the set_event file prefixedwith an exclamation point:

# echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event

To disable all events, echo an empty line to the set_event file:

# echo > /sys/kernel/debug/tracing/set_event

To enable all events, echo*:* or*: to the set_event file:

# echo *:* > /sys/kernel/debug/tracing/set_event

The events are organized into subsystems, such as ext4, irq, sched,etc., and a full event name looks like this: <subsystem>:<event>. Thesubsystem name is optional, but it is displayed in the available_eventsfile. All of the events in a subsystem can be specified via the syntax<subsystem>:*; for example, to enable all irq events, you can use thecommand:

# echo 'irq:*' > /sys/kernel/debug/tracing/set_event

2.2 Via the ‘enable’ toggle¶

The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchyof directories.

To enable event ‘sched_wakeup’:

# echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable

To disable it:

# echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable

To enable all events in sched subsystem:

# echo 1 > /sys/kernel/debug/tracing/events/sched/enable

To enable all events:

# echo 1 > /sys/kernel/debug/tracing/events/enable

When reading one of these enable files, there are four results:

• 0 - all events this file affects are disabled
• 1 - all events this file affects are enabled
• X - there is a mixture of events enabled and disabled
• ? - this file does not affect any event

2.3 Boot option¶

In order to facilitate early boot debugging, use boot option:

trace_event=[event-list]

event-list is a comma separated list of events. See section 2.1 for eventformat.

5.1 Expression syntax¶

A filter expression consists of one or more ‘predicates’ that can becombined using the logical operators ‘&&’ and ‘||’. A predicate issimply a clause that compares the value of a field contained within alogged event with a constant value and returns either 0 or 1 dependingon whether the field value matched (1) or didn’t match (0):

field-name relational-operator value

Parentheses can be used to provide arbitrary logical groupings anddouble-quotes can be used to prevent the shell from interpretingoperators as shell metacharacters.

The field-names available for use in filters can be found in the‘format’ files for trace events (see section 4).

The relational-operators depend on the type of the field being tested:

The operators available for numeric fields are:

==, !=, <, <=, >, >=, &

And for string fields they are:

==, !=, ~

The glob (~) accepts a wild card character (*,?) and character classes([). For example:

prev_comm ~ "*sh"prev_comm ~ "sh*"prev_comm ~ "*sh*"prev_comm ~ "ba*sh"

5.2 Setting filters¶

A filter for an individual event is set by writing a filter expressionto the ‘filter’ file for the given event.

For example:

# cd /sys/kernel/debug/tracing/events/sched/sched_wakeup# echo "common_preempt_count > 4" > filter

A slightly more involved example:

# cd /sys/kernel/debug/tracing/events/signal/signal_generate# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter

If there is an error in the expression, you’ll get an ‘Invalidargument’ error when setting it, and the erroneous string along withan error message can be seen by looking at the filter e.g.:

# cd /sys/kernel/debug/tracing/events/signal/signal_generate# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter-bash: echo: write error: Invalid argument# cat filter((sig >= 10 && sig < 15) || dsig == 17) && comm != bash^parse_error: Field not found

Currently the caret (‘^’) for an error always appears at the beginning ofthe filter string; the error message should still be useful thougheven without more accurate position info.

5.3 Clearing filters¶

To clear the filter for an event, write a ‘0’ to the event’s filterfile.

To clear the filters for all events in a subsystem, write a ‘0’ to thesubsystem’s filter file.

5.3 Subsystem filters¶

For convenience, filters for every event in a subsystem can be set orcleared as a group by writing a filter expression into the filter fileat the root of the subsystem. Note however, that if a filter for anyevent within the subsystem lacks a field specified in the subsystemfilter, or if the filter can’t be applied for any other reason, thefilter for that event will retain its previous setting. This canresult in an unintended mixture of filters which could lead toconfusing (to the user who might think different filters are ineffect) trace output. Only filters that reference just the commonfields can be guaranteed to propagate successfully to all events.

Here are a few subsystem filter examples that also illustrate theabove points:

Clear the filters on all events in the sched subsystem:

# cd /sys/kernel/debug/tracing/events/sched# echo 0 > filter# cat sched_switch/filternone# cat sched_wakeup/filternone

Set a filter using only common fields for all events in the schedsubsystem (all events end up with the same filter):

# cd /sys/kernel/debug/tracing/events/sched# echo common_pid == 0 > filter# cat sched_switch/filtercommon_pid == 0# cat sched_wakeup/filtercommon_pid == 0

Attempt to set a filter using a non-common field for all events in thesched subsystem (all events but those that have a prev_pid field retaintheir old filters):

# cd /sys/kernel/debug/tracing/events/sched# echo prev_pid == 0 > filter# cat sched_switch/filterprev_pid == 0# cat sched_wakeup/filtercommon_pid == 0

5.4 PID filtering¶

The set_event_pid file in the same directory as the top events directoryexists, will filter all events from tracing any task that does not have thePID listed in the set_event_pid file.

# cd /sys/kernel/debug/tracing# echo $$ > set_event_pid# echo 1 > events/enable

Will only trace events for the current task.

To add more PIDs without losing the PIDs already included, use ‘>>’.

# echo 123 244 1 >> set_event_pid

Note

Writing into trace_marker (See Documentation/trace/ftrace.rst)can also enable triggers that are written into/sys/kernel/tracing/events/ftrace/print/trigger

6.1 Expression syntax¶

Triggers are added by echoing the command to the ‘trigger’ file:

# echo 'command[:count] [if filter]' > trigger

Triggers are removed by echoing the same command but starting with ‘!’to the ‘trigger’ file:

# echo '!command[:count] [if filter]' > trigger

The [if filter] part isn’t used in matching commands when removing, soleaving that off in a ‘!’ command will accomplish the same thing ashaving it in.

The filter syntax is the same as that described in the ‘Eventfiltering’ section above.

For ease of use, writing to the trigger file using ‘>’ currently justadds or removes a single trigger and there’s no explicit ‘>>’ support(‘>’ actually behaves like ‘>>’) or truncation support to remove alltriggers (you have to use ‘!’ for each one added.)

6.2 Supported trigger commands¶

The following commands are supported:

• enable_event/disable_event

  These commands can enable or disable another trace event wheneverthe triggering event is hit. When these commands are registered,the other trace event is activated, but disabled in a “soft” mode.That is, the tracepoint will be called, but just will not be traced.The event tracepoint stays in this mode as long as there’s a triggerin effect that can trigger it.

  For example, the following trigger causes kmalloc events to betraced when a read system call is entered, and the :1 at the endspecifies that this enablement happens only once:

  # echo 'enable_event:kmem:kmalloc:1' > \    /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger

  The following trigger causes kmalloc events to stop being tracedwhen a read system call exits. This disablement happens on everyread system call exit:

  # echo 'disable_event:kmem:kmalloc' > \    /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger

  The format is:

  enable_event:<system>:<event>[:count]disable_event:<system>:<event>[:count]

  To remove the above commands:

  # echo '!enable_event:kmem:kmalloc:1' > \    /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger# echo '!disable_event:kmem:kmalloc' > \    /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger

  Note that there can be any number of enable/disable_event triggersper triggering event, but there can only be one trigger pertriggered event. e.g. sys_enter_read can have triggers enabling bothkmem:kmalloc and sched:sched_switch, but can’t have two kmem:kmallocversions such as kmem:kmalloc and kmem:kmalloc:1 or ‘kmem:kmalloc ifbytes_req == 256’ and ‘kmem:kmalloc if bytes_alloc == 256’ (theycould be combined into a single filter on kmem:kmalloc though).

• stacktrace

  This command dumps a stacktrace in the trace buffer whenever thetriggering event occurs.

  For example, the following trigger dumps a stacktrace every time thekmalloc tracepoint is hit:

  # echo 'stacktrace' > \      /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger

  The following trigger dumps a stacktrace the first 5 times a kmallocrequest happens with a size >= 64K:

  # echo 'stacktrace:5 if bytes_req >= 65536' > \      /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger

  The format is:

  stacktrace[:count]

  To remove the above commands:

  # echo '!stacktrace' > \      /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger# echo '!stacktrace:5 if bytes_req >= 65536' > \      /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger

  The latter can also be removed more simply by the following (withoutthe filter):

  # echo '!stacktrace:5' > \      /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger

  Note that there can be only one stacktrace trigger per triggeringevent.

• snapshot

  This command causes a snapshot to be triggered whenever thetriggering event occurs.

  The following command creates a snapshot every time a block requestqueue is unplugged with a depth > 1. If you were tracing a set ofevents or functions at the time, the snapshot trace buffer wouldcapture those events when the trigger event occurred:

  # echo 'snapshot if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger

  To only snapshot once:

  # echo 'snapshot:1 if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger

  To remove the above commands:

  # echo '!snapshot if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger# echo '!snapshot:1 if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger

  Note that there can be only one snapshot trigger per triggeringevent.

• traceon/traceoff

  These commands turn tracing on and off when the specified events arehit. The parameter determines how many times the tracing system isturned on and off. If unspecified, there is no limit.

  The following command turns tracing off the first time a blockrequest queue is unplugged with a depth > 1. If you were tracing aset of events or functions at the time, you could then examine thetrace buffer to see the sequence of events that led up to thetrigger event:

  # echo 'traceoff:1 if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger

  To always disable tracing when nr_rq > 1:

  # echo 'traceoff if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger

  To remove the above commands:

  # echo '!traceoff:1 if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger# echo '!traceoff if nr_rq > 1' > \      /sys/kernel/debug/tracing/events/block/block_unplug/trigger

  Note that there can be only one traceon or traceoff trigger pertriggering event.

• hist

  This command aggregates event hits into a hash table keyed on one ormore trace event format fields (or stacktrace) and a set of runningtotals derived from one or more trace event format fields and/orevent counts (hitcount).

  See Documentation/trace/histogram.rst for details and examples.

7.1 Dyamically creating synthetic event definitions¶

There are a couple ways to create a new synthetic event from a kernelmodule or other kernel code.

The first creates the event in one step, using synth_event_create().In this method, the name of the event to create and an array definingthe fields is supplied to synth_event_create(). If successful, asynthetic event with that name and fields will exist following thatcall. For example, to create a new “schedtest” synthetic event:

ret = synth_event_create("schedtest", sched_fields,                         ARRAY_SIZE(sched_fields), THIS_MODULE);

The sched_fields param in this example points to an array of structsynth_field_desc, each of which describes an event field by type andname:

static struct synth_field_desc sched_fields[] = {      { .type = "pid_t",              .name = "next_pid_field" },      { .type = "char[16]",           .name = "next_comm_field" },      { .type = "u64",                .name = "ts_ns" },      { .type = "u64",                .name = "ts_ms" },      { .type = "unsigned int",       .name = "cpu" },      { .type = "char[64]",           .name = "my_string_field" },      { .type = "int",                .name = "my_int_field" },};

See synth_field_size() for available types. If field_name contains [n]the field is considered to be an array.

If the event is created from within a module, a pointer to the modulemust be passed to synth_event_create(). This will ensure that thetrace buffer won’t contain unreadable events when the module isremoved.

At this point, the event object is ready to be used for generating newevents.

In the second method, the event is created in several steps. Thisallows events to be created dynamically and without the need to createand populate an array of fields beforehand.

To use this method, an empty or partially empty synthetic event shouldfirst be created using synth_event_gen_cmd_start() orsynth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(),the name of the event along with one or more pairs of args each pairrepresenting a ‘type field_name;’ field specification should besupplied. For synth_event_gen_cmd_array_start(), the name of theevent along with an array of struct synth_field_desc should besupplied. Before calling synth_event_gen_cmd_start() orsynth_event_gen_cmd_array_start(), the user should create andinitialize a dynevent_cmd object using synth_event_cmd_init().

For example, to create a new “schedtest” synthetic event with twofields:

struct dynevent_cmd cmd;char *buf;/* Create a buffer to hold the generated command */buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);/* Before generating the command, initialize the cmd object */synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,                                "pid_t", "next_pid_field",                                "u64", "ts_ns");

Alternatively, using an array of struct synth_field_desc fieldscontaining the same information:

ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,                                      fields, n_fields);

Once the synthetic event object has been created, it can then bepopulated with more fields. Fields are added one by one usingsynth_event_add_field(), supplying the dynevent_cmd object, a fieldtype, and a field name. For example, to add a new int field named“intfield”, the following call should be made:

ret = synth_event_add_field(&cmd, "int", "intfield");

See synth_field_size() for available types. If field_name contains [n]the field is considered to be an array.

A group of fields can also be added all at once using an array ofsynth_field_desc with add_synth_fields(). For example, this would addjust the first four sched_fields:

ret = synth_event_add_fields(&cmd, sched_fields, 4);

If you already have a string of the form ‘type field_name’,synth_event_add_field_str() can be used to add it as-is; it willalso automatically append a ‘;’ to the string.

Once all the fields have been added, the event should be finalized andregistered by calling the synth_event_gen_cmd_end() function:

ret = synth_event_gen_cmd_end(&cmd);

At this point, the event object is ready to be used for tracing newevents.

7.2 Tracing synthetic events from in-kernel code¶

To trace a synthetic event, there are several options. The firstoption is to trace the event in one call, using synth_event_trace()with a variable number of values, or synth_event_trace_array() with anarray of values to be set. A second option can be used to avoid theneed for a pre-formed array of values or list of arguments, viasynth_event_trace_start() and synth_event_trace_end() along withsynth_event_add_next_val() or synth_event_add_val() to add the valuespiecewise.

7.2.1 Tracing a synthetic event all at once¶

To trace a synthetic event all at once, the synth_event_trace() orsynth_event_trace_array() functions can be used.

The synth_event_trace() function is passed the trace_event_filerepresenting the synthetic event (which can be retrieved usingtrace_get_event_file() using the synthetic event name, “synthetic” asthe system name, and the trace instance name (NULL if using the globaltrace array)), along with an variable number of u64 args, one for eachsynthetic event field, and the number of values being passed.

So, to trace an event corresponding to the synthetic event definitionabove, code like the following could be used:

ret = synth_event_trace(create_synth_test, 7, /* number of values */                        444,             /* next_pid_field */                        (u64)"clackers", /* next_comm_field */                        1000000,         /* ts_ns */                        1000,            /* ts_ms */                        smp_processor_id(),/* cpu */                        (u64)"Thneed",   /* my_string_field */                        999);            /* my_int_field */

All vals should be cast to u64, and string vals are just pointers tostrings, cast to u64. Strings will be copied into space reserved inthe event for the string, using these pointers.

Alternatively, the synth_event_trace_array() function can be used toaccomplish the same thing. It is passed the trace_event_filerepresenting the synthetic event (which can be retrieved usingtrace_get_event_file() using the synthetic event name, “synthetic” asthe system name, and the trace instance name (NULL if using the globaltrace array)), along with an array of u64, one for each syntheticevent field.

To trace an event corresponding to the synthetic event definitionabove, code like the following could be used:

u64 vals[7];vals[0] = 777;                  /* next_pid_field */vals[1] = (u64)"tiddlywinks";   /* next_comm_field */vals[2] = 1000000;              /* ts_ns */vals[3] = 1000;                 /* ts_ms */vals[4] = smp_processor_id();   /* cpu */vals[5] = (u64)"thneed";        /* my_string_field */vals[6] = 398;                  /* my_int_field */

The ‘vals’ array is just an array of u64, the number of which mustmatch the number of field in the synthetic event, and which must be inthe same order as the synthetic event fields.

All vals should be cast to u64, and string vals are just pointers tostrings, cast to u64. Strings will be copied into space reserved inthe event for the string, using these pointers.

In order to trace a synthetic event, a pointer to the trace event fileis needed. The trace_get_event_file() function can be used to getit - it will find the file in the given trace instance (in this caseNULL since the top trace array is being used) while at the same timepreventing the instance containing it from going away:

schedtest_event_file = trace_get_event_file(NULL, "synthetic",                                            "schedtest");

Before tracing the event, it should be enabled in some way, otherwisethe synthetic event won’t actually show up in the trace buffer.

To enable a synthetic event from the kernel, trace_array_set_clr_event()can be used (which is not specific to synthetic events, so does needthe “synthetic” system name to be specified explicitly).

To enable the event, pass ‘true’ to it:

trace_array_set_clr_event(schedtest_event_file->tr,                          "synthetic", "schedtest", true);

To disable it pass false:

trace_array_set_clr_event(schedtest_event_file->tr,                          "synthetic", "schedtest", false);

Finally, synth_event_trace_array() can be used to actually trace theevent, which should be visible in the trace buffer afterwards:

ret = synth_event_trace_array(schedtest_event_file, vals,                              ARRAY_SIZE(vals));

To remove the synthetic event, the event should be disabled, and thetrace instance should be ‘put’ back using trace_put_event_file():

trace_array_set_clr_event(schedtest_event_file->tr,                          "synthetic", "schedtest", false);trace_put_event_file(schedtest_event_file);

If those have been successful, synth_event_delete() can be called toremove the event:

ret = synth_event_delete("schedtest");

7.2.2 Tracing a synthetic event piecewise¶

To trace a synthetic using the piecewise method described above, thesynth_event_trace_start() function is used to ‘open’ the syntheticevent trace:

struct synth_trace_state trace_state;ret = synth_event_trace_start(schedtest_event_file, &trace_state);

It’s passed the trace_event_file representing the synthetic eventusing the same methods as described above, along with a pointer to astruct synth_trace_state object, which will be zeroed before use andused to maintain state between this and following calls.

Once the event has been opened, which means space for it has beenreserved in the trace buffer, the individual fields can be set. Thereare two ways to do that, either one after another for each field inthe event, which requires no lookups, or by name, which does. Thetradeoff is flexibility in doing the assignments vs the cost of alookup per field.

To assign the values one after the other without lookups,synth_event_add_next_val() should be used. Each call is passed thesame synth_trace_state object used in the synth_event_trace_start(),along with the value to set the next field in the event. After eachfield is set, the ‘cursor’ points to the next field, which will be setby the subsequent call, continuing until all the fields have been setin order. The same sequence of calls as in the above examples usingthis method would be (without error-handling code):

/* next_pid_field */ret = synth_event_add_next_val(777, &trace_state);/* next_comm_field */ret = synth_event_add_next_val((u64)"slinky", &trace_state);/* ts_ns */ret = synth_event_add_next_val(1000000, &trace_state);/* ts_ms */ret = synth_event_add_next_val(1000, &trace_state);/* cpu */ret = synth_event_add_next_val(smp_processor_id(), &trace_state);/* my_string_field */ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);/* my_int_field */ret = synth_event_add_next_val(395, &trace_state);

To assign the values in any order, synth_event_add_val() should beused. Each call is passed the same synth_trace_state object used inthe synth_event_trace_start(), along with the field name of the fieldto set and the value to set it to. The same sequence of calls as inthe above examples using this method would be (without error-handlingcode):

ret = synth_event_add_val("next_pid_field", 777, &trace_state);ret = synth_event_add_val("next_comm_field", (u64)"silly putty",                          &trace_state);ret = synth_event_add_val("ts_ns", 1000000, &trace_state);ret = synth_event_add_val("ts_ms", 1000, &trace_state);ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);ret = synth_event_add_val("my_string_field", (u64)"thneed_9",                          &trace_state);ret = synth_event_add_val("my_int_field", 3999, &trace_state);

Note that synth_event_add_next_val() and synth_event_add_val() areincompatible if used within the same trace of an event - either onecan be used but not both at the same time.

Finally, the event won’t be actually traced until it’s ‘closed’,which is done using synth_event_trace_end(), which takes only thestruct synth_trace_state object used in the previous calls:

ret = synth_event_trace_end(&trace_state);

Note that synth_event_trace_end() must be called at the end regardlessof whether any of the add calls failed (say due to a bad field namebeing passed in).

7.3 Dyamically creating kprobe and kretprobe event definitions¶

To create a kprobe or kretprobe trace event from kernel code, thekprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()functions can be used.

To create a kprobe event, an empty or partially empty kprobe eventshould first be created using kprobe_event_gen_cmd_start(). The nameof the event and the probe location should be specfied along with oneor args each representing a probe field should be supplied to thisfunction. Before calling kprobe_event_gen_cmd_start(), the usershould create and initialize a dynevent_cmd object usingkprobe_event_cmd_init().

For example, to create a new “schedtest” kprobe event with two fields:

struct dynevent_cmd cmd;char *buf;/* Create a buffer to hold the generated command */buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);/* Before generating the command, initialize the cmd object */kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);/* * Define the gen_kprobe_test event with the first 2 kprobe * fields. */ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",                                 "dfd=%ax", "filename=%dx");

Once the kprobe event object has been created, it can then bepopulated with more fields. Fields can be added usingkprobe_event_add_fields(), supplying the dynevent_cmd object alongwith a variable arg list of probe fields. For example, to add acouple additional fields, the following call could be made:

ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");

Once all the fields have been added, the event should be finalized andregistered by calling the kprobe_event_gen_cmd_end() orkretprobe_event_gen_cmd_end() functions, depending on whether a kprobeor kretprobe command was started:

ret = kprobe_event_gen_cmd_end(&cmd);

or:

ret = kretprobe_event_gen_cmd_end(&cmd);

At this point, the event object is ready to be used for tracing newevents.

Similarly, a kretprobe event can be created usingkretprobe_event_gen_cmd_start() with a probe name and location andadditional params such as $retval:

ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",                                    "do_sys_open", "$retval");

Similar to the synthetic event case, code like the following can beused to enable the newly created kprobe event:

gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");ret = trace_array_set_clr_event(gen_kprobe_test->tr,                                "kprobes", "gen_kprobe_test", true);

Finally, also similar to synthetic events, the following code can beused to give the kprobe event file back and delete the event:

trace_put_event_file(gen_kprobe_test);ret = kprobe_event_delete("gen_kprobe_test");

7.4 The “dynevent_cmd” low-level API¶

Both the in-kernel synthetic event and kprobe interfaces are built ontop of a lower-level “dynevent_cmd” interface. This interface ismeant to provide the basis for higher-level interfaces such as thesynthetic and kprobe interfaces, which can be used as examples.

The basic idea is simple and amounts to providing a general-purposelayer that can be used to generate trace event commands. Thegenerated command strings can then be passed to the command-parsingand event creation code that already exists in the trace eventsubystem for creating the corresponding trace events.

In a nutshell, the way it works is that the higher-level interfacecode creates a struct dynevent_cmd object, then uses a couplefunctions, dynevent_arg_add() and dynevent_arg_pair_add() to build upa command string, which finally causes the command to be executedusing the dynevent_create() function. The details of the interfaceare described below.

The first step in building a new command string is to create andinitialize an instance of a dynevent_cmd. Here, for instance, wecreate a dynevent_cmd on the stack and initialize it:

struct dynevent_cmd cmd;char *buf;int ret;buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,                  foo_event_run_command);

The dynevent_cmd initialization needs to be given a user-specifiedbuffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be usedfor this purpose - at 2k it’s generally too big to be comfortably puton the stack, so is dynamically allocated), a dynevent type id, whichis meant to be used to check that further API calls are for thecorrect command type, and a pointer to an event-specific run_command()callback that will be called to actually execute the event-specificcommand function.

Once that’s done, the command string can by built up by successivecalls to argument-adding functions.

To add a single argument, define and initialize a struct dynevent_argor struct dynevent_arg_pair object. Here’s an example of the simplestpossible arg addition, which is simply to append the given string asa whitespace-separated argument to the command:

struct dynevent_arg arg;dynevent_arg_init(&arg, NULL, 0);arg.str = name;ret = dynevent_arg_add(cmd, &arg);

The arg object is first initialized using dynevent_arg_init() and inthis case the parameters are NULL or 0, which means there’s nooptional sanity-checking function or separator appended to the end ofthe arg.

Here’s another more complicated example using an ‘arg pair’, which isused to create an argument that consists of a couple components addedtogether as a unit, for example, a ‘type field_name;’ arg or a simpleexpression arg e.g. ‘flags=%cx’:

struct dynevent_arg_pair arg_pair;dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');arg_pair.lhs = type;arg_pair.rhs = name;ret = dynevent_arg_pair_add(cmd, &arg_pair);

Again, the arg_pair is first initialized, in this case with a callbackfunction used to check the sanity of the args (for example, thatneither part of the pair is NULL), along with a character to be usedto add an operator between the pair (here none) and a separator to beappended onto the end of the arg pair (here ‘;’).

There’s also a dynevent_str_add() function that can be used to simplyadd a string as-is, with no spaces, delimeters, or arg check.

Any number of dynevent_*_add() calls can be made to build up the string(until its length surpasses cmd->maxlen). When all the arguments havebeen added and the command string is complete, the only thing left todo is run the command, which happens by simply callingdynevent_create():

ret = dynevent_create(&cmd);

At that point, if the return value is 0, the dynamic event has beencreated and is ready to use.

See the dynevent_cmd function definitions themselves for the detailsof the API.