Overview¶

User based trace events allow user processes to create events and trace datathat can be viewed via existing tools, such as ftrace and perf.To enable this feature, build your kernel with CONFIG_USER_EVENTS=y.

Programs can view status of the events via/sys/kernel/tracing/user_events_status and can both register and writedata out via /sys/kernel/tracing/user_events_data.

Programs can also use /sys/kernel/tracing/dynamic_events to register anddelete user based events via the u: prefix. The format of the command todynamic_events is the same as the ioctl with the u: prefix applied. Thisrequires CAP_PERFMON due to the event persisting, otherwise -EPERM is returned.

Typically programs will register a set of events that they wish to expose totools that can read trace_events (such as ftrace and perf). The registrationprocess tells the kernel which address and bit to reflect if any tool hasenabled the event and data should be written. The registration will give backa write index which describes the data when a write() orwritev() is calledon the /sys/kernel/tracing/user_events_data file.

The structures referenced in this document are contained within the/include/uapi/linux/user_events.h file in the source tree.

NOTE:Both user_events_status and user_events_data are under the tracefsfilesystem and may be mounted at different paths than above.

Registering¶

Registering within a user process is done via ioctl() out to the/sys/kernel/tracing/user_events_data file. The command to issue isDIAG_IOCSREG.

This command takes a packedstructuser_reg as an argument:

struct user_reg {      /* Input: Size of the user_reg structure being used */      __u32 size;      /* Input: Bit in enable address to use */      __u8 enable_bit;      /* Input: Enable size in bytes at address */      __u8 enable_size;      /* Input: Flags to use, if any */      __u16 flags;      /* Input: Address to update when enabled */      __u64 enable_addr;      /* Input: Pointer to string with event name, description and flags */      __u64 name_args;      /* Output: Index of the event to use when writing data */      __u32 write_index;} __attribute__((__packed__));

Thestructuser_reg requires all the above inputs to be set appropriately.

• size: This must be set to sizeof(structuser_reg).

• enable_bit: The bit to reflect the event status at the address specified byenable_addr.

• enable_size: The size of the value specified by enable_addr.This must be 4 (32-bit) or 8 (64-bit). 64-bit values are only allowed to beused on 64-bit kernels, however, 32-bit can be used on all kernels.

• flags: The flags to use, if any.Callers should first attempt to use flags and retry without flags to ensuresupport for lower versions of the kernel. If a flag is not supported -EINVALis returned.

• enable_addr: The address of the value to use to reflect event status. Thismust be naturally aligned and write accessible within the user program.

• name_args: The name and arguments to describe the event, see command formatfor details.

The following flags are currently supported.

• USER_EVENT_REG_PERSIST: The event will not delete upon the last referenceclosing. Callers may use this if an event should exist even after theprocess closes or unregisters the event. Requires CAP_PERFMON otherwise-EPERM is returned.

• USER_EVENT_REG_MULTI_FORMAT: The event can contain multiple formats. Thisallows programs to prevent themselves from being blocked when their eventformat changes and they wish to use the same name. When this flag is used thetracepoint name will be in the new format of “name.unique_id” vs the olderformat of “name”. A tracepoint will be created for each unique pair of nameand format. This means if several processes use the same name and format,they will use the same tracepoint. If yet another process uses the same name,but a different format than the other processes, it will use a differenttracepoint with a new unique id. Recording programs need to scan tracefs forthe various different formats of the event name they are interested inrecording. The system name of the tracepoint will also use “user_events_multi”instead of “user_events”. This prevents single-format event names conflictingwith any multi-format event names within tracefs. The unique_id is output asa hex string. Recording programs should ensure the tracepoint name starts withthe event name they registered and has a suffix that starts with . and onlyhas hex characters. For example to find all versions of the event “test” youcan use the regex “^test.[0-9a-fA-F]+$”.

Upon successful registration the following is set.

• write_index: The index to use for this file descriptor that represents thisevent when writing out data. The index is unique to this instance of the filedescriptor that was used for the registration. See writing data for details.

User based events show up under tracefs like any other event under thesubsystem named “user_events”. This means tools that wish to attach to theevents need to use /sys/kernel/tracing/events/user_events/[name]/enableor perf record -e user_events:[name] when attaching/recording.

NOTE: The event subsystem name by default is “user_events”. Callers shouldnot assume it will always be “user_events”. Operators reserve the right in thefuture to change the subsystem name per-process to accommodate event isolation.In addition if the USER_EVENT_REG_MULTI_FORMAT flag is used the tracepoint namewill have a unique id appended to it and the system name will be“user_events_multi” as described above.

name[:FLAG1[,FLAG2...]] [Field1[;Field2...]]

type name [size]

struct mytype {  char data[20];};

struct mytype myname 20

Deleting¶

Deleting an event from within a user process is done via ioctl() out to the/sys/kernel/tracing/user_events_data file. The command to issue isDIAG_IOCSDEL.

This command only requires a single string specifying the event to delete byits name. Delete will only succeed if there are no references left to theevent (in both user and kernel space). User programs should use a separate fileto request deletes than the one used for registration due to this.

NOTE: By default events will auto-delete when there are no references leftto the event. If programs do not want auto-delete, they must use theUSER_EVENT_REG_PERSIST flag when registering the event. Once that flag is usedthe event exists until DIAG_IOCSDEL is invoked. Both register and delete of anevent that persists requires CAP_PERFMON, otherwise -EPERM is returned. Whenthere are multiple formats of the same event name, all events with the samename will be attempted to be deleted. If only a specific version is wanted tobe deleted then the /sys/kernel/tracing/dynamic_events file should be used forthat specific format of the event.

Unregistering¶

If after registering an event it is no longer wanted to be updated then it canbe disabled via ioctl() out to the /sys/kernel/tracing/user_events_data file.The command to issue is DIAG_IOCSUNREG. This is different than deleting, wheredeleting actually removes the event from the system. Unregistering simply tellsthe kernel your process is no longer interested in updates to the event.

This command takes a packedstructuser_unreg as an argument:

struct user_unreg {      /* Input: Size of the user_unreg structure being used */      __u32 size;      /* Input: Bit to unregister */      __u8 disable_bit;      /* Input: Reserved, set to 0 */      __u8 __reserved;      /* Input: Reserved, set to 0 */      __u16 __reserved2;      /* Input: Address to unregister */      __u64 disable_addr;} __attribute__((__packed__));

Thestructuser_unreg requires all the above inputs to be set appropriately.

• size: This must be set to sizeof(structuser_unreg).

• disable_bit: This must be set to the bit to disable (same bit that waspreviously registered via enable_bit).

• disable_addr: This must be set to the address to disable (same address that waspreviously registered via enable_addr).

NOTE: Events are automatically unregistered when execve() is invoked. Duringfork() the registered events will be retained and must be unregistered manuallyin each process if wanted.

Status¶

When tools attach/record user based events the status of the event is updatedin realtime. This allows user programs to only incur the cost of the write() orwritev() calls when something is actively attached to the event.

The kernel will update the specified bit that was registered for the event astools attach/detach from the event. User programs simply check if the bit is setto see if something is attached or not.

Administrators can easily check the status of all registered events by readingthe user_events_status file directly via a terminal. The output is as follows:

Name [# Comments]...Active: ActiveCountBusy: BusyCount

For example, on a system that has a single event the output looks like this:

testActive: 1Busy: 0

If a user enables the user event via ftrace, the output would change to this:

test # Used by ftraceActive: 1Busy: 1

Writing Data¶

After registering an event the same fd that was used to register can be usedto write an entry for that event. The write_index returned must be at the startof the data, then the remaining data is treated as the payload of the event.

For example, if write_index returned was 1 and I wanted to write out an intpayload of the event. Then the data would have to be 8 bytes (2 ints) in size,with the first 4 bytes being equal to 1 and the last 4 bytes being equal to thevalue I want as the payload.

In memory this would look like this:

int index;int payload;

User programs might have well known structs that they wish to use to emit outas payloads. In those caseswritev() can be used, with the first vector beingthe index and the following vector(s) being the actual event payload.

For example, if I have astructlike this:

struct payload {      int src;      int dst;      int flags;} __attribute__((__packed__));

It’s advised for user programs to do the following:

struct iovec io[2];struct payload e;io[0].iov_base = &write_index;io[0].iov_len = sizeof(write_index);io[1].iov_base = &e;io[1].iov_len = sizeof(e);writev(fd, (const struct iovec*)io, 2);

NOTE:The write_index is not emitted out into the trace being recorded.

Example Code¶

See sample code in samples/user_events.