Note

the host filesystem doesn’t need to be mounted for kernelclients to create or use channels - it only needs to bemounted when user space applications need access to the bufferdata.

Creating a channel¶

relay_open() is used to create a channel, along with its per-cpuchannel buffers. Each channel buffer will have an associated filecreated for it in the host filesystem, which can be and mmapped orread from in user space. The files are named basename0…basenameN-1where N is the number of online cpus, and by default will be createdin the root of the filesystem (if the parent param is NULL). If youwant a directory structure to contain your relay files, you shouldcreate it using the host filesystem’s directory creation function,e.g.debugfs_create_dir(), and pass the parent directory torelay_open(). Users are responsible for cleaning up any directorystructure they create, when the channel is closed - again the hostfilesystem’s directory removal functions should be used for that,e.g.debugfs_remove().

In order for a channel to be created and the host filesystem’s filesassociated with its channel buffers, the user must provide definitionsfor two callback functions, create_buf_file() and remove_buf_file().create_buf_file() is called once for each per-cpu buffer fromrelay_open() and allows the user to create the file which will be usedto represent the corresponding channel buffer. The callback shouldreturn the dentry of the file created to represent the channel buffer.remove_buf_file() must also be defined; it’s responsible for deletingthe file(s) created in create_buf_file() and is called duringrelay_close().

Here are some typical definitions for these callbacks, in this caseusing debugfs:

/** create_buf_file() callback.  Creates relay file in debugfs.*/static struct dentry *create_buf_file_handler(const char *filename,                                            struct dentry *parent,                                            umode_t mode,                                            struct rchan_buf *buf,                                            int *is_global){        return debugfs_create_file(filename, mode, parent, buf,                                &relay_file_operations);}/** remove_buf_file() callback.  Removes relay file from debugfs.*/static int remove_buf_file_handler(struct dentry *dentry){        debugfs_remove(dentry);        return 0;}/** relay interface callbacks*/static struct rchan_callbacks relay_callbacks ={        .create_buf_file = create_buf_file_handler,        .remove_buf_file = remove_buf_file_handler,};

And an examplerelay_open() invocation using them:

chan = relay_open("cpu", NULL, SUBBUF_SIZE, N_SUBBUFS, &relay_callbacks, NULL);

If the create_buf_file() callback fails, or isn’t defined, channelcreation and thusrelay_open() will fail.

The total size of each per-cpu buffer is calculated by multiplying thenumber of sub-buffers by the sub-buffer size passed intorelay_open().The idea behind sub-buffers is that they’re basically an extension ofdouble-buffering to N buffers, and they also allow applications toeasily implement random-access-on-buffer-boundary schemes, which canbe important for some high-volume applications. The number and sizeof sub-buffers is completely dependent on the application and even forthe same application, different conditions will warrant differentvalues for these parameters at different times. Typically, the rightvalues to use are best decided after some experimentation; in general,though, it’s safe to assume that having only 1 sub-buffer is a badidea - you’re guaranteed to either overwrite data or lose eventsdepending on the channel mode being used.

The create_buf_file() implementation can also be defined in such a wayas to allow the creation of a single ‘global’ buffer instead of thedefault per-cpu set. This can be useful for applications interestedmainly in seeing the relative ordering of system-wide events withoutthe need to bother with saving explicit timestamps for the purpose ofmerging/sorting per-cpu files in a postprocessing step.

To haverelay_open() create a global buffer, the create_buf_file()implementation should set the value of the is_global outparam to anon-zero value in addition to creating the file that will be used torepresent the single buffer. In the case of a global buffer,create_buf_file() and remove_buf_file() will be called only once. Thenormal channel-writing functions, e.g. relay_write(), can still beused - writes from any cpu will transparently end up in the globalbuffer - but since it is a global buffer, callers should make surethey use the proper locking for such a buffer, either by wrappingwrites in a spinlock, or by copying a write function from relay.h andcreating a local version that internally does the proper locking.

The private_data passed intorelay_open() allows clients to associateuser-defined data with a channel, and is immediately available(including in create_buf_file()) via chan->private_data orbuf->chan->private_data.

Buffer-only channels¶

These channels have no files associated and can be created withrelay_open(NULL, NULL, …). Such channels are useful in scenarios suchas when doing early tracing in the kernel, before the VFS is up. In thesecases, one may open a buffer-only channel and then callrelay_late_setup_files() when the kernel is ready to handle files,to expose the buffered data to the userspace.

Channel ‘modes’¶

relay channels can be used in either of two modes - ‘overwrite’ or‘no-overwrite’. The mode is entirely determined by the implementationof the subbuf_start() callback, as described below. The default if nosubbuf_start() callback is defined is ‘no-overwrite’ mode. If thedefault mode suits your needs, and you plan to use the read()interface to retrieve channel data, you can ignore the details of thissection, as it pertains mainly to mmap() implementations.

In ‘overwrite’ mode, also known as ‘flight recorder’ mode, writescontinuously cycle around the buffer and will never fail, but willunconditionally overwrite old data regardless of whether it’s actuallybeen consumed. In no-overwrite mode, writes will fail, i.e. data willbe lost, if the number of unconsumed sub-buffers equals the totalnumber of sub-buffers in the channel. It should be clear that ifthere is no consumer or if the consumer can’t consume sub-buffers fastenough, data will be lost in either case; the only difference iswhether data is lost from the beginning or the end of a buffer.

As explained above, a relay channel is made of up one or moreper-cpu channel buffers, each implemented as a circular buffersubdivided into one or more sub-buffers. Messages are written intothe current sub-buffer of the channel’s current per-cpu buffer via thewrite functions described below. Whenever a message can’t fit intothe current sub-buffer, because there’s no room left for it, theclient is notified via the subbuf_start() callback that a switch to anew sub-buffer is about to occur. The client uses this callback to 1)initialize the next sub-buffer if appropriate 2) finalize the previoussub-buffer if appropriate and 3) return a boolean value indicatingwhether or not to actually move on to the next sub-buffer.

To implement ‘no-overwrite’ mode, the userspace client would providean implementation of the subbuf_start() callback something like thefollowing:

static int subbuf_start(struct rchan_buf *buf,                        void *subbuf,                        void *prev_subbuf,                        unsigned int prev_padding){        if (prev_subbuf)                *((unsigned *)prev_subbuf) = prev_padding;        if (relay_buf_full(buf))                return 0;        subbuf_start_reserve(buf, sizeof(unsigned int));        return 1;}

If the current buffer is full, i.e. all sub-buffers remain unconsumed,the callback returns 0 to indicate that the buffer switch should notoccur yet, i.e. until the consumer has had a chance to read thecurrent set of ready sub-buffers. For therelay_buf_full() functionto make sense, the consumer is responsible for notifying the relayinterface when sub-buffers have been consumed viarelay_subbufs_consumed(). Any subsequent attempts to write into thebuffer will again invoke the subbuf_start() callback with the sameparameters; only when the consumer has consumed one or more of theready sub-buffers willrelay_buf_full() return 0, in which case thebuffer switch can continue.

The implementation of the subbuf_start() callback for ‘overwrite’ modewould be very similar:

static int subbuf_start(struct rchan_buf *buf,                        void *subbuf,                        void *prev_subbuf,                        size_t prev_padding){        if (prev_subbuf)                *((unsigned *)prev_subbuf) = prev_padding;        subbuf_start_reserve(buf, sizeof(unsigned int));        return 1;}

In this case, therelay_buf_full() check is meaningless and thecallback always returns 1, causing the buffer switch to occurunconditionally. It’s also meaningless for the client to use therelay_subbufs_consumed() function in this mode, as it’s neverconsulted.

The default subbuf_start() implementation, used if the client doesn’tdefine any callbacks, or doesn’t define the subbuf_start() callback,implements the simplest possible ‘no-overwrite’ mode, i.e. it doesnothing but return 0.

Header information can be reserved at the beginning of each sub-bufferby calling the subbuf_start_reserve() helper function from within thesubbuf_start() callback. This reserved area can be used to storewhatever information the client wants. In the example above, room isreserved in each sub-buffer to store the padding count for thatsub-buffer. This is filled in for the previous sub-buffer in thesubbuf_start() implementation; the padding value for the previoussub-buffer is passed into the subbuf_start() callback along with apointer to the previous sub-buffer, since the padding value isn’tknown until a sub-buffer is filled. The subbuf_start() callback isalso called for the first sub-buffer when the channel is opened, togive the client a chance to reserve space in it. In this case theprevious sub-buffer pointer passed into the callback will be NULL, sothe client should check the value of the prev_subbuf pointer beforewriting into the previous sub-buffer.

Writing to a channel¶

Kernel clients write data into the current cpu’s channel buffer usingrelay_write() or __relay_write(). relay_write() is the main loggingfunction - it uses local_irqsave() to protect the buffer and should beused if you might be logging from interrupt context. If you knowyou’ll never be logging from interrupt context, you can use__relay_write(), which only disables preemption. These functionsdon’t return a value, so you can’t determine whether or not theyfailed - the assumption is that you wouldn’t want to check a returnvalue in the fast logging path anyway, and that they’ll always succeedunless the buffer is full and no-overwrite mode is being used, inwhich case you can detect a failed write in the subbuf_start()callback by calling therelay_buf_full() helper function.

relay_reserve() is used to reserve a slot in a channel buffer whichcan be written to later. This would typically be used in applicationsthat need to write directly into a channel buffer without having tostage data in a temporary buffer beforehand. Because the actual writemay not happen immediately after the slot is reserved, applicationsusing relay_reserve() can keep a count of the number of bytes actuallywritten, either in space reserved in the sub-buffers themselves or asa separate array. See the ‘reserve’ example in the relay-apps tarballathttp://relayfs.sourceforge.net for an example of how this can bedone. Because the write is under control of the client and isseparated from the reserve, relay_reserve() doesn’t protect the bufferat all - it’s up to the client to provide the appropriatesynchronization when using relay_reserve().

Closing a channel¶

The client callsrelay_close() when it’s finished using the channel.The channel and its associated buffers are destroyed when there are nolonger any references to any of the channel buffers.relay_flush()forces a sub-buffer switch on all the channel buffers, and can be usedto finalize and process the last sub-buffers before the channel isclosed.

Misc¶

Some applications may want to keep a channel around and re-use itrather than open and close a new channel for each use.relay_reset()can be used for this purpose - it resets a channel to its initialstate without reallocating channel buffer memory or destroyingexisting mappings. It should however only be called when it’s safe todo so, i.e. when the channel isn’t currently being written to.

Finally, there are a couple of utility callbacks that can be used fordifferent purposes. buf_mapped() is called whenever a channel bufferis mmapped from user space and buf_unmapped() is called when it’sunmapped. The client can use this notification to trigger actionswithin the kernel application, such as enabling/disabling logging tothe channel.