Netconsole

started by Ingo Molnar <mingo@redhat.com>, 2001.09.17

2.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003

IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013

Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015

Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023

Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024

Sysdata append support by Breno Leitao <leitao@debian.org>, Jan 15 2025

Introduction:

This module logs kernel printk messages over UDP allowing debugging ofproblem where disk logging fails and serial consoles are impractical.

It can be used either built-in or as a module. As a built-in,netconsole initializes immediately after NIC cards and will bring upthe specified interface as soon as possible. While this doesn’t allowcapture of early kernel panics, it does capture most of the bootprocess.

Sender and receiver configuration:

It takes a string configuration parameter “netconsole” in thefollowing format:

netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]  where       +             if present, enable extended console support       r             if present, prepend kernel version (release) to the message       src-port      source for UDP packets (defaults to 6665)       src-ip        source IP to use (interface address)       dev           network interface name (eth0) or MAC address       tgt-port      port for logging agent (6666)       tgt-ip        IP address for logging agent       tgt-macaddr   ethernet MAC address for logging agent (broadcast)

Examples:

linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc

or:

insmod netconsole netconsole=@/,@10.0.0.2/

or using IPv6:

insmod netconsole netconsole=@/,@fd00:1:2:3::1/

or using a MAC address to select the egress interface:

linux netconsole=4444@10.0.0.1/22:33:44:55:66:77,9353@10.0.0.2/12:34:56:78:9a:bc

It also supports logging to multiple remote agents by specifyingparameters for the multiple agents separated by semicolons and thecomplete string enclosed in “quotes”, thusly:

modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/"

Built-in netconsole starts immediately after the TCP stack isinitialized and attempts to bring up the supplied dev at the suppliedaddress.

The remote host has several options to receive the kernel messages,for example:

  1. syslogd

  2. netcat

    On distributions using a BSD-based netcat version (e.g. Fedora,openSUSE and Ubuntu) the listening port must be specified withoutthe -p switch:

    nc -u -l -p <port>' / 'nc -u -l <port>

    or:

    netcat -u -l -p <port>' / 'netcat -u -l <port>

  3. socat

socat udp-recv:<port> -

Dynamic reconfiguration:

Dynamic reconfigurability is a useful addition to netconsole that enablesremote logging targets to be dynamically added, removed, or have theirparameters reconfigured at runtime from a configfs-based userspace interface.

To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building thenetconsole module (or kernel, if netconsole is built-in).

Some examples follow (where configfs is mounted at the /sys/kernel/configmountpoint).

To add a remote logging target (target names can be arbitrary):

cd /sys/kernel/config/netconsole/mkdir target1

Note that newly created targets have default parameter values (as mentionedabove) and are disabled by default -- they must first be enabled by writing“1” to the “enabled” attribute (usually after setting parameters accordingly)as described below.

To remove a target:

rmdir /sys/kernel/config/netconsole/othertarget/

The interface exposes these parameters of a netconsole target to userspace:

enabled

Is this target currently enabled?

(read-write)

extended

Extended mode enabled

(read-write)

release

Prepend kernel release to message

(read-write)

dev_name

Local network interface name

(read-write)

local_port

Source UDP port to use

(read-write)

remote_port

Remote agent’s UDP port

(read-write)

local_ip

Source IP address to use

(read-write)

remote_ip

Remote agent’s IP address

(read-write)

local_mac

Local interface’s MAC address

(read-only)

remote_mac

Remote agent’s MAC address

(read-write)

transmit_errors

Number of packet send errors

(read-only)

The “enabled” attribute is also used to control whether the parameters ofa target can be updated or not -- you can modify the parameters of onlydisabled targets (i.e. if “enabled” is 0).

To update a target’s parameters:

cat enabled                            # check if enabled is 1echo 0 > enabled                       # disable the target (if required)echo eth2 > dev_name                   # set local interfaceecho 10.0.0.4 > remote_ip              # update some parameterecho cb:a9:87:65:43:21 > remote_mac    # update more parametersecho 1 > enabled                       # enable target again

You can also update the local interface dynamically. This is especiallyuseful if you want to use interfaces that have newly come up (and may nothave existed when netconsole was loaded / initialized).

Netconsole targets defined at boot time (or module load time) with thenetconsole= param are assigned the namecmdline<index>. For example, thefirst target in the parameter is namedcmdline0. You can control and modifythese targets by creating configfs directories with the matching name.

Let’s suppose you have two netconsole targets defined at boot time:

netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc

You can modify these targets in runtime by creating the following targets:

mkdir cmdline0cat cmdline0/remote_ip10.0.0.2mkdir cmdline1cat cmdline1/remote_ip10.0.0.3

Append User Data

Custom user data can be appended to the end of messages with netconsoledynamic configuration enabled. User data entries can be modified withoutchanging the “enabled” attribute of a target.

Directories (keys) underuserdata are limited to 53 character length, anddata inuserdata/<key>/value are limited to 200 bytes:

cd /sys/kernel/config/netconsole && mkdir cmdline0cd cmdline0mkdir userdata/fooecho bar > userdata/foo/valuemkdir userdata/quxecho baz > userdata/qux/value

Messages will now include this additional user data:

echo "This is a message" > /dev/kmsg

Sends:

12,607,22085407756,-;This is a message foo=bar qux=baz

Preview the userdata that will be appended with:

cd /sys/kernel/config/netconsole/cmdline0/userdatafor f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done

If auserdata entry is created but no data is written to thevalue file,the entry will be omitted from netconsole messages:

cd /sys/kernel/config/netconsole && mkdir cmdline0cd cmdline0mkdir userdata/fooecho bar > userdata/foo/valuemkdir userdata/qux

Thequx key is omitted since it has no value:

echo "This is a message" > /dev/kmsg12,607,22085407756,-;This is a message foo=bar

Deleteuserdata entries withrmdir:

rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux

Warning

When writing strings to user data values, input is broken up per line inconfigfs store calls and this can cause confusing behavior:

mkdir userdata/testingprintf "val1\nval2" > userdata/testing/value# userdata store value is called twice, first with "val1\n" then "val2"# so "val2" is stored, being the last value storedcat userdata/testing/valueval2

It is recommended to not write user data values with newlines.

Task name auto population in userdata

Inside the netconsole configfs hierarchy, there is a file calledtaskname_enabled under theuserdata directory. This file is used to enableor disable the automatic task name population feature. This featureautomatically populates the current task name that is scheduled in the CPUsneding the message.

To enable task name auto-population:

echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled

When this option is enabled, the netconsole messages will include an additionalline in the userdata field with the formattaskname=<task name>. This allowsthe receiver of the netconsole messages to easily find which application wascurrently scheduled when that message was generated, providing extra contextfor kernel messages and helping to categorize them.

Example:

echo "This is a message" > /dev/kmsg12,607,22085407756,-;This is a message taskname=echo

In this example, the message was generated while “echo” was the currentscheduled process.

Kernel release auto population in userdata

Within the netconsole configfs hierarchy, there is a file namedrelease_enabledlocated in theuserdata directory. This file controls the kernel release(version) auto-population feature, which appends the kernel release informationto userdata dictionary in every message sent.

To enable the release auto-population:

echo 1 > /sys/kernel/config/netconsole/target1/userdata/release_enabled

Example:

echo "This is a message" > /dev/kmsg12,607,22085407756,-;This is a message release=6.14.0-rc6-01219-g3c027fbd941d

Note

This feature provides the same data as the “release prepend” feature.However, in this case, the release information is appended to the userdatadictionary rather than being included in the message header.

CPU number auto population in userdata

Inside the netconsole configfs hierarchy, there is a file calledcpu_nr under theuserdata directory. This file is used to enable or disablethe automatic CPU number population feature. This feature automaticallypopulates the CPU number that is sending the message.

To enable the CPU number auto-population:

echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr

When this option is enabled, the netconsole messages will include an additionalline in the userdata field with the formatcpu=<cpu_number>. This allows thereceiver of the netconsole messages to easily differentiate and demultiplexmessages originating from different CPUs, which is particularly useful whendealing with parallel log output.

Example:

echo "This is a message" > /dev/kmsg12,607,22085407756,-;This is a message cpu=42

In this example, the message was sent by CPU 42.

Note

If the user has set a conflictingcpu key in the userdata dictionary,both keys will be reported, with the kernel-populated entry appearing afterthe user one. For example:

# User-defined CPU entrymkdir -p /sys/kernel/config/netconsole/target1/userdata/cpuecho "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value

Output might look like:

12,607,22085407756,-;This is a message cpu=1 cpu=42    # kernel-populated value

Message ID auto population in userdata

Within the netconsole configfs hierarchy, there is a file namedmsgid_enabledlocated in theuserdata directory. This file controls the message IDauto-population feature, which assigns a numeric id to each message sent to agiven target and appends the ID to userdata dictionary in every message sent.

The message ID is generated using a per-target 32 bit counter that isincremented for every message sent to the target. Note that this counter willeventually wrap around after reaching uint32_t max value, so the message ID isnot globally unique over time. However, it can still be used by the target todetect if messages were dropped before reaching the target by identifying gapsin the sequence of IDs.

It is important to distinguish message IDs from the message <sequnum> field.Some kernel messages may never reach netconsole (for example, due to printkrate limiting). Thus, a gap in <sequnum> cannot be solely relied upon toindicate that a message was dropped during transmission, as it may never havebeen sent via netconsole. The message ID, on the other hand, is only assignedto messages that are actually transmitted via netconsole.

Example:

echo "This is message #1" > /dev/kmsgecho "This is message #2" > /dev/kmsg13,434,54928466,-;This is message #1 msgid=113,435,54934019,-;This is message #2 msgid=2

Extended console:

If ‘+’ is prefixed to the configuration line or “extended” config fileis set to 1, extended console support is enabled. An example bootparam follows:

linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc

Log messages are transmitted with extended metadata header in thefollowing format which is the same as /dev/kmsg:

<level>,<sequnum>,<timestamp>,<contflag>;<message text>

If ‘r’ (release) feature is enabled, the kernel release version isprepended to the start of the message. Example:

6.4.0,6,444,501151268,-;netconsole: network logging started

Non printable characters in <message text> are escaped using “xff”notation. If the message contains optional dictionary, verbatimnewline is used as the delimiter.

If a message doesn’t fit in certain number of bytes (currently 1000),the message is split into multiple fragments by netconsole. Thesefragments are transmitted with “ncfrag” header field added:

ncfrag=<byte-offset>/<total-bytes>

For example, assuming a lot smaller chunk size, a message “the firstchunk, the 2nd chunk.” may be split as follows:

6,416,1758426,-,ncfrag=0/31;the first chunk,6,416,1758426,-,ncfrag=16/31; the 2nd chunk.

Miscellaneous notes:

Warning

the default target ethernet setting uses the broadcastethernet address to send packets, which can cause increased load onother systems on the same ethernet segment.

Tip

some LAN switches may be configured to suppress ethernet broadcastsso it is advised to explicitly specify the remote agents’ MAC addressesfrom the config parameters passed to netconsole.

Tip

to find out the MAC address of, say, 10.0.0.2, you may try using:

ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2

Tip

in case the remote logging agent is on a separate LAN subnet thanthe sender, it is suggested to try specifying the MAC address of thedefault gateway (you may use /sbin/route -n to find it out) as theremote MAC address instead.

Note

the network device (eth1 in the above case) can run any kindof other network traffic, netconsole is not intrusive. Netconsolemight cause slight delays in other traffic if the volume of kernelmessages is high, but should have no other impact.

Note

if you find that the remote logging agent is not receiving orprinting all messages from the sender, it is likely that you have setthe “console_loglevel” parameter (on the sender) to only send highpriority messages to the console. You can change this at runtime using:

dmesg -n 8

or by specifying “debug” on the kernel command line at boot, to sendall kernel messages to the console. A specific value for this parametercan also be set using the “loglevel” kernel boot option. See thedmesg(8) man page andThe kernel’s command-line parametersfor details.

Netconsole was designed to be as instantaneous as possible, toenable the logging of even the most critical kernel bugs. It worksfrom IRQ contexts as well, and does not enable interrupts whilesending packets. Due to these unique needs, configuration cannotbe more automatic, and some fundamental limitations will remain:only IP networks, UDP packets and ethernet devices are supported.