Interface statistics

Overview

This document is a guide to Linux network interface statistics.

There are three main sources of interface statistics in Linux:

  • standard interface statistics based onstructrtnl_link_stats64;

  • protocol-specific statistics; and

  • driver-defined statistics available via ethtool.

Standard interface statistics

There are multiple interfaces to reach the standard statistics.Most commonly used is theip command fromiproute2:

$ ip -s -s link show dev ens4u1u16: ens4u1u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000  link/ether 48:2a:e3:4c:b1:d1 brd ff:ff:ff:ff:ff:ff  RX: bytes  packets  errors  dropped overrun mcast  74327665117 69016965 0       0       0       0  RX errors: length   crc     frame   fifo    missed             0        0       0       0       0  TX: bytes  packets  errors  dropped carrier collsns  21405556176 44608960 0       0       0       0  TX errors: aborted  fifo   window heartbeat transns             0        0       0       0       128  altname enp58s0u1u1

Note that-s has been specified twice to see all members ofstructrtnl_link_stats64.If-s is specified once the detailed errors won’t be shown.

ip supports JSON formatting via the-j option.

Queue statistics

Queue statistics are accessible via the netdev netlink family.

Currently no widely distributed CLI exists to access those statistics.Kernel development tools (ynl) can be used to experiment with them,seeUsing Netlink protocol specifications.

Protocol-specific statistics

Protocol-specific statistics are exposed via relevant interfaces,the same interfaces as are used to configure them.

ethtool

Ethtool exposes common low-level statistics.All the standard statistics are expected to be maintainedby the device, not the driver (as opposed to driver-defined statsdescribed in the next section which mix software and hardware stats).For devices which contain unmanagedswitches (e.g. legacy SR-IOV or multi-host NICs) the events countedmay not pertain exclusively to the packets destined tothe local host interface. In other words the events maybe counted at the network port (MAC/PHY blocks) without separationfor different host side (PCIe) devices. Such ambiguity must notbe present when internal switch is managed by Linux (so calledswitchdev mode for NICs).

Standard ethtool statistics can be accessed via the interfaces usedfor configuration. For example ethtool interface usedto configure pause frames can report corresponding hardware counters:

$ ethtool --include-statistics -a eth0Pause parameters for eth0:Autonegotiate:        onRX:                   onTX:                   onStatistics:  tx_pause_frames: 1  rx_pause_frames: 1

General Ethernet statistics not associated with any particularfunctionality are exposed viaethtool-S$ifc by specifyingthe--groups parameter:

$ ethtool -S eth0 --groups eth-phy eth-mac eth-ctrl rmonStats for eth0:eth-phy-SymbolErrorDuringCarrier: 0eth-mac-FramesTransmittedOK: 1eth-mac-FrameTooLongErrors: 1eth-ctrl-MACControlFramesTransmitted: 1eth-ctrl-MACControlFramesReceived: 0eth-ctrl-UnsupportedOpcodesReceived: 1rmon-etherStatsUndersizePkts: 1rmon-etherStatsJabbers: 0rmon-rx-etherStatsPkts64Octets: 1rmon-rx-etherStatsPkts65to127Octets: 0rmon-rx-etherStatsPkts128to255Octets: 0rmon-tx-etherStatsPkts64Octets: 2rmon-tx-etherStatsPkts65to127Octets: 3rmon-tx-etherStatsPkts128to255Octets: 0

Driver-defined statistics

Driver-defined ethtool statistics can be dumped usingethtool -S $ifc, e.g.:

$ ethtool -S ens4u1u1NIC statistics:   tx_single_collisions: 0   tx_multi_collisions: 0

uAPIs

procfs

The historical/proc/net/dev text interface gives access to the listof interfaces as well as their statistics.

Note that even though this interface is usingstructrtnl_link_stats64internally it combines some of the fields.

sysfs

Each device directory in sysfs contains astatistics directory (e.g./sys/class/net/lo/statistics/) with files corresponding tomembers ofstructrtnl_link_stats64.

This simple interface is convenient especially in constrained/embeddedenvironments without access to tools. However, it’s inefficient whenreading multiple stats as it internally performs a full dump ofstructrtnl_link_stats64and reports only the stat corresponding to the accessed file.

Sysfs files are documented inABI file testing/sysfs-class-net-statistics.

netlink

rtnetlink (NETLINK_ROUTE) is the preferred method of accessingstructrtnl_link_stats64 stats.

Statistics are reported both in the responses to link informationrequests (RTM_GETLINK) and statistic requests (RTM_GETSTATS,whenIFLA_STATS_LINK_64 bit is set in the.filter_mask of the request).

netdev (netlink)

netdev generic netlink family allows accessing page pool and per queuestatistics.

ethtool

Ethtool IOCTL interface allows drivers to report implementationspecific statistics. Historically it has also been used to reportstatistics for which other APIs did not exist, like per-device-queuestatistics, or standard-based statistics (e.g. RFC 2863).

Statistics and their string identifiers are retrieved separately.Identifiers viaETHTOOL_GSTRINGS withstring_set set toETH_SS_STATS,and values viaETHTOOL_GSTATS. User space should useETHTOOL_GDRVINFOto retrieve the number of statistics (.n_stats).

ethtool-netlink

Ethtool netlink is a replacement for the older IOCTL interface.

Protocol-related statistics can be requested in get commands by settingtheETHTOOL_FLAG_STATS flag inETHTOOL_A_HEADER_FLAGS. Currentlystatistics are supported in the following commands:

  • ETHTOOL_MSG_FEC_GET

  • ETHTOOL_MSG_LINKSTATE_GET

  • ETHTOOL_MSG_MM_GET

  • ETHTOOL_MSG_PAUSE_GET

  • ETHTOOL_MSG_TSINFO_GET

debugfs

Some drivers expose extra statistics viadebugfs.

struct rtnl_link_stats64

structrtnl_link_stats64

The main device statistics structure.

Definition:

struct rtnl_link_stats64 {    __u64 rx_packets;    __u64 tx_packets;    __u64 rx_bytes;    __u64 tx_bytes;    __u64 rx_errors;    __u64 tx_errors;    __u64 rx_dropped;    __u64 tx_dropped;    __u64 multicast;    __u64 collisions;    __u64 rx_length_errors;    __u64 rx_over_errors;    __u64 rx_crc_errors;    __u64 rx_frame_errors;    __u64 rx_fifo_errors;    __u64 rx_missed_errors;    __u64 tx_aborted_errors;    __u64 tx_carrier_errors;    __u64 tx_fifo_errors;    __u64 tx_heartbeat_errors;    __u64 tx_window_errors;    __u64 rx_compressed;    __u64 tx_compressed;    __u64 rx_nohandler;    __u64 rx_otherhost_dropped;};

Members

rx_packets

Number of good packets received by the interface.For hardware interfaces counts all good packets received from the deviceby the host, including packets which host had to drop at various stagesof processing (even in the driver).

tx_packets

Number of packets successfully transmitted.For hardware interfaces counts packets which host was able to successfullyhand over to the device, which does not necessarily mean that packetshad been successfully transmitted out of the device, only that deviceacknowledged it copied them out of host memory.

rx_bytes

Number of good received bytes, corresponding torx_packets.

tx_bytes

Number of good transmitted bytes, corresponding totx_packets.

rx_errors

Total number of bad packets received on this network device.This counter must include events counted byrx_length_errors,rx_crc_errors,rx_frame_errors and other errors not otherwisecounted.

tx_errors

Total number of transmit problems.This counter must include events counter bytx_aborted_errors,tx_carrier_errors,tx_fifo_errors,tx_heartbeat_errors,tx_window_errors and other errors not otherwise counted.

rx_dropped

Number of packets received but not processed,e.g. due to lack of resources or unsupported protocol.For hardware interfaces this counter may include packets discardeddue to L2 address filtering but should not include packets droppedby the device due to buffer exhaustion which are counted separately inrx_missed_errors (since procfs folds those two counters together).

tx_dropped

Number of packets dropped on their way to transmission,e.g. due to lack of resources.

multicast

Multicast packets received.For hardware interfaces this statistic is commonly calculatedat the device level (unlikerx_packets) and therefore may includepackets which did not reach the host.

collisions

Number of collisions during packet transmissions.

rx_length_errors

Number of packets dropped due to invalid length.Part of aggregate “frame” errors in/proc/net/dev.

rx_over_errors

Receiver FIFO overflow event counter.

rx_crc_errors

Number of packets received with a CRC error.Part of aggregate “frame” errors in/proc/net/dev.

rx_frame_errors

Receiver frame alignment errors.Part of aggregate “frame” errors in/proc/net/dev.

rx_fifo_errors

Receiver FIFO error counter.

rx_missed_errors

Count of packets missed by the host.Folded into the “drop” counter in/proc/net/dev.

tx_aborted_errors

Part of aggregate “carrier” errors in/proc/net/dev.For IEEE 802.3 devices capable of half-duplex operation this countermust be equivalent to:

tx_carrier_errors

Number of frame transmission errors due to lossof carrier during transmission.Part of aggregate “carrier” errors in/proc/net/dev.

tx_fifo_errors

Number of frame transmission errors due to deviceFIFO underrun / underflow. This condition occurs when the devicebegins transmission of a frame but is unable to deliver theentire frame to the transmitter in time for transmission.Part of aggregate “carrier” errors in/proc/net/dev.

tx_heartbeat_errors

Number of Heartbeat / SQE Test errors forold half-duplex Ethernet.Part of aggregate “carrier” errors in/proc/net/dev.

tx_window_errors

Number of frame transmission errors dueto late collisions (for Ethernet - after the first 64B of transmission).Part of aggregate “carrier” errors in/proc/net/dev.

rx_compressed

Number of correctly received compressed packets.This counters is only meaningful for interfaces which supportpacket compression (e.g. CSLIP, PPP).

tx_compressed

Number of transmitted compressed packets.This counters is only meaningful for interfaces which supportpacket compression (e.g. CSLIP, PPP).

rx_nohandler

Number of packets received on the interfacebut dropped by the networking stack because the device isnot designated to receive packets (e.g. backup link in a bond).

rx_otherhost_dropped

Number of packets dropped due to mismatchin destination MAC address.

Description

For IEEE 802.3 devices should count the length of Ethernet Framesexcluding the FCS.

For IEEE 802.3 devices should count the length of Ethernet Framesexcluding the FCS.

For IEEE 802.3 devices this counter may be equivalent to:

  • 30.3.1.1.21 aMulticastFramesReceivedOK

For IEEE 802.3 devices this counter should be equivalent to a sumof the following attributes:

  • 30.3.1.1.23 aInRangeLengthErrors

  • 30.3.1.1.24 aOutOfRangeLengthField

  • 30.3.1.1.25 aFrameTooLongErrors

Historically the count of overflow events. Such events may bereported in the receive descriptors or via interrupts, and maynot correspond one-to-one with dropped packets.

The recommended interpretation for high speed interfaces is -number of packets dropped because they did not fit into buffersprovided by the host, e.g. packets larger than MTU or next bufferin the ring was not available for a scatter transfer.

Part of aggregate “frame” errors in/proc/net/dev.

This statistics was historically used interchangeably withrx_fifo_errors.

This statistic corresponds to hardware events and is not commonly usedon software devices.

For IEEE 802.3 devices this counter must be equivalent to:

  • 30.3.1.1.6 aFrameCheckSequenceErrors

For IEEE 802.3 devices this counter should be equivalent to:

  • 30.3.1.1.7 aAlignmentErrors

Historically the count of overflow events. Those events may bereported in the receive descriptors or via interrupts, and maynot correspond one-to-one with dropped packets.

This statistics was used interchangeably withrx_over_errors.Not recommended for use in drivers for high speed interfaces.

This statistic is used on software devices, e.g. to count softwarepacket queue overflow (can) or sequencing errors (GRE).

Counts number of packets dropped by the device due to lackof buffer space. This usually indicates that the host interfaceis slower than the network interface, or host is not keeping upwith the receive packet rate.

This statistic corresponds to hardware events and is not usedon software devices.

  • 30.3.1.1.11 aFramesAbortedDueToXSColls

High speed interfaces may use this counter as a general devicediscard counter.

For IEEE 802.3 devices this counter must be equivalent to:

  • 30.3.1.1.13 aCarrierSenseErrors

For IEEE 802.3 devices possibly equivalent to:

  • 30.3.2.1.4 aSQETestErrors

For IEEE 802.3 devices this counter must be equivalent to:

  • 30.3.1.1.10 aLateCollisions

Notes for driver authors

Drivers should report all statistics which have a matching member instructrtnl_link_stats64 exclusivelyvia.ndo_get_stats64. Reporting such standard stats via ethtoolor debugfs will not be accepted.

Drivers must ensure best possible compliance withstructrtnl_link_stats64.Please note for example that detailed error statistics must beadded into the generalrx_error /tx_error counters.

The.ndo_get_stats64 callback can not sleep because of accessesvia/proc/net/dev. If driver may sleep when retrieving the statisticsfrom the device it should do so periodically asynchronously and only returna recent copy from.ndo_get_stats64. Ethtool interrupt coalescing interfaceallows setting the frequency of refreshing statistics, if needed.

Retrieving ethtool statistics is a multi-syscall process, drivers are advisedto keep the number of statistics constant to avoid race conditions withuser space trying to read them.

Statistics must persist across routine operations like bringing the interfacedown and up.

Kernel-internal data structures

The following structures are internal to the kernel, their members aretranslated to netlink attributes when dumped. Drivers must not overwritethe statistics they don’t report with 0.