1.3.1 Timestamp Generation¶

Some bits are requests to the stack to try to generate timestamps. Anycombination of them is valid. Changes to these bits apply to newlycreated packets, not to packets already in the stack. As a result, itis possible to selectively request timestamps for a subset of packets(e.g., for sampling) by embedding an send() call within two setsockoptcalls, one to enable timestamp generation and one to disable it.Timestamps may also be generated for reasons other than beingrequested by a particular socket, such as when receive timestamping isenabled system wide, as explained earlier.

SOF_TIMESTAMPING_RX_HARDWARE:

Request rx timestamps generated by the network adapter.

SOF_TIMESTAMPING_RX_SOFTWARE:

Request rx timestamps when data enters the kernel. These timestampsare generated just after a device driver hands a packet to thekernel receive stack.

SOF_TIMESTAMPING_TX_HARDWARE:

Request tx timestamps generated by the network adapter. This flagcan be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SOFTWARE:

Request tx timestamps when data leaves the kernel. These timestampsare generated in the device driver as close as possible, but alwaysprior to, passing the packet to the network interface. Hence, theyrequire driver support and may not be available for all devices.This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SCHED:

Request tx timestamps prior to entering the packet scheduler. Kerneltransmit latency is, if long, often dominated by queuing delay. Thedifference between this timestamp and one taken atSOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independentof protocol processing. The latency incurred in protocolprocessing, if any, can be computed by subtracting a userspacetimestamp taken immediately before send() from this timestamp. Onmachines with virtual devices where a transmitted packet travelsthrough multiple devices and, hence, multiple packet schedulers,a timestamp is generated at each layer. This allows for finegrained measurement of queuing delay. This flag can be enabledvia both socket options and control messages.

SOF_TIMESTAMPING_TX_ACK:

Request tx timestamps when all data in the send buffer has beenacknowledged. This only makes sense for reliable protocols. It iscurrently only implemented for TCP. For that protocol, it mayover-report measurement, because the timestamp is generated when alldata up to and including the buffer at send() was acknowledged: thecumulative acknowledgment. The mechanism ignores SACK and FACK.This flag can be enabled via both socket options and control messages.

1.3.2 Timestamp Reporting¶

The other three bits control which timestamps will be reported in agenerated control message. Changes to the bits take immediateeffect at the timestamp reporting locations in the stack. Timestampsare only reported for packets that also have the relevant timestampgeneration request set.

SOF_TIMESTAMPING_SOFTWARE:

Report any software timestamps when available.

SOF_TIMESTAMPING_SYS_HARDWARE:

This option is deprecated and ignored.

SOF_TIMESTAMPING_RAW_HARDWARE:

Report hardware timestamps as generated bySOF_TIMESTAMPING_TX_HARDWARE when available.

1.3.3 Timestamp Options¶

The interface supports the options

SOF_TIMESTAMPING_OPT_ID:

Generate a unique identifier along with each packet. A process canhave multiple concurrent timestamping requests outstanding. Packetscan be reordered in the transmit path, for instance in the packetscheduler. In that case timestamps will be queued onto the errorqueue out of order from the original send() calls. It is not alwayspossible to uniquely match timestamps to the original send() callsbased on timestamp order or payload inspection alone, then.

This option associates each packet at send() with a uniqueidentifier and returns that along with the timestamp. The identifieris derived from a per-socket u32 counter (that wraps). For datagramsockets, the counter increments with each sent packet. For streamsockets, it increments with every byte.

The counter starts at zero. It is initialized the first time thatthe socket option is enabled. It is reset each time the option isenabled after having been disabled. Resetting the counter does notchange the identifiers of existing packets in the system.

This option is implemented only for transmit timestamps. There, thetimestamp is always looped along with a struct sock_extended_err.The option modifies field ee_data to pass an id that is uniqueamong all possibly concurrently outstanding timestamp requests forthat socket.

SOF_TIMESTAMPING_OPT_CMSG:

Support recv() cmsg for all timestamped packets. Control messagesare already supported unconditionally on all packets with receivetimestamps and on IPv6 packets with transmit timestamp. This optionextends them to IPv4 packets with transmit timestamp. One use caseis to correlate packets with their egress device, by enabling socketoption IP_PKTINFO simultaneously.

SOF_TIMESTAMPING_OPT_TSONLY:

Applies to transmit timestamps only. Makes the kernel return thetimestamp as a cmsg alongside an empty packet, as opposed toalongside the original packet. This reduces the amount of memorycharged to the socket’s receive budget (SO_RCVBUF) and deliversthe timestamp even if sysctl net.core.tstamp_allow_data is 0.This option disables SOF_TIMESTAMPING_OPT_CMSG.

SOF_TIMESTAMPING_OPT_STATS:

Optional stats that are obtained along with the transmit timestamps.It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When thetransmit timestamp is available, the stats are available in aseparate control message of type SCM_TIMESTAMPING_OPT_STATS, as alist of TLVs (struct nlattr) of types. These stats allow theapplication to associate various transport layer stats withthe transmit timestamps, such as how long a certain block ofdata was limited by peer’s receiver window.

SOF_TIMESTAMPING_OPT_PKTINFO:

Enable the SCM_TIMESTAMPING_PKTINFO control message for incomingpackets with hardware timestamps. The message contains structscm_ts_pktinfo, which supplies the index of the real interface whichreceived the packet and its length at layer 2. A valid (non-zero)interface index will be returned only if CONFIG_NET_RX_BUSY_POLL isenabled and the driver is using NAPI. The struct contains also twoother fields, but they are reserved and undefined.

SOF_TIMESTAMPING_OPT_TX_SWHW:

Request both hardware and software timestamps for outgoing packetswhen SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWAREare enabled at the same time. If both timestamps are generated,two separate messages will be looped to the socket’s error queue,each containing just one timestamp.

New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID todisambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operateregardless of the setting of sysctl net.core.tstamp_allow_data.

An exception is when a process needs additional cmsg data, forinstance SOL_IP/IP_PKTINFO to detect the egress network interface.Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends onhaving access to the contents of the original packet, so cannot becombined with SOF_TIMESTAMPING_OPT_TSONLY.

1.3.4. Enabling timestamps via control messages¶

In addition to socket options, timestamp generation can be requestedper write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).Using this feature, applications can sample timestamps per sendmsg()without paying the overhead of enabling and disabling timestamps viasetsockopt:

struct msghdr *msg;...cmsg                         = CMSG_FIRSTHDR(msg);cmsg->cmsg_level             = SOL_SOCKET;cmsg->cmsg_type              = SO_TIMESTAMPING;cmsg->cmsg_len               = CMSG_LEN(sizeof(__u32));*((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |                               SOF_TIMESTAMPING_TX_SOFTWARE |                               SOF_TIMESTAMPING_TX_ACK;err = sendmsg(fd, msg, 0);

The SOF_TIMESTAMPING_TX_* flags set via cmsg will overridethe SOF_TIMESTAMPING_TX_* flags set via setsockopt.

Moreover, applications must still enable timestamp reporting viasetsockopt to receive timestamps:

__u32 val = SOF_TIMESTAMPING_SOFTWARE |            SOF_TIMESTAMPING_OPT_ID /* or any other flag */;err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));

2.1.1 Transmit timestamps with MSG_ERRQUEUE¶

For transmit timestamps the outgoing packet is looped back to thesocket’s error queue with the send timestamp(s) attached. A processreceives the timestamps by calling recvmsg() with flag MSG_ERRQUEUEset and with a msg_control buffer sufficiently large to receive therelevant metadata structures. The recvmsg call returns the originaloutgoing data packet with two ancillary messages attached.

A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERRembeds a struct sock_extended_err. This defines the error type. Fortimestamps, the ee_errno field is ENOMSG. The other ancillary messagewill have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. Thisembeds the struct scm_timestamping.

2.1.2 Receive timestamps¶

On reception, there is no reason to read from the socket error queue.The SCM_TIMESTAMPING ancillary data is sent along with the packet dataon a normal recvmsg(). Since this is not a socket error, it is notaccompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,the meaning of the three fields in struct scm_timestamping isimplicitly defined. ts[0] holds a software timestamp if set, ts[1]is again deprecated and ts[2] holds a hardware timestamp if set.

struct hwtstamp_config {        int flags;      /* no flags defined right now, must be zero */        int tx_type;    /* HWTSTAMP_TX_* */        int rx_filter;  /* HWTSTAMP_FILTER_* */};

/* possible values for hwtstamp_config->tx_type */enum {        /*        * no outgoing packet will need hardware time stamping;        * should a packet arrive which asks for it, no hardware        * time stamping will be done        */        HWTSTAMP_TX_OFF,        /*        * enables hardware time stamping for outgoing packets;        * the sender of the packet decides which are to be        * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE        * before sending the packet        */        HWTSTAMP_TX_ON,};/* possible values for hwtstamp_config->rx_filter */enum {        /* time stamp no incoming packet at all */        HWTSTAMP_FILTER_NONE,        /* time stamp any incoming packet */        HWTSTAMP_FILTER_ALL,        /* return value: time stamp all packets requested plus some others */        HWTSTAMP_FILTER_SOME,        /* PTP v1, UDP, any kind of event packet */        HWTSTAMP_FILTER_PTP_V1_L4_EVENT,        /* for the complete list of values, please check        * the include file include/uapi/linux/net_tstamp.h        */};

3.2.1 DSA (Distributed Switch Architecture) switches¶

These are Ethernet switches which have one of their ports connected to an(otherwise completely unaware) host Ethernet interface, and perform the role ofa port multiplier with optional forwarding acceleration features. Each DSAswitch port is visible to the user as a standalone (virtual) network interface,and its network I/O is performed, under the hood, indirectly through the hostinterface (redirecting to the host port on TX, and intercepting frames on RX).

When a DSA switch is attached to a host port, PTP synchronization has tosuffer, since the switch’s variable queuing delay introduces a path delayjitter between the host port and its PTP partner. For this reason, some DSAswitches include a timestamping clock of their own, and have the ability toperform network timestamping on their own MAC, such that path delays onlymeasure wire and PHY propagation latencies. Timestamping DSA switches aresupported in Linux and expose the same ABI as any other network interface (savefor the fact that the DSA interfaces are in fact virtual in terms of networkI/O, they do have their own PHC). It is typical, but not mandatory, for allinterfaces of a DSA switch to share the same PHC.

By design, PTP timestamping with a DSA switch does not need any specialhandling in the driver for the host port it is attached to. However, when thehost port also supports PTP timestamping, DSA will take care of interceptingthe.ndo_do_ioctl calls towards the host port, and block attempts to enablehardware timestamping on it. This is because the SO_TIMESTAMPING API does notallow the delivery of multiple hardware timestamps for the same packet, soanybody else except for the DSA switch port must be prevented from doing so.

In code, DSA provides for most of the infrastructure for timestamping already,in generic code: a BPF classifier (ptp_classify_raw) is used to identifyPTP event messages (any other packets, including PTP general messages, are nottimestamped), and provides two hooks to drivers:

• .port_txtstamp(): The driver is passed a clone of the timestampable skbto be transmitted, before actually transmitting it. Typically, a switch willhave a PTP TX timestamp register (or sometimes a FIFO) where the timestampbecomes available. There may be an IRQ that is raised upon this timestamp’savailability, or the driver might have to poll after invokingdev_queue_xmit() towards the host interface. Either way, in the.port_txtstamp() method, the driver only needs to save the clone forlater use (when the timestamp becomes available). Each skb is annotated witha pointer to its clone, inDSA_SKB_CB(skb)->clone, to ease the driver’sjob of keeping track of which clone belongs to which skb.
• .port_rxtstamp(): The original (and only) timestampable skb is providedto the driver, for it to annotate it with a timestamp, if that is immediatelyavailable, or defer to later. On reception, timestamps might either beavailable in-band (through metadata in the DSA header, or attached in otherways to the packet), or out-of-band (through another RX timestamping FIFO).Deferral on RX is typically necessary when retrieving the timestamp needs asleepable context. In that case, it is the responsibility of the DSA driverto callnetif_rx_ni() on the freshly timestamped skb.

3.2.2 Ethernet PHYs¶

These are devices that typically fulfill a Layer 1 role in the network stack,hence they do not have a representation in terms of a network interface as DSAswitches do. However, PHYs may be able to detect and timestamp PTP packets, forperformance reasons: timestamps taken as close as possible to the wire have thepotential to yield a more stable and precise synchronization.

A PHY driver that supports PTP timestamping must create astructmii_timestamper and add a pointer to it inphydev->mii_ts. The presenceof this pointer will be checked by the networking stack.

Since PHYs do not have network interface representations, the timestamping andethtool ioctl operations for them need to be mediated by their respective MACdriver. Therefore, as opposed to DSA switches, modifications need to be doneto each individual MAC driver for PHY timestamping support. This entails:

• Checking, in.ndo_do_ioctl, whetherphy_has_hwtstamp(netdev->phydev)is true or not. If it is, then the MAC driver should not process this requestbut instead pass it on to the PHY usingphy_mii_ioctl().

• On RX, special intervention may or may not be needed, depending on thefunction used to deliver skb’s up the network stack. In the case of plainnetif_rx() and similar, MAC drivers must check whetherskb_defer_rx_timestamp(skb) is necessary or not - and if it is, don’tcallnetif_rx() at all. IfCONFIG_NETWORK_PHY_TIMESTAMPING isenabled, andskb->dev->phydev->mii_ts exists, its.rxtstamp() hookwill be called now, to determine, using logic very similar to DSA, whetherdeferral for RX timestamping is necessary. Again like DSA, it becomes theresponsibility of the PHY driver to send the packet up the stack when thetimestamp is available.

  For other skb receive functions, such asnapi_gro_receive andnetif_receive_skb, the stack automatically checks whetherskb_defer_rx_timestamp() is necessary, so this check is not needed insidethe driver.

• On TX, again, special intervention might or might not be needed. Thefunction that calls themii_ts->txtstamp() hook is namedskb_clone_tx_timestamp(). This function can either be called directly(case in which explicit MAC driver support is indeed needed), but thefunction also piggybacks from theskb_tx_timestamp() call, which many MACdrivers already perform for software timestamping purposes. Therefore, if aMAC supports software timestamping, it does not need to do anything furtherat this stage.

3.2.3 MII bus snooping devices¶

These perform the same role as timestamping Ethernet PHYs, save for the factthat they are discrete devices and can therefore be used in conjunction withany PHY even if it doesn’t support timestamping. In Linux, they arediscoverable and attachable to astructphy_device through Device Tree, andfor the rest, they use the same mii_ts infrastructure as those. SeeDocumentation/devicetree/bindings/ptp/timestamper.txt for more details.

3.2.4 Other caveats for MAC drivers¶

Stacked PHCs, especially DSA (but not only) - since that doesn’t require anymodification to MAC drivers, so it is more difficult to ensure correctness ofall possible code paths - is that they uncover bugs which were impossible totrigger before the existence of stacked PTP clocks. One example has to do withthis line of code, already presented earlier:

skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;

Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHYdriver or a MII bus snooping device driver, should set this flag.But a MAC driver that is unaware of PHC stacking might get tripped up bysomebody other than itself setting this flag, and deliver a duplicatetimestamp.For example, a typical driver design for TX timestamping might be to split thetransmission part into 2 portions:

1. “TX”: checks whether PTP timestamping has been previously enabled throughthe.ndo_do_ioctl (“priv->hwtstamp_tx_enabled==true”) and thecurrent skb requires a TX timestamp (“skb_shinfo(skb)->tx_flags&SKBTX_HW_TSTAMP”). If this is true, it sets the“skb_shinfo(skb)->tx_flags|=SKBTX_IN_PROGRESS” flag. Note: asdescribed above, in the case of a stacked PHC system, this condition shouldnever trigger, as this MAC is certainly not the outermost PHC. But this isnot where the typical issue is. Transmission proceeds with this packet.
2. “TX confirmation”: Transmission has finished. The driver checks whether itis necessary to collect any TX timestamp for it. Here is where the typicalissues are: the MAC driver takes a shortcut and only checks whether“skb_shinfo(skb)->tx_flags&SKBTX_IN_PROGRESS” was set. With a stackedPHC system, this is incorrect because this MAC driver is not the only entityin the TX data path who could have enabled SKBTX_IN_PROGRESS in the firstplace.

The correct solution for this problem is for MAC drivers to have a compoundcheck in their “TX confirmation” portion, not only for“skb_shinfo(skb)->tx_flags&SKBTX_IN_PROGRESS”, but also for“priv->hwtstamp_tx_enabled==true”. Because the rest of the system ensuresthat PTP timestamping is not enabled for anything other than the outermost PHC,this enhanced check will avoid delivering a duplicated TX timestamp to userspace.