1.3.1 Timestamp Generation¶

Some bits are requests to the stack to try to generate timestamps. Anycombination of them is valid. Changes to these bits apply to newlycreated packets, not to packets already in the stack. As a result, itis possible to selectively request timestamps for a subset of packets(e.g., for sampling) by embedding ansend() call within two setsockoptcalls, one to enable timestamp generation and one to disable it.Timestamps may also be generated for reasons other than beingrequested by a particular socket, such as when receive timestamping isenabled system wide, as explained earlier.

SOF_TIMESTAMPING_RX_HARDWARE:

Request rx timestamps generated by the network adapter.

SOF_TIMESTAMPING_RX_SOFTWARE:

Request rx timestamps when data enters the kernel. These timestampsare generated just after a device driver hands a packet to thekernel receive stack.

SOF_TIMESTAMPING_TX_HARDWARE:

Request tx timestamps generated by the network adapter. This flagcan be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SOFTWARE:

Request tx timestamps when data leaves the kernel. These timestampsare generated in the device driver as close as possible, but alwaysprior to, passing the packet to the network interface. Hence, theyrequire driver support and may not be available for all devices.This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SCHED:

Request tx timestamps prior to entering the packet scheduler. Kerneltransmit latency is, if long, often dominated by queuing delay. Thedifference between this timestamp and one taken atSOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independentof protocol processing. The latency incurred in protocolprocessing, if any, can be computed by subtracting a userspacetimestamp taken immediately beforesend() from this timestamp. Onmachines with virtual devices where a transmitted packet travelsthrough multiple devices and, hence, multiple packet schedulers,a timestamp is generated at each layer. This allows for finegrained measurement of queuing delay. This flag can be enabledvia both socket options and control messages.

SOF_TIMESTAMPING_TX_ACK:

Request tx timestamps when all data in the send buffer has beenacknowledged. This only makes sense for reliable protocols. It iscurrently only implemented for TCP. For that protocol, it mayover-report measurement, because the timestamp is generated when alldata up to and including the buffer atsend() was acknowledged: thecumulative acknowledgment. The mechanism ignores SACK and FACK.This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_COMPLETION:

Request tx timestamps on packet tx completion. The completiontimestamp is generated by the kernel when it receives packet acompletion report from the hardware. Hardware may report multiplepackets at once, and completion timestamps reflect the timing of thereport and not actual tx time. This flag can be enabled via bothsocket options and control messages.

1.3.2 Timestamp Reporting¶

The other three bits control which timestamps will be reported in agenerated control message. Changes to the bits take immediateeffect at the timestamp reporting locations in the stack. Timestampsare only reported for packets that also have the relevant timestampgeneration request set.

SOF_TIMESTAMPING_SOFTWARE:

Report any software timestamps when available.

SOF_TIMESTAMPING_SYS_HARDWARE:

This option is deprecated and ignored.

SOF_TIMESTAMPING_RAW_HARDWARE:

Report hardware timestamps as generated bySOF_TIMESTAMPING_TX_HARDWARE or SOF_TIMESTAMPING_RX_HARDWAREwhen available.

1.3.3 Timestamp Options¶

The interface supports the options

SOF_TIMESTAMPING_OPT_ID:

Generate a unique identifier along with each packet. A process canhave multiple concurrent timestamping requests outstanding. Packetscan be reordered in the transmit path, for instance in the packetscheduler. In that case timestamps will be queued onto the errorqueue out of order from the originalsend() calls. It is not alwayspossible to uniquely match timestamps to the originalsend() callsbased on timestamp order or payload inspection alone, then.

This option associates each packet atsend() with a uniqueidentifier and returns that along with the timestamp. The identifieris derived from a per-socket u32 counter (that wraps). For datagramsockets, the counter increments with each sent packet. For streamsockets, it increments with every byte. For stream sockets, also setSOF_TIMESTAMPING_OPT_ID_TCP, see the section below.

The counter starts at zero. It is initialized the first time thatthe socket option is enabled. It is reset each time the option isenabled after having been disabled. Resetting the counter does notchange the identifiers of existing packets in the system.

This option is implemented only for transmit timestamps. There, thetimestamp is always looped along with astructsock_extended_err.The option modifies field ee_data to pass an id that is uniqueamong all possibly concurrently outstanding timestamp requests forthat socket.

The process can optionally override the default generated ID, bypassing a specific ID with control message SCM_TS_OPT_ID (notsupported for TCP sockets):

struct msghdr *msg;...cmsg                         = CMSG_FIRSTHDR(msg);cmsg->cmsg_level             = SOL_SOCKET;cmsg->cmsg_type              = SCM_TS_OPT_ID;cmsg->cmsg_len               = CMSG_LEN(sizeof(__u32));*((__u32 *) CMSG_DATA(cmsg)) = opt_id;err = sendmsg(fd, msg, 0);

SOF_TIMESTAMPING_OPT_ID_TCP:

Pass this modifier along with SOF_TIMESTAMPING_OPT_ID for new TCPtimestamping applications. SOF_TIMESTAMPING_OPT_ID defines how thecounter increments for stream sockets, but its starting point isnot entirely trivial. This option fixes that.

For stream sockets, if SOF_TIMESTAMPING_OPT_ID is set, this shouldalways be set too. On datagram sockets the option has no effect.

A reasonable expectation is that the counter is reset to zero withthe system call, so that a subsequent write() of N bytes generatesa timestamp with counter N-1. SOF_TIMESTAMPING_OPT_ID_TCPimplements this behavior under all conditions.

SOF_TIMESTAMPING_OPT_ID without modifier often reports the same,especially when the socket option is set when no data is intransmission. If data is being transmitted, it may be off by thelength of the output queue (SIOCOUTQ).

The difference is due to being based on snd_una versus write_seq.snd_una is the offset in the stream acknowledged by the peer. Thisdepends on factors outside of process control, such as network RTT.write_seq is the last byte written by the process. This offset isnot affected by external inputs.

The difference is subtle and unlikely to be noticed when configuredat initial socket creation, when no data is queued or sent. ButSOF_TIMESTAMPING_OPT_ID_TCP behavior is more robust regardless ofwhen the socket option is set.

SOF_TIMESTAMPING_OPT_CMSG:

Supportrecv() cmsg for all timestamped packets. Control messagesare already supported unconditionally on all packets with receivetimestamps and on IPv6 packets with transmit timestamp. This optionextends them to IPv4 packets with transmit timestamp. One use caseis to correlate packets with their egress device, by enabling socketoption IP_PKTINFO simultaneously.

SOF_TIMESTAMPING_OPT_TSONLY:

Applies to transmit timestamps only. Makes the kernel return thetimestamp as a cmsg alongside an empty packet, as opposed toalongside the original packet. This reduces the amount of memorycharged to the socket’s receive budget (SO_RCVBUF) and deliversthe timestamp even if sysctl net.core.tstamp_allow_data is 0.This option disables SOF_TIMESTAMPING_OPT_CMSG.

SOF_TIMESTAMPING_OPT_STATS:

Optional stats that are obtained along with the transmit timestamps.It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When thetransmit timestamp is available, the stats are available in aseparate control message of type SCM_TIMESTAMPING_OPT_STATS, as alist of TLVs (structnlattr) of types. These stats allow theapplication to associate various transport layer stats withthe transmit timestamps, such as how long a certain block ofdata was limited by peer’s receiver window.

SOF_TIMESTAMPING_OPT_PKTINFO:

Enable the SCM_TIMESTAMPING_PKTINFO control message for incomingpackets with hardware timestamps. The message containsstructscm_ts_pktinfo, which supplies the index of the real interface whichreceived the packet and its length at layer 2. A valid (non-zero)interface index will be returned only if CONFIG_NET_RX_BUSY_POLL isenabled and the driver is using NAPI. Thestructcontains also twoother fields, but they are reserved and undefined.

SOF_TIMESTAMPING_OPT_TX_SWHW:

Request both hardware and software timestamps for outgoing packetswhen SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWAREare enabled at the same time. If both timestamps are generated,two separate messages will be looped to the socket’s error queue,each containing just one timestamp.

SOF_TIMESTAMPING_OPT_RX_FILTER:

Filter out spurious receive timestamps: report a receive timestamponly if the matching timestamp generation flag is enabled.

Receive timestamps are generated early in the ingress path, before apacket’s destination socket is known. If any socket enables receivetimestamps, packets for all socket will receive timestamped packets.Including those that request timestamp reporting withSOF_TIMESTAMPING_SOFTWARE and/or SOF_TIMESTAMPING_RAW_HARDWARE, butdo not request receive timestamp generation. This can happen whenrequesting transmit timestamps only.

Receiving spurious timestamps is generally benign. A process canignore the unexpected non-zero value. But it makes behavior subtlydependent on other sockets. This flag isolates the socket for moredeterministic behavior.

New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID todisambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operateregardless of the setting of sysctl net.core.tstamp_allow_data.

An exception is when a process needs additional cmsg data, forinstance SOL_IP/IP_PKTINFO to detect the egress network interface.Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends onhaving access to the contents of the original packet, so cannot becombined with SOF_TIMESTAMPING_OPT_TSONLY.

1.3.4. Enabling timestamps via control messages¶

In addition to socket options, timestamp generation can be requestedper write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).Using this feature, applications can sample timestamps persendmsg()without paying the overhead of enabling and disabling timestamps viasetsockopt:

struct msghdr *msg;...cmsg                         = CMSG_FIRSTHDR(msg);cmsg->cmsg_level             = SOL_SOCKET;cmsg->cmsg_type              = SO_TIMESTAMPING;cmsg->cmsg_len               = CMSG_LEN(sizeof(__u32));*((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |                               SOF_TIMESTAMPING_TX_SOFTWARE |                               SOF_TIMESTAMPING_TX_ACK;err = sendmsg(fd, msg, 0);

The SOF_TIMESTAMPING_TX_* flags set via cmsg will overridethe SOF_TIMESTAMPING_TX_* flags set via setsockopt.

Moreover, applications must still enable timestamp reporting viasetsockopt to receive timestamps:

__u32 val = SOF_TIMESTAMPING_SOFTWARE |            SOF_TIMESTAMPING_OPT_ID /* or any other flag */;err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));

2.1.1 Transmit timestamps with MSG_ERRQUEUE¶

For transmit timestamps the outgoing packet is looped back to thesocket’s error queue with the send timestamp(s) attached. A processreceives the timestamps by callingrecvmsg() with flag MSG_ERRQUEUEset and with a msg_control buffer sufficiently large to receive therelevant metadata structures. The recvmsg call returns the originaloutgoing data packet with two ancillary messages attached.

A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERRembeds astructsock_extended_err. This defines the error type. Fortimestamps, the ee_errno field is ENOMSG. The other ancillary messagewill have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. Thisembeds thestructscm_timestamping.

2.1.2 Receive timestamps¶

On reception, there is no reason to read from the socket error queue.The SCM_TIMESTAMPING ancillary data is sent along with the packet dataon a normalrecvmsg(). Since this is not a socket error, it is notaccompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,the meaning of the three fields instructscm_timestamping isimplicitly defined. ts[0] holds a software timestamp if set, ts[1]is again deprecated and ts[2] holds a hardware timestamp if set.

3.2.1 DSA (Distributed Switch Architecture) switches¶

These are Ethernet switches which have one of their ports connected to an(otherwise completely unaware) host Ethernet interface, and perform the role ofa port multiplier with optional forwarding acceleration features. Each DSAswitch port is visible to the user as a standalone (virtual) network interface,and its network I/O is performed, under the hood, indirectly through the hostinterface (redirecting to the host port on TX, and intercepting frames on RX).

When a DSA switch is attached to a host port, PTP synchronization has tosuffer, since the switch’s variable queuing delay introduces a path delayjitter between the host port and its PTP partner. For this reason, some DSAswitches include a timestamping clock of their own, and have the ability toperform network timestamping on their own MAC, such that path delays onlymeasure wire and PHY propagation latencies. Timestamping DSA switches aresupported in Linux and expose the same ABI as any other network interface (savefor the fact that the DSA interfaces are in fact virtual in terms of networkI/O, they do have their own PHC). It is typical, but not mandatory, for allinterfaces of a DSA switch to share the same PHC.

By design, PTP timestamping with a DSA switch does not need any specialhandling in the driver for the host port it is attached to. However, when thehost port also supports PTP timestamping, DSA will take care of interceptingthe.ndo_eth_ioctl calls towards the host port, and block attempts to enablehardware timestamping on it. This is because the SO_TIMESTAMPING API does notallow the delivery of multiple hardware timestamps for the same packet, soanybody else except for the DSA switch port must be prevented from doing so.

In the generic layer, DSA provides the following infrastructure for PTPtimestamping:

• .port_txtstamp(): a hook called prior to the transmission ofpackets with a hardware TX timestamping request from user space.This is required for two-step timestamping, since the hardwaretimestamp becomes available after the actual MAC transmission, so thedriver must be prepared to correlate the timestamp with the originalpacket so that it can re-enqueue the packet back into the socket’serror queue. To save the packet for when the timestamp becomesavailable, the driver can callskb_clone_sk , save the clone pointerin skb->cb and enqueue a tx skb queue. Typically, a switch will have aPTP TX timestamp register (or sometimes a FIFO) where the timestampbecomes available. In case of a FIFO, the hardware might storekey-value pairs of PTP sequence ID/message type/domain number and theactual timestamp. To perform the correlation correctly between thepackets in a queue waiting for timestamping and the actual timestamps,drivers can use a BPF classifier (ptp_classify_raw) to identifythe PTP transport type, andptp_parse_header to interpret the PTPheader fields. There may be an IRQ that is raised upon thistimestamp’s availability, or the driver might have to poll afterinvokingdev_queue_xmit() towards the host interface.One-step TX timestamping do not require packet cloning, since there isno follow-up message required by the PTP protocol (because theTX timestamp is embedded into the packet by the MAC), and thereforeuser space does not expect the packet annotated with the TX timestampto be re-enqueued into its socket’s error queue.

• .port_rxtstamp(): On RX, the BPF classifier is run by DSA toidentify PTP event messages (any other packets, including PTP generalmessages, are not timestamped). The original (and only) timestampableskb is provided to the driver, for it to annotate it with a timestamp,if that is immediately available, or defer to later. On reception,timestamps might either be available in-band (through metadata in theDSA header, or attached in other ways to the packet), or out-of-band(through another RX timestamping FIFO). Deferral on RX is typicallynecessary when retrieving the timestamp needs a sleepable context. Inthat case, it is the responsibility of the DSA driver to callnetif_rx() on the freshly timestamped skb.

3.2.2 Ethernet PHYs¶

These are devices that typically fulfill a Layer 1 role in the network stack,hence they do not have a representation in terms of a network interface as DSAswitches do. However, PHYs may be able to detect and timestamp PTP packets, forperformance reasons: timestamps taken as close as possible to the wire have thepotential to yield a more stable and precise synchronization.

A PHY driver that supports PTP timestamping must create astructmii_timestamper and add a pointer to it inphydev->mii_ts. The presenceof this pointer will be checked by the networking stack.

Since PHYs do not have network interface representations, the timestamping andethtool ioctl operations for them need to be mediated by their respective MACdriver. Therefore, as opposed to DSA switches, modifications need to be doneto each individual MAC driver for PHY timestamping support. This entails:

• Checking, in.ndo_eth_ioctl, whetherphy_has_hwtstamp(netdev->phydev)is true or not. If it is, then the MAC driver should not process this requestbut instead pass it on to the PHY usingphy_mii_ioctl().

• On RX, special intervention may or may not be needed, depending on thefunction used to deliver skb’s up the network stack. In the case of plainnetif_rx() and similar, MAC drivers must check whetherskb_defer_rx_timestamp(skb) is necessary or not - and if it is, don’tcallnetif_rx() at all. IfCONFIG_NETWORK_PHY_TIMESTAMPING isenabled, andskb->dev->phydev->mii_ts exists, its.rxtstamp() hookwill be called now, to determine, using logic very similar to DSA, whetherdeferral for RX timestamping is necessary. Again like DSA, it becomes theresponsibility of the PHY driver to send the packet up the stack when thetimestamp is available.

  For other skb receive functions, such asnapi_gro_receive andnetif_receive_skb, the stack automatically checks whetherskb_defer_rx_timestamp() is necessary, so this check is not needed insidethe driver.

• On TX, again, special intervention might or might not be needed. Thefunction that calls themii_ts->txtstamp() hook is namedskb_clone_tx_timestamp(). This function can either be called directly(case in which explicit MAC driver support is indeed needed), but thefunction also piggybacks from theskb_tx_timestamp() call, which many MACdrivers already perform for software timestamping purposes. Therefore, if aMAC supports software timestamping, it does not need to do anything furtherat this stage.

3.2.3 MII bus snooping devices¶

These perform the same role as timestamping Ethernet PHYs, save for the factthat they are discrete devices and can therefore be used in conjunction withany PHY even if it doesn’t support timestamping. In Linux, they arediscoverable and attachable to astructphy_device through Device Tree, andfor the rest, they use the same mii_ts infrastructure as those. SeeDocumentation/devicetree/bindings/ptp/timestamper.txt for more details.

3.2.4 Other caveats for MAC drivers¶

The use of stacked PHCs may uncover MAC driver bugs which were impossible totrigger without them. One example has to do with this line of code, alreadypresented earlier:

skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;

Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHYdriver or a MII bus snooping device driver, should set this flag.But a MAC driver that is unaware of PHC stacking might get tripped up bysomebody other than itself setting this flag, and deliver a duplicatetimestamp.For example, a typical driver design for TX timestamping might be to split thetransmission part into 2 portions:

1. “TX”: checks whether PTP timestamping has been previously enabled throughthe.ndo_eth_ioctl (”priv->hwtstamp_tx_enabled==true”) and thecurrent skb requires a TX timestamp (”skb_shinfo(skb)->tx_flags&SKBTX_HW_TSTAMP”). If this is true, it sets the“skb_shinfo(skb)->tx_flags|=SKBTX_IN_PROGRESS” flag. Note: asdescribed above, in the case of a stacked PHC system, this condition shouldnever trigger, as this MAC is certainly not the outermost PHC. But this isnot where the typical issue is. Transmission proceeds with this packet.

2. “TX confirmation”: Transmission has finished. The driver checks whether itis necessary to collect any TX timestamp for it. Here is where the typicalissues are: the MAC driver takes a shortcut and only checks whether“skb_shinfo(skb)->tx_flags&SKBTX_IN_PROGRESS” was set. With a stackedPHC system, this is incorrect because this MAC driver is not the only entityin the TX data path who could have enabled SKBTX_IN_PROGRESS in the firstplace.

The correct solution for this problem is for MAC drivers to have a compoundcheck in their “TX confirmation” portion, not only for“skb_shinfo(skb)->tx_flags&SKBTX_IN_PROGRESS”, but also for“priv->hwtstamp_tx_enabled==true”. Because the rest of the system ensuresthat PTP timestamping is not enabled for anything other than the outermost PHC,this enhanced check will avoid delivering a duplicated TX timestamp to userspace.