1. Introduction¶

>> The IDs of MKTs MUST NOT overlap where their TCP connectionidentifiers overlap.

It is presumed that an MKT affecting a particular connection cannotbe destroyed during an active connection -- or, equivalently, thatits parameters are copied to an area local to the connection (i.e.,instantiated) and so changes would affect only new connections.

Deciding when to start using a key is a performance issue. Decidingwhen to remove an MKT is a security issue. Invalid MKTs are expectedto be removed. TCP-AO provides no mechanism to coordinate their removal,as we consider this a key management operation.

The only way to avoid reuse of previously used MKTs is to remove the MKTwhen it is no longer considered permitted.

If the segment is a SYN, then this is the first segment of a newconnection. Find the matching MKT for this segment, using the segment'ssocket pair and its TCP-AO KeyID, matched against the MKT's TCP connectionidentifier and the MKT's RecvID.   i. If there is no matching MKT, remove TCP-AO from the segment.      Proceed with further TCP handling of the segment.      NOTE: this presumes that connections that do not match any MKT      should be silently accepted, as noted in Section 7.3.

>> A TCP-AO implementation MUST allow for configuration of the behaviorof segments with TCP-AO but that do not match an MKT. The initial defaultof this configuration SHOULD be to silently accept such connections.If this is not the desired case, an MKT can be included to match suchconnections, or the connection can indicate that TCP-AO is required.Alternately, the configuration can be changed to discard segments withthe AO option not matching an MKT.

Connections not matching any MKT do not require TCP-AO. Further, incomingsegments with TCP-AO are not discarded solely because they includethe option, provided they do not match any MKT.

>> All TCP segments MUST be checked against the set of MKTs for matchingTCP connection identifiers.

ii. If they differ, determine whether the RNextKeyID MKT is ready.    1. If the MKT corresponding to the segment’s socket pair and RNextKeyID    is not available, no action is required (RNextKeyID of a received    segment needs to match the MKT’s SendID).

Rnext_key is changed only by manual user intervention or MKT managementprotocol operation. It is not manipulated by TCP-AO. Current_key is updatedby TCP-AO when processing received TCP segments as discussed in the segmentprocessing description in Section 7.5. Note that the algorithm allowsthe current_key to change to a new MKT, then change back to a previouslyused MKT (known as "backing up"). This can occur during an MKT change whensegments are received out of order, and is considered a feature of TCP-AO,because reordering does not result in drops.

2. If the matching MKT corresponding to the segment’s socket pair andRNextKeyID is available:   a. Set current_key to the RNextKeyID MKT.

Multiple MKTs may match a single outgoing segment, e.g., when MKTsare being changed. Those MKTs cannot have conflicting IDs (as notedelsewhere), and some mechanism must determine which MKT to use for eachgiven outgoing segment.>> An outgoing TCP segment MUST match at most one desired MKT, indicatedby the segment’s socket pair. The segment MAY match multiple MKTs, providedthat exactly one MKT is indicated as desired. Other information inthe segment MAY be used to determine the desired MKT when multiple MKTsmatch; such information MUST NOT include values in any TCP option fields.

TCP MD5-protected connections cannot be migrated to TCP-AO because TCP MD5does not support any changes to a connection’s security algorithmonce established.

TCP-AO requires that every protected TCP segment match exactly one MKT.

>> An incoming TCP segment including TCP-AO MUST match exactly one MKT,indicated solely by the segment’s socket pair and its TCP-AO KeyID.

One or more MKTs. These are the MKTs that match this connection’ssocket pair.

2. In-kernel MKTs database vs database in userspace¶

Linux TCP-AO support is implemented usingsetsockopt()s, in a similar wayto TCP-MD5. It means that a userspace application that wants to use TCP-AOshould performsetsockopt() on a TCP socket when it wants to add,remove or rotate MKTs. This approach moves the key management responsibilityto userspace as well as decisions on corner cases, i.e. what to do ifthe peer doesn’t respect RNextKeyID; moving more code to userspace, especiallyresponsible for the policy decisions. Besides, it’s flexible and scales well(with less locking needed than in the case of an in-kernel database). One alsoshould keep in mind that mainly intended users are BGP processes, not anyrandom applications, which means that compared to IPsec tunnels,no transparency is really needed and modern BGP daemons already havesetsockopt()s for TCP-MD5 support.

3. uAPI¶

Linux provides a set ofsetsockopt()s andgetsockopt()s that letuserspace manage TCP-AO on a per-socket basis. In order to add/delete MKTsTCP_AO_ADD_KEY andTCP_AO_DEL_KEY TCP socket options must be used.It is not allowed to add a key on an established non-TCP-AO connectionas well as to remove the last key from TCP-AO connection.

setsockopt(TCP_AO_DEL_KEY) command may specifytcp_ao_del::current_key+tcp_ao_del::set_current and/ortcp_ao_del::rnext+tcp_ao_del::set_rnext which makes such delete “forced”: itprovides userspace a way to delete a key that’s being used and atomically setanother one instead. This is not intended for normal use and should be usedonly when the peer ignores RNextKeyID and keeps requesting/using an old key.It provides a way to force-delete a key that’s not trusted but may breakthe TCP-AO connection.

The usual/normal key-rotation can be performed withsetsockopt(TCP_AO_INFO).It also provides a uAPI to change per-socket TCP-AO settings, such asignoring ICMPs, as well as clear per-socket TCP-AO packet counters.The correspondinggetsockopt(TCP_AO_INFO) can be used to get thoseper-socket TCP-AO settings.

Another useful command isgetsockopt(TCP_AO_GET_KEYS). One can use itto list all MKTs on a TCP socket or use a filter to get keys for a specificpeer and/or sndid/rcvid, VRF L3 interface or get current_key/rnext_key.

To repair TCP-AO connectionssetsockopt(TCP_AO_REPAIR) is available,provided that the user previously has checkpointed/dumped the socket withgetsockopt(TCP_AO_REPAIR).

A tip here for scaled TCP_LISTEN sockets, that may have some thousands TCP-AOkeys, is: use filters ingetsockopt(TCP_AO_GET_KEYS) and asynchronousdelete withsetsockopt(TCP_AO_DEL_KEY).

Linux TCP-AO also provides a bunch of segment counters that can be helpfulwith troubleshooting/debugging issues. Every MKT has good/bad countersthat reflect how many packets passed/failed verification.Each TCP-AO socket has the following counters:- for good segments (properly signed)- for bad segments (failed TCP-AO verification)- for segments with unknown keys- for segments where an AO signature was expected, but wasn’t found- for the number of ignored ICMPs

TCP-AO per-socket counters are also duplicated with per-netns counters,exposed with SNMP. Those areTCPAOGood,TCPAOBad,TCPAOKeyNotFound,TCPAORequired andTCPAODroppedIcmps.

For monitoring purposes, there are following TCP-AO trace events:tcp_hash_bad_header,tcp_hash_ao_required,tcp_ao_handshake_failure,tcp_ao_wrong_maclen,tcp_ao_wrong_maclen,tcp_ao_key_not_found,tcp_ao_rnext_request,tcp_ao_synack_no_key,tcp_ao_snd_sne_update,tcp_ao_rcv_sne_update. It’s possible to separately enable any of them andone can filter them by net-namespace, 4-tuple, family, L3 index, and TCP headerflags. If a segment has a TCP-AO header, the filters may also includekeyid, rnext, and maclen. SNE updates include the rolled-over numbers.

RFC 5925 very permissively specifies how TCP port matching can be done forMKTs:

TCP connection identifier. A TCP socket pair, i.e., a local IPaddress, a remote IP address, a TCP local port, and a TCP remote port.Values can be partially specified using ranges (e.g., 2-30), masks(e.g., 0xF0), wildcards (e.g., "*"), or any other suitable indication.

Currently Linux TCP-AO implementation doesn’t provide any TCP port matching.Probably, port ranges are the most flexible for uAPI, but so farnot implemented.

4.setsockopt() vsaccept() race¶

In contrast with an established TCP-MD5 connection which has just one key,TCP-AO connections may have many keys, which means that accepted connectionson a listen socket may have any amount of keys as well. As copying all thosekeys on a first properly signed SYN would make the request socket bigger, thatwould be undesirable. Currently, the implementation doesn’t copy keysto request sockets, but rather look them up on the “parent” listener socket.

The result is that when userspace removes TCP-AO keys, that may breaknot-yet-established connections on request sockets as well as not removingkeys from sockets that were already established, but not yetaccept()’ed,hanging in the accept queue.

The reverse is valid as well: if userspace adds a new key for a peer ona listener socket, the established sockets in the accept queue won’thave the new keys.

At this moment, the resolution for the two races:setsockopt(TCP_AO_ADD_KEY) vsaccept()andsetsockopt(TCP_AO_DEL_KEY) vsaccept() is delegated to userspace.This means that it’s expected that userspace would check the MKTs on the socketthat was returned byaccept() to verify that any key rotation thathappened on the listen socket is reflected on the newly established connection.

This is a similar “do-nothing” approach to TCP-MD5 from the kernel side andmay be changed later by introducing new flags totcp_ao_addandtcp_ao_del.

Note that this race is rare for it needs TCP-AO key rotation to happenduring the 3-way handshake for the new TCP connection.

5. Interaction with TCP-MD5¶

A TCP connection can not migrate between TCP-AO and TCP-MD5 options. Theestablished sockets that have either AO or MD5 keys are restricted foradding keys of the other option.

For listening sockets the picture is different: BGP server may want to receiveboth TCP-AO and (deprecated) TCP-MD5 clients. As a result, both types of keysmay be added to TCP_CLOSED or TCP_LISTEN sockets. It’s not allowed to adddifferent types of keys for the same peer.

6. SNE Linux implementation¶

RFC 5925 [6.2] describes the algorithm of how to extend TCP sequence numberswith SNE. In short: TCP has to track the previous sequence numbers and setsne_flag when the current SEQ number rolls over. The flag is cleared whenboth current and previous SEQ numbers cross 0x7fff, which is 32Kb.

In times when sne_flag is set, the algorithm compares SEQ for each packet with0x7fff and if it’s higher than 32Kb, it assumes that the packet should beverified with SNE before the increment. As a result, there’sthis [0; 32Kb] window, when packets with (SNE - 1) can be accepted.

Linux implementation simplifies this a bit: as the network stack already tracksthe first SEQ byte that ACK is wanted for (snd_una) and the next SEQ byte thatis wanted (rcv_nxt) - that’s enough information for a rough estimationon where in the 4GB SEQ number space both sender and receiver are.When they roll over to zero, the corresponding SNE gets incremented.

tcp_ao_compute_sne() is called for each TCP-AO segment. It compares SEQ numbersfrom the segment with snd_una or rcv_nxt and fits the result into a 2GB window around them,detecting SEQ numbers rolling over. That simplifies the code a lot and onlyrequires SNE numbers to be stored on every TCP-AO socket.

The 2GB window at first glance seems much more permissive compared toRFC 5926. But that is only used to pick the correct SNE before/aftera rollover. It allows more TCP segment replays, but yet all regularTCP checks intcp_sequence() are applied on the verified segment.So, it trades a bit more permissive acceptance of replayed/retransmittedsegments for the simplicity of the algorithm and what seems better behaviourfor large TCP windows.

7. Links¶

RFC 5925 The TCP Authentication Option

https://www.rfc-editor.org/rfc/pdfrfc/rfc5925.txt.pdf

RFC 5926 Cryptographic Algorithms for the TCP Authentication Option (TCP-AO)

https://www.rfc-editor.org/rfc/pdfrfc/rfc5926.txt.pdf

Draft “SHA-2 Algorithm for the TCP Authentication Option (TCP-AO)”

https://datatracker.ietf.org/doc/html/draft-nayak-tcp-sha2-03

RFC 2385 Protection of BGP Sessions via the TCP MD5 Signature Option

https://www.rfc-editor.org/rfc/pdfrfc/rfc2385.txt.pdf

Author:

Dmitry Safonov <dima@arista.com>