6.1.Current Encapsulations

Appendix A includes a detailed comparison between the threeproposed encapsulations. The comparison indicates several commonproperties but also three major differences among theencapsulations:¶

• Extensibility: Geneve and GUE were defined with built-in extensibility, while VXLAN-GPE is not inherently extensible. Note that any of the three encapsulations can be extended using the Network Service Header (NSH)[RFC8300].¶
• Extension method: Geneve is extensible using Type-Length-Value (TLV) fields, while GUE uses a small set of possible extensions and a set of flags that indicate which extensions are present.¶
• Length field: Geneve and GUE include a Length field, indicating the length of the encapsulation header, while VXLAN-GPE does not include such a field. Thus, it may be harder to skip the encapsulation header with VXLAN-GPE¶

6.4.Extension Size

Extension header length has a significant impact on hardware andsoftware implementations. A maximum total header length that is toosmall will unnecessarily constrain software flexibility. A maximumtotal header length that is too large will place a nontrivial cost onhardware implementations. Thus, the DT recommends that there be aminimum and maximum total available extension header length specified.The maximum total header length is determined by the size of the bitfield allocated for the total extension header length field. The riskwith this approach is that it may be difficult to extend the totalheader size in the future. The minimum total header length isdetermined by a requirement in the specifications that allimplementations must meet. The risk with this approach is that allimplementations will only implement support for the minimum totalheader length, which would then become the de facto maximum totalheader length.¶

The recommended minimum total available header length is 64bytes.¶

The size of an extension header should always be 4-bytealigned.¶

The maximum length of a single option should be large enough tomeet the different extension use case requirements, e.g., for in-bandtelemetry and future use.¶

6.5.Ordering of Extension Headers

To support hardware nodes at the target NVE or at a transit devicethat can process one or a few extension headers in TCAM, a controlplane in such a deployment could signal a capability to ensure that aspecific extension header will always appear in a specific order, forexample, that such a specific extension header appear first in the packet.¶

The order of the extension headers should be hardware friendly forboth the sender and the receiver and possibly some transit devicesas well. This may require that the extension headers and their order bedetermined dynamically based on the hardware of those devices.¶

Transit devices don't participate in control plane communicationbetween the endpoints and are not required to process the extensionheaders; however, if they do, they may need to process only a smallsubset of the extension headers that will be consumed by targetNVEs.¶

6.6.TLV versus Bit Fields

If there is a well-known initial set of options that is likely tobe implemented in software and in hardware, it can be efficient to usethe bit fields approach to indicate the presence of extensions as inGUE. However, as described inSection 6.3, if options are added overtime and different subsets of options are likely to be implemented indifferent pieces of hardware, then it would be hard for the IETF tospecify which options should get the early bit fields. TLVs are a lotmore flexible, which avoids the need to determine the relativeimportance of different options. However, general TLVs of arbitraryorder, size, and repetition are difficult to implement in hardware. Amiddle ground is to use TLVs with restrictions on their size andalignment, observing that individual TLVs can have a fixed length, andto support via the control plane a method such that an NVE will onlyreceive options that it needs and implements. The control planeapproach can potentially be used to control the order of the TLVs sentto a particular NVE. Note that transit devices are not likely toparticipate in the control plane; hence, to the extent that they needto participate in option processing, some other method must beused. Transit devices would have issues with future GUE bit fieldsbeing defined for future options as well.¶

A benefit of TLVs from a hardware perspective is that they are self describing,i.e., all the information is in the TLV. In a bit fieldapproach, the hardware needs to look up the bit to determine thelength of the data associated with the bit through some separatetable, which would add hardware complexity.¶

There are use cases where multiple modules of software are runningon an NVE. These can be modules such as a diagnostic module by onevendor that does packet sampling and another module from a differentvendor that implements a firewall. Using a TLV format, it is easierto have different software modules process different TLVs without conflicting with each other. Such TLVs could be standard extensions or vendor-specific extensions. This can helpwith hardware modularity as well. There are some implementations withoptions that allow different software modules, like MAC learning andsecurity, to process different options.¶

6.7.Control Plane Considerations

Given that we want to allow considerable flexibility andextensibility (e.g., for software NVEs), yet want to be able to supportimportant extensions in less flexible contexts such as hardware NVEs,it is useful to consider the control plane. By control plane in thissection we mean protocols, such as EVPN[RFC8365]and others, and deployment-specific configurations.¶

If each NVE can express in the control plane that it only supportscertain extensions (which could be a single extension, or a few), andthe source NVEs only include supported extensions in the NVO3 packets,then the target NVE can use a simpler parser (e.g., a TCAM mightbe usable to look for a single NVO3 extension) and the depth of theinner payload in the NVO3 packet will be minimized. Furthermore, ifthe target NVE cares about a few extensions and can express in thecontrol plane the desired order of those extensions in the NVO3packets, then the deployment can provide useful functionality withsimplified hardware requirements for the target NVE.¶

Transit devices that are not aware of the NVO3 extensions somewhatbenefit from such an approach, since the inner payload is less deep inthe packet if no extraneous extension headers are included in thepacket. In general, a transit device is not likely to participate inthe NVO3 control plane. However, configuration mechanisms can takeinto account limitations of the transit devices used in particulardeployments.¶

Note that with this approach, different NVEs could desire differentextensions or sets of extensions, which means that the source NVEneeds to be able to place different sets of extensions in differentNVO3 packets, and perhaps in a different order. It also assumes thatunderlay multicast or replication servers are not used together withNVO3 extension headers.¶

There is a need to consider mandatory extensions versus optionalextensions. Mandatory extensions require the receiver to drop thepacket if the extension is unknown. A control plane mechanism canprevent the need for dropping unknown extensions, since they would notbe included to target NVEs that do not support them.¶

The control planes defined today need to add the ability todescribe the different encapsulations. Thus, perhaps EVPN[RFC8365] and any other control plane protocol that the IETFdefines should have a way to indicate the supported NVO3 extensionsand their order for each of the encapsulations supported.¶

Developing a separate document on guidance for option processing andcontrol plane participation should be considered. This should provideexamples and guidance on the range of usage models and deployment scenariosfor specific options. It should also provide examples of option ordering that are relevant for that specificdeployment. This includes endpoints and middleboxes that are using theoptions. Having the control plane negotiate the constraints is themost appropriate and flexible way to address these requirements.¶

6.8.Split NVE

If there is a need for hosts to send and receive options in a splitNVE case[RFC8394], this is possible using any of theexisting extensible encapsulations (GPE with NSH, GUE, or Geneve) by defininga way to carry those over other transports. An NSH can already be usedover different transports.¶

If this is needed with other encapsulations, it can be done bydefining an Ethertype so that it can be carried over Ethernet andIEEE Std 802.1Q[IEEE802.1Q].¶

If there is a need to carry other encapsulations over MPLS, itwould require an EVPN control plane to signal that other encapsulationheaders and options will be present in front of the Layer 2 (L2) packet. The VNIcan be ignored in the header, and the MPLS label will be the one usedto identify the EVPN L2 instance.¶

A.1.Overview

This section presents a comparison of the three NVO3 encapsulation proposals: Geneve[RFC8926], GUE[GUE], and VXLAN-GPE[VXLAN-GPE]. The three encapsulations use an outer UDP/IP transport. Geneve and VXLAN-GPE use an 8-octet header, while GUE uses a 4-octet header. In addition to the base header, optional extensions may be included in the encapsulation, as discussed inAppendix A.2 below.¶

A.3.Encapsulation Header

A.4.Comparison Summary

The following table summarizes the comparison between the threeNVO3 encapsulations. In some cases, a plus sign ("+") or minus sign("-") is used to indicate that the header is stronger or weaker in anarea, respectively.¶