Switch ID¶

The switchdev driver must implement the net_device operationndo_get_port_parent_id for each port netdev, returning the same physical ID foreach port of a switch. The ID must be unique between switches on the samesystem. The ID does not need to be unique between switches on differentsystems.

The switch ID is used to locate ports on a switch and to know if aggregatedports belong to the same switch.

Port Netdev Naming¶

Udev rules should be used for port netdev naming, using some unique attributeof the port as a key, for example the port MAC address or the port PHYS name.Hard-coding of kernel netdev names within the driver is discouraged; let thekernel pick the default netdev name, and let udev set the final name based on aport attribute.

Using port PHYS name (ndo_get_phys_port_name) for the key is particularlyuseful for dynamically-named ports where the device names its ports based onexternal configuration. For example, if a physical 40G port is split logicallyinto 4 10G ports, resulting in 4 port netdevs, the device can give a uniquename for each port using port PHYS name. The udev rule would be:

SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<phys_switch_id>", \        ATTR{phys_port_name}!="", NAME="swX$attr{phys_port_name}"

Suggested naming convention is “swXpYsZ”, where X is the switch name or ID, Yis the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0would be sub-port 0 on port 1 on switch 1.

Port Features¶

dev->netns_immutable

If the switchdev driver (and device) only supports offloading of the defaultnetwork namespace (netns), the driver should set this private flag to preventthe port netdev from being moved out of the default netns. A netns-awaredriver/device would not set this flag and be responsible for partitioninghardware to preserve netns containment. This means hardware cannot forwardtraffic from a port in one namespace to another port in another namespace.

Port Topology¶

The port netdevs representing the physical switch ports can be organized intohigher-level switching constructs. The default construct is a standalonerouter port, used to offload L3 forwarding. Two or more ports can be bondedtogether to form a LAG. Two or more ports (or LAGs) can be bridged to bridgeL2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3tunnels can be built on ports. These constructs are built using standard Linuxtools such as the bridge driver, the bonding/team drivers, and netlink-basedtools such as iproute2.

The switchdev driver can know a particular port’s position in the topology bymonitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into abond will see its upper master change. If that bond is moved into a bridge,the bond’s upper master will change. And so on. The driver will track suchmovements to know what position a port is in in the overall topology byregistering for netdevice events and acting on NETDEV_CHANGEUPPER.

Static FDB Entries¶

A driver which implements thendo_fdb_add,ndo_fdb_del andndo_fdb_dump operations is able to support the command below, which adds astatic bridge FDB entry:

bridge fdb add dev DEV ADDRESS [vlan VID] [self] static

(the “static” keyword is non-optional: if not specified, the entry defaults tobeing “local”, which means that it should not be forwarded)

The “self” keyword (optional because it is implicit) has the role ofinstructing the kernel to fulfill the operation through thendo_fdb_addimplementation of theDEV device itself. IfDEV is a bridge port, thiswill bypass the bridge and therefore leave the software database out of syncwith the hardware one.

To avoid this, the “master” keyword can be used:

bridge fdb add dev DEV ADDRESS [vlan VID] master static

The above command instructs the kernel to search for a master interface ofDEV and fulfill the operation through thendo_fdb_add method of that.This time, the bridge generates aSWITCHDEV_FDB_ADD_TO_DEVICE notificationwhich the port driver can handle and use it to program its hardware table. Thisway, the software and the hardware database will both contain this static FDBentry.

Note: for new switchdev drivers that offload the Linux bridge, implementing thendo_fdb_add andndo_fdb_del bridge bypass methods is stronglydiscouraged: all static FDB entries should be added on a bridge port using the“master” flag. Thendo_fdb_dump is an exception and can be implemented tovisualize the hardware tables, if the device does not have an interrupt fornotifying the operating system of newly learned/forgotten dynamic FDBaddresses. In that case, the hardware FDB might end up having entries that thesoftware FDB does not, and implementingndo_fdb_dump is the only way to seethem.

Note: by default, the bridge does not filter on VLAN and only bridges untaggedtraffic. To enable VLAN support, turn on VLAN filtering:

echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering

Notification of Learned/Forgotten Source MAC/VLANs¶

The switch device will learn/forget source MAC address/VLAN on ingress packetsand notify the switch driver of the mac/vlan/port tuples. The switch driver,in turn, will notify the bridge driver using the switchdev notifier call:

err = call_switchdev_notifiers(val, dev, info, extack);

Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL whenforgetting, and info points to astructswitchdev_notifier_fdb_info. OnSWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into thebridge’s FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridgecommand will label these entries “offload”:

$ bridge fdb52:54:00:12:35:01 dev sw1p1 master br0 permanent00:02:00:00:02:00 dev sw1p1 master br0 offload00:02:00:00:02:00 dev sw1p1 self52:54:00:12:35:02 dev sw1p2 master br0 permanent00:02:00:00:03:00 dev sw1p2 master br0 offload00:02:00:00:03:00 dev sw1p2 self33:33:00:00:00:01 dev eth0 self permanent01:00:5e:00:00:01 dev eth0 self permanent33:33:ff:00:00:00 dev eth0 self permanent01:80:c2:00:00:0e dev eth0 self permanent33:33:00:00:00:01 dev br0 self permanent01:00:5e:00:00:01 dev br0 self permanent33:33:ff:12:35:01 dev br0 self permanent

Learning on the port should be disabled on the bridge using the bridge command:

bridge link set dev DEV learning off

Learning on the device port should be enabled, as well as learning_sync:

bridge link set dev DEV learning on selfbridge link set dev DEV learning_sync on self

Learning_sync attribute enables syncing of the learned/forgotten FDB entry tothe bridge’s FDB. It’s possible, but not optimal, to enable learning on thedevice port and on the bridge port, and disable learning_sync.

To support learning, the driver implements switchdev opswitchdev_port_attr_set for SWITCHDEV_ATTR_PORT_ID_{PRE}_BRIDGE_FLAGS.

FDB Ageing¶

The bridge will skip ageing FDB entries marked with NTF_EXT_LEARNED and it isthe responsibility of the port driver/device to age out these entries. If theport device supports ageing, when the FDB entry expires, it will notify thedriver which in turn will notify the bridge with SWITCHDEV_FDB_DEL. If thedevice does not support ageing, the driver can simulate ageing using agarbage collection timer to monitor FDB entries. Expired entries will benotified to the bridge using SWITCHDEV_FDB_DEL. See rocker driver forexample of driver running ageing timer.

To keep an NTF_EXT_LEARNED entry “alive”, the driver should refresh the FDBentry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). Thenotification will reset the FDB entry’s last-used time to now. The drivershould rate limit refresh notifications, for example, no more than once asecond. (The last-used time is visible using the bridge -s fdb option).

STP State Change on Port¶

Internally or with a third-party STP protocol implementation (e.g. mstpd), thebridge driver maintains the STP state for ports, and will notify the switchdriver of STP state change on a port using the switchdev opswitchdev_attr_port_set for SWITCHDEV_ATTR_PORT_ID_STP_UPDATE.

State is one of BR_STATE_*. The switch driver can use STP state updates toupdate ingress packet filter list for the port. For example, if port isDISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUsand other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass.

Note that STP BDPUs are untagged and STP state applies to all VLANs on the portso packet filters should be applied consistently across untagged and taggedVLANs on the port.

Flooding L2 domain¶

For a given L2 VLAN domain, the switch device should flood multicast/broadcastand unknown unicast packets to all ports in domain, if allowed by port’scurrent STP state. The switch driver, knowing which ports are within whichvlan L2 domain, can program the switch device for flooding. The packet maybe sent to the port netdev for processing by the bridge driver. Thebridge should not reflood the packet to the same ports the device flooded,otherwise there will be duplicate packets on the wire.

To avoid duplicate packets, the switch driver should mark a packet as alreadyforwarded by setting the skb->offload_fwd_mark bit. The bridge driver will markthe skb using the ingress bridge port’s mark and prevent it from being forwardedthrough any bridge port with the same mark.

It is possible for the switch device to not handle flooding and push thepackets up to the bridge driver for flooding. This is not ideal as the numberof ports scale in the L2 domain as the device is much more efficient atflooding packets that software.

If supported by the device, flood control can be offloaded to it, preventingcertain netdevs from flooding unicast traffic for which there is no FDB entry.

IGMP Snooping¶

In order to support IGMP snooping, the port netdevs should trap to the bridgedriver all IGMP join and leave messages.The bridge multicast module will notify port netdevs on every multicast groupchanged whether it is static configured or dynamically joined/leave.The hardware implementation should be forwarding all registered multicasttraffic groups only to the configured ports.

Nexthop Resolution¶

The FIB entry’s nexthop list contains the nexthop tuple (gateway, dev), but forthe switch device to forward the packet with the correct dst mac address, thenexthop gateways must be resolved to the neighbor’s mac address. Neighbor macaddress discovery comes via the ARP (or ND) process and is available via thearp_tbl neighbor table. To resolve the routes nexthop gateways, the drivershould trigger the kernel’s neighbor resolution process. See the rockerdriver’srocker_port_ipv4_resolve() for an example.

The driver can monitor for updates to arp_tbl using the netevent notifierNETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthopsfor the routes as arp_tbl updates. The driver implements ndo_neigh_destroyto know when arp_tbl neighbor entries are purged from the port.

Configuration-less state¶

Upon driver bring up, the network devices must be fully operational, and thebacking driver must configure the network device such that it is possible tosend and receive traffic to this network device and it is properly separatedfrom other network devices/ports (e.g.: as is frequent with a switch ASIC). Howthis is achieved is heavily hardware dependent, but a simple solution can be touse per-port VLAN identifiers unless a better mechanism is available(proprietary metadata for each network port for instance).

The network device must be capable of running a full IP protocol stackincluding multicast, DHCP, IPv4/6, etc. If necessary, it should program theappropriate filters for VLAN, multicast, unicast etc. The underlying devicedriver must effectively be configured in a similar fashion to what it would dowhen IGMP snooping is enabled for IP multicast over these switchdev networkdevices and unsolicited multicast must be filtered as early as possible inthe hardware.

When configuring VLANs on top of the network device, all VLANs must be working,irrespective of the state of other network devices (e.g.: other ports being partof a VLAN-aware bridge doing ingress VID checking). See below for details.

If the device implements e.g.: VLAN filtering, putting the interface inpromiscuous mode should allow the reception of all VLAN tags (including thosenot present in the filter(s)).

Bridged switch ports¶

When a switchdev enabled network device is added as a bridge member, it shouldnot disrupt any functionality of non-bridged network devices and theyshould continue to behave as normal network devices. Depending on the bridgeconfiguration knobs below, the expected behavior is documented.

Bridge VLAN filtering¶

The Linux bridge allows the configuration of a VLAN filtering mode (statically,at device creation time, and dynamically, during run time) which must beobserved by the underlying switchdev network device/hardware:

• with VLAN filtering turned off: the bridge is strictly VLAN unaware and itsdata path will process all Ethernet frames as if they are VLAN-untagged.The bridge VLAN database can still be modified, but the modifications shouldhave no effect while VLAN filtering is turned off. Frames ingressing thedevice with a VID that is not programmed into the bridge/switch’s VLAN tablemust be forwarded and may be processed using a VLAN device (see below).

• with VLAN filtering turned on: the bridge is VLAN-aware and frames ingressingthe device with a VID that is not programmed into the bridges/switch’s VLANtable must be dropped (strict VID checking).

When there is a VLAN device (e.g: sw0p1.100) configured on top of a switchdevnetwork device which is a bridge port member, the behavior of the softwarenetwork stack must be preserved, or the configuration must be refused if thatis not possible.

• with VLAN filtering turned off, the bridge will process all ingress trafficfor the port, except for the traffic tagged with a VLAN ID destined for aVLAN upper. The VLAN upper interface (which consumes the VLAN tag) can evenbe added to a second bridge, which includes other switch ports or softwareinterfaces. Some approaches to ensure that the forwarding domain for trafficbelonging to the VLAN upper interfaces are managed properly:

  • If forwarding destinations can be managed per VLAN, the hardware could beconfigured to map all traffic, except the packets tagged with a VIDbelonging to a VLAN upper interface, to an internal VID corresponding tountagged packets. This internal VID spans all ports of the VLAN-unawarebridge. The VID corresponding to the VLAN upper interface spans thephysical port of that VLAN interface, as well as the other ports thatmight be bridged with it.

  • Treat bridge ports with VLAN upper interfaces as standalone, and letforwarding be handled in the software data path.

• with VLAN filtering turned on, these VLAN devices can be created as long asthe bridge does not have an existing VLAN entry with the same VID on anybridge port. These VLAN devices cannot be enslaved into the bridge since theyduplicate functionality/use case with the bridge’s VLAN data path processing.

Non-bridged network ports of the same switch fabric must not be disturbed in anyway by the enabling of VLAN filtering on the bridge device(s). If the VLANfiltering setting is global to the entire chip, then the standalone portsshould indicate to the network stack that VLAN filtering is required by setting‘rx-vlan-filter: on [fixed]’ in the ethtool features.

Because VLAN filtering can be turned on/off at runtime, the switchdev drivermust be able to reconfigure the underlying hardware on the fly to honor thetoggling of that option and behave appropriately. If that is not possible, theswitchdev driver can also refuse to support dynamic toggling of the VLANfiltering knob at runtime and require a destruction of the bridge device(s) andcreation of new bridge device(s) with a different VLAN filtering value toensure VLAN awareness is pushed down to the hardware.

Even when VLAN filtering in the bridge is turned off, the underlying switchhardware and driver may still configure itself in a VLAN-aware mode providedthat the behavior described above is observed.

The VLAN protocol of the bridge plays a role in deciding whether a packet istreated as tagged or not: a bridge using the 802.1ad protocol must treat bothVLAN-untagged packets, as well as packets tagged with 802.1Q headers, asuntagged.

The 802.1p (VID 0) tagged packets must be treated in the same way by the deviceas untagged packets, since the bridge device does not allow the manipulation ofVID 0 in its database.

When the bridge has VLAN filtering enabled and a PVID is not configured on theingress port, untagged and 802.1p tagged packets must be dropped. When the bridgehas VLAN filtering enabled and a PVID exists on the ingress port, untagged andpriority-tagged packets must be accepted and forwarded according to thebridge’s port membership of the PVID VLAN. When the bridge has VLAN filteringdisabled, the presence/lack of a PVID should not influence the packetforwarding decision.

Bridge IGMP snooping¶

The Linux bridge allows the configuration of IGMP snooping (statically, atinterface creation time, or dynamically, during runtime) which must be observedby the underlying switchdev network device/hardware in the following way:

• when IGMP snooping is turned off, multicast traffic must be flooded to allports within the same bridge that have mcast_flood=true. The CPU/managementport should ideally not be flooded (unless the ingress interface hasIFF_ALLMULTI or IFF_PROMISC) and continue to learn multicast traffic throughthe network stack notifications. If the hardware is not capable of doing thatthen the CPU/management port must also be flooded and multicast filteringhappens in software.

• when IGMP snooping is turned on, multicast traffic must selectively flowto the appropriate network ports (including CPU/management port). Flooding ofunknown multicast should be only towards the ports connected to a multicastrouter (the local device may also act as a multicast router).

The switch must adhere to RFC 4541 and flood multicast traffic accordinglysince that is what the Linux bridge implementation does.

Because IGMP snooping can be turned on/off at runtime, the switchdev drivermust be able to reconfigure the underlying hardware on the fly to honor thetoggling of that option and behave appropriately.

A switchdev driver can also refuse to support dynamic toggling of the multicastsnooping knob at runtime and require the destruction of the bridge device(s)and creation of a new bridge device(s) with a different multicast snoopingvalue.