Virtual Routing and Forwarding (VRF)

The VRF Device

The VRF device combined with ip rules provides the ability to create virtualrouting and forwarding domains (aka VRFs, VRF-lite to be specific) in theLinux network stack. One use case is the multi-tenancy problem where eachtenant has their own unique routing tables and in the very least needdifferent default gateways.

Processes can be “VRF aware” by binding a socket to the VRF device. Packetsthrough the socket then use the routing table associated with the VRFdevice. An important feature of the VRF device implementation is that itimpacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected(ie., they do not need to be run in each VRF). The design also allowsthe use of higher priority ip rules (Policy Based Routing, PBR) to takeprecedence over the VRF device rules directing specific traffic as desired.

In addition, VRF devices allow VRFs to be nested within namespaces. Forexample network namespaces provide separation of network interfaces at thedevice layer, VLANs on the interfaces within a namespace provide L2 separationand then VRF devices provide L3 separation.

Design

A VRF device is created with an associated route table. Network interfacesare then enslaved to a VRF device:

+-----------------------------+|           vrf-blue          |  ===> route table 10+-----------------------------+   |        |            |+------+ +------+     +-------------+| eth1 | | eth2 | ... |    bond1    |+------+ +------+     +-------------+                         |       |                     +------+ +------+                     | eth8 | | eth9 |                     +------+ +------+

Packets received on an enslaved device and are switched to the VRF devicein the IPv4 and IPv6 processing stacks giving the impression that packetsflow through the VRF device. Similarly on egress routing rules are used tosend packets to the VRF device driver before getting sent out the actualinterface. This allows tcpdump on a VRF device to capture all packets intoand out of the VRF as a whole[1]. Similarly, netfilter[2] and tc rulescan be applied using the VRF device to specify rules that apply to the VRFdomain as a whole.

[1]

Packets in the forwarded state do not flow through the device, so thosepackets are not seen by tcpdump. Will revisit this limitation in afuture release.

[2]

Iptables on ingress supports PREROUTING with skb->dev set to the realingress device and both INPUT and PREROUTING rules with skb->dev set tothe VRF device. For egress POSTROUTING and OUTPUT rules can be writtenusing either the VRF device or real egress device.

Setup

  1. VRF device is created with an association to a FIB table.e.g,:

    ip link add vrf-blue type vrf table 10ip link set dev vrf-blue up

  2. An l3mdev FIB rule directs lookups to the table associated with the device.A single l3mdev rule is sufficient for all VRFs. The VRF device adds thel3mdev rule for IPv4 and IPv6 when the first device is created with adefault preference of 1000. Users may delete the rule if desired and addwith a different priority or install per-VRF rules.

    Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:

    ip ru add oif vrf-blue table 10ip ru add iif vrf-blue table 10

  3. Set the default route for the table (and hence default route for the VRF):

    ip route add table 10 unreachable default metric 4278198272

    This high metric value ensures that the default unreachable route canbe overridden by a routing protocol suite. FRRouting interpretskernel metrics as a combined admin distance (upper byte) and priority(lower 3 bytes). Thus the above metric translates to [255/8192].

  4. Enslave L3 interfaces to a VRF device:

    ip link set dev eth1 master vrf-blue

    Local and connected routes for enslaved devices are automatically moved tothe table associated with VRF device. Any additional routes depending onthe enslaved device are dropped and will need to be reinserted to the VRFFIB table following the enslavement.

    The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 globaladdresses as VRF enslavement changes:

    sysctl -w net.ipv6.conf.all.keep_addr_on_down=1

  5. Additional VRF routes are added to associated table:

    ip route add table 10 ...

Applications

Applications that are to work within a VRF need to bind their socket to theVRF device:

setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);

or to specify the output device using cmsg and IP_PKTINFO.

By default the scope of the port bindings for unbound sockets islimited to the default VRF. That is, it will not be matched by packetsarriving on interfaces enslaved to an l3mdev and processes may bind tothe same port if they bind to an l3mdev.

TCP & UDP services running in the default VRF context (ie., not boundto any VRF device) can work across all VRF domains by enabling thetcp_l3mdev_accept and udp_l3mdev_accept sysctl options:

sysctl -w net.ipv4.tcp_l3mdev_accept=1sysctl -w net.ipv4.udp_l3mdev_accept=1

These options are disabled by default so that a socket in a VRF is onlyselected for packets in that VRF. There is a similar option for RAWsockets, which is enabled by default for reasons of backwards compatibility.This is so as to specify the output device with cmsg and IP_PKTINFO, butusing a socket not bound to the corresponding VRF. This allows e.g. older pingimplementations to be run with specifying the device but without executing itin the VRF. This option can be disabled so that packets received in a VRFcontext are only handled by a raw socket bound to the VRF, and packets in thedefault VRF are only handled by a socket not bound to any VRF:

sysctl -w net.ipv4.raw_l3mdev_accept=0

netfilter rules on the VRF device can be used to limit access to servicesrunning in the default VRF context as well.

Using VRF-aware applications (applications which simultaneously create socketsoutside and inside VRFs) in conjunction withnet.ipv4.tcp_l3mdev_accept=1is possible but may lead to problems in some situations. With that sysctlvalue, it is unspecified which listening socket will be selected to handleconnections for VRF traffic; ie. either a socket bound to the VRF or an unboundsocket may be used to accept new connections from a VRF. This somewhatunexpected behavior can lead to problems if sockets are configured with extraoptions (ex. TCP MD5 keys) with the expectation that VRF traffic willexclusively be handled by sockets bound to VRFs, as would be the case withnet.ipv4.tcp_l3mdev_accept=0. Finally and as a reminder, regardless ofwhich listening socket is selected, established sockets will be created in theVRF based on the ingress interface, as documented earlier.

Using iproute2 for VRFs

iproute2 supports the vrf keyword as of v4.7. For backwards compatibility thissection lists both commands where appropriate -- with the vrf keyword and theolder form without it.

  1. Create a VRF

    To instantiate a VRF device and associate it with a table:

    $ ip link add dev NAME type vrf table ID

    As of v4.8 the kernel supports the l3mdev FIB rule where a single rulecovers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on firstdevice create.

  2. List VRFs

    To list VRFs that have been created:

    $ ip [-d] link show type vrf  NOTE: The -d option is needed to show the table id

    For example:

    $ ip -d link show type vrf11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000    link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0    vrf table 1 addrgenmode eui6412: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000    link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0    vrf table 10 addrgenmode eui6413: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000    link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0    vrf table 66 addrgenmode eui6414: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000    link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0    vrf table 81 addrgenmode eui64

    Or in brief output:

    $ ip -br link show type vrfmgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>

  3. Assign a Network Interface to a VRF

    Network interfaces are assigned to a VRF by enslaving the netdevice to aVRF device:

    $ ip link set dev NAME master NAME

    On enslavement connected and local routes are automatically moved to thetable associated with the VRF device.

    For example:

    $ ip link set dev eth0 master mgmt

  4. Show Devices Assigned to a VRF

    To show devices that have been assigned to a specific VRF add the masteroption to the ip command:

    $ ip link show vrf NAME$ ip link show master NAME

    For example:

    $ ip link show vrf red3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000    link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000    link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000    link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

    Or using the brief output:

    $ ip -br link show vrf redeth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>

  5. Show Neighbor Entries for a VRF

    To list neighbor entries associated with devices enslaved to a VRF deviceadd the master option to the ip command:

    $ ip [-6] neigh show vrf NAME$ ip [-6] neigh show master NAME

    For example:

    $  ip neigh show vrf red10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE$ ip -6 neigh show vrf red2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE

  6. Show Addresses for a VRF

    To show addresses for interfaces associated with a VRF add the masteroption to the ip command:

    $ ip addr show vrf NAME$ ip addr show master NAME

    For example:

    $ ip addr show vrf red3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000    link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff    inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1       valid_lft forever preferred_lft forever    inet6 2002:1::2/120 scope global       valid_lft forever preferred_lft forever    inet6 fe80::ff:fe00:202/64 scope link       valid_lft forever preferred_lft forever4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000    link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff    inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2       valid_lft forever preferred_lft forever    inet6 2002:2::2/120 scope global       valid_lft forever preferred_lft forever    inet6 fe80::ff:fe00:203/64 scope link       valid_lft forever preferred_lft forever7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000    link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

    Or in brief format:

    $ ip -br addr show vrf redeth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64eth5             DOWN

  7. Show Routes for a VRF

    To show routes for a VRF use the ip command to display the table associatedwith the VRF device:

    $ ip [-6] route show vrf NAME$ ip [-6] route show table ID

    For example:

    $ ip route show vrf redunreachable default  metric 4278198272broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.210.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.210.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2$ ip -6 route show vrf redlocal 2002:1:: dev lo  proto none  metric 0  pref mediumlocal 2002:1::2 dev lo  proto none  metric 0  pref medium2002:1::/120 dev eth1  proto kernel  metric 256  pref mediumlocal 2002:2:: dev lo  proto none  metric 0  pref mediumlocal 2002:2::2 dev lo  proto none  metric 0  pref medium2002:2::/120 dev eth2  proto kernel  metric 256  pref mediumlocal fe80:: dev lo  proto none  metric 0  pref mediumlocal fe80:: dev lo  proto none  metric 0  pref mediumlocal fe80::ff:fe00:202 dev lo  proto none  metric 0  pref mediumlocal fe80::ff:fe00:203 dev lo  proto none  metric 0  pref mediumfe80::/64 dev eth1  proto kernel  metric 256  pref mediumfe80::/64 dev eth2  proto kernel  metric 256  pref mediumff00::/8 dev red  metric 256  pref mediumff00::/8 dev eth1  metric 256  pref mediumff00::/8 dev eth2  metric 256  pref mediumunreachable default dev lo  metric 4278198272  error -101 pref medium

  8. Route Lookup for a VRF

    A test route lookup can be done for a VRF:

    $ ip [-6] route get vrf NAME ADDRESS$ ip [-6] route get oif NAME ADDRESS

    For example:

    $ ip route get 10.2.1.40 vrf red10.2.1.40 dev eth1  table red  src 10.2.1.2    cache$ ip -6 route get 2002:1::32 vrf red2002:1::32 from :: dev eth1  table red  proto kernel  src 2002:1::2  metric 256  pref medium

  9. Removing Network Interface from a VRF

    Network interfaces are removed from a VRF by breaking the enslavement tothe VRF device:

    $ ip link set dev NAME nomaster

    Connected routes are moved back to the default table and local entries aremoved to the local table.

    For example:

    $ ip link set dev eth0 nomaster

Commands used in this example:

cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF1  mgmt10 red66 blue81 greenEOFfunction vrf_create{    VRF=$1    TBID=$2    # create VRF device    ip link add ${VRF} type vrf table ${TBID}    if [ "${VRF}" != "mgmt" ]; then        ip route add table ${TBID} unreachable default metric 4278198272    fi    ip link set dev ${VRF} up}vrf_create mgmt 1ip link set dev eth0 master mgmtvrf_create red 10ip link set dev eth1 master redip link set dev eth2 master redip link set dev eth5 master redvrf_create blue 66ip link set dev eth3 master bluevrf_create green 81ip link set dev eth4 master greenInterface addresses from /etc/network/interfaces:auto eth0iface eth0 inet static      address 10.0.0.2      netmask 255.255.255.0      gateway 10.0.0.254iface eth0 inet6 static      address 2000:1::2      netmask 120auto eth1iface eth1 inet static      address 10.2.1.2      netmask 255.255.255.0iface eth1 inet6 static      address 2002:1::2      netmask 120auto eth2iface eth2 inet static      address 10.2.2.2      netmask 255.255.255.0iface eth2 inet6 static      address 2002:2::2      netmask 120auto eth3iface eth3 inet static      address 10.2.3.2      netmask 255.255.255.0iface eth3 inet6 static      address 2002:3::2      netmask 120auto eth4iface eth4 inet static      address 10.2.4.2      netmask 255.255.255.0iface eth4 inet6 static      address 2002:4::2      netmask 120