Tunnel Sockets¶

L2TPv2 always uses UDP. L2TPv3 may use UDP or IP encapsulation.

To create a tunnel socket for use by L2TP, the standard POSIXsocket API is used.

For example, for a tunnel using IPv4 addresses and UDP encapsulation:

int sockfd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

Or for a tunnel using IPv6 addresses and IP encapsulation:

int sockfd = socket(AF_INET6, SOCK_DGRAM, IPPROTO_L2TP);

UDP socket programming doesn’t need to be covered here.

IPPROTO_L2TP is an IP protocol type implemented by the kernel’s L2TPsubsystem. The L2TPIP socket address is defined instructsockaddr_l2tpip andstructsockaddr_l2tpip6 atinclude/uapi/linux/l2tp.h. The address includes the L2TP tunnel(connection) id. To use L2TP IP encapsulation, an L2TPv3 applicationshould bind the L2TPIP socket using the locally assignedtunnel id. When the peer’s tunnel id and IP address is known, aconnect must be done.

If the L2TP application needs to handle L2TPv3 tunnel setup requestsfrom peers using L2TPIP, it must open a dedicated L2TPIPsocket to listen for those requests and bind the socket using tunnelid 0 since tunnel setup requests are addressed to tunnel id 0.

An L2TP tunnel and all of its sessions are automatically closed whenits tunnel socket is closed.

Netlink API¶

L2TP applications use netlink to manage L2TP tunnel and sessioninstances in the kernel. The L2TP netlink API is defined ininclude/uapi/linux/l2tp.h.

L2TP usesGeneric Netlink (GENL). Several commands are defined:Create, Delete, Modify and Get for tunnel and sessioninstances, e.g.L2TP_CMD_TUNNEL_CREATE. The API header lists thenetlink attribute types that can be used with each command.

Tunnel and session instances are identified by a locally unique32-bit id. L2TP tunnel ids are given byL2TP_ATTR_CONN_ID andL2TP_ATTR_PEER_CONN_ID attributes and L2TP session ids are givenbyL2TP_ATTR_SESSION_ID andL2TP_ATTR_PEER_SESSION_IDattributes. If netlink is used to manage L2TPv2 tunnel and sessioninstances, the L2TPv2 16-bit tunnel/session id is cast to a 32-bitvalue in these attributes.

In theL2TP_CMD_TUNNEL_CREATE command,L2TP_ATTR_FD tells thekernel the tunnel socket fd being used. If not specified, the kernelcreates a kernel socket for the tunnel, using IP parameters set inL2TP_ATTR_IP[6]_SADDR,L2TP_ATTR_IP[6]_DADDR,L2TP_ATTR_UDP_SPORT,L2TP_ATTR_UDP_DPORT attributes. Kernelsockets are used to implement unmanaged L2TPv3 tunnels (iproute2’s “ipl2tp” commands). IfL2TP_ATTR_FD is given, it must be a socket fdthat is already bound and connected. There is more information aboutunmanaged tunnels later in this document.

L2TP_CMD_TUNNEL_CREATE attributes:-

L2TP_CMD_TUNNEL_DESTROY attributes:-

L2TP_CMD_TUNNEL_MODIFY attributes:-

L2TP_CMD_TUNNEL_GET attributes:-

L2TP_CMD_SESSION_CREATE attributes:-

For Ethernet session types, this will create an l2tpeth virtualinterface which can then be configured as required. For PPP sessiontypes, a PPPoL2TP socket must also be opened and connected, mapping itonto the new session. This is covered in “PPPoL2TP Sockets” later.

L2TP_CMD_SESSION_DESTROY attributes:-

L2TP_CMD_SESSION_MODIFY attributes:-

L2TP_CMD_SESSION_GET attributes:-

Application developers should refer toinclude/uapi/linux/l2tp.h fornetlink command and attribute definitions.

Sample userspace code usinglibmnl:

• Open L2TP netlink socket:

  struct nl_sock *nl_sock;int l2tp_nl_family_id;nl_sock = nl_socket_alloc();genl_connect(nl_sock);genl_id = genl_ctrl_resolve(nl_sock, L2TP_GENL_NAME);

• Create a tunnel:

  struct nlmsghdr *nlh;struct genlmsghdr *gnlh;nlh = mnl_nlmsg_put_header(buf);nlh->nlmsg_type = genl_id; /* assigned to genl socket */nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;nlh->nlmsg_seq = seq;gnlh = mnl_nlmsg_put_extra_header(nlh, sizeof(*gnlh));gnlh->cmd = L2TP_CMD_TUNNEL_CREATE;gnlh->version = L2TP_GENL_VERSION;gnlh->reserved = 0;mnl_attr_put_u32(nlh, L2TP_ATTR_FD, tunl_sock_fd);mnl_attr_put_u32(nlh, L2TP_ATTR_CONN_ID, tid);mnl_attr_put_u32(nlh, L2TP_ATTR_PEER_CONN_ID, peer_tid);mnl_attr_put_u8(nlh, L2TP_ATTR_PROTO_VERSION, protocol_version);mnl_attr_put_u16(nlh, L2TP_ATTR_ENCAP_TYPE, encap);

• Create a session:

  struct nlmsghdr *nlh;struct genlmsghdr *gnlh;nlh = mnl_nlmsg_put_header(buf);nlh->nlmsg_type = genl_id; /* assigned to genl socket */nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;nlh->nlmsg_seq = seq;gnlh = mnl_nlmsg_put_extra_header(nlh, sizeof(*gnlh));gnlh->cmd = L2TP_CMD_SESSION_CREATE;gnlh->version = L2TP_GENL_VERSION;gnlh->reserved = 0;mnl_attr_put_u32(nlh, L2TP_ATTR_CONN_ID, tid);mnl_attr_put_u32(nlh, L2TP_ATTR_PEER_CONN_ID, peer_tid);mnl_attr_put_u32(nlh, L2TP_ATTR_SESSION_ID, sid);mnl_attr_put_u32(nlh, L2TP_ATTR_PEER_SESSION_ID, peer_sid);mnl_attr_put_u16(nlh, L2TP_ATTR_PW_TYPE, pwtype);/* there are other session options which can be set using netlink * attributes during session creation -- see l2tp.h */

• Delete a session:

  struct nlmsghdr *nlh;struct genlmsghdr *gnlh;nlh = mnl_nlmsg_put_header(buf);nlh->nlmsg_type = genl_id; /* assigned to genl socket */nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;nlh->nlmsg_seq = seq;gnlh = mnl_nlmsg_put_extra_header(nlh, sizeof(*gnlh));gnlh->cmd = L2TP_CMD_SESSION_DELETE;gnlh->version = L2TP_GENL_VERSION;gnlh->reserved = 0;mnl_attr_put_u32(nlh, L2TP_ATTR_CONN_ID, tid);mnl_attr_put_u32(nlh, L2TP_ATTR_SESSION_ID, sid);

• Delete a tunnel and all of its sessions (if any):

  struct nlmsghdr *nlh;struct genlmsghdr *gnlh;nlh = mnl_nlmsg_put_header(buf);nlh->nlmsg_type = genl_id; /* assigned to genl socket */nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;nlh->nlmsg_seq = seq;gnlh = mnl_nlmsg_put_extra_header(nlh, sizeof(*gnlh));gnlh->cmd = L2TP_CMD_TUNNEL_DELETE;gnlh->version = L2TP_GENL_VERSION;gnlh->reserved = 0;mnl_attr_put_u32(nlh, L2TP_ATTR_CONN_ID, tid);

PPPoL2TP Session Socket API¶

For PPP session types, a PPPoL2TP socket must be opened and connectedto the L2TP session.

When creating PPPoL2TP sockets, the application provides informationto the kernel about the tunnel and session in a socketconnect()call. Source and destination tunnel and session ids are provided, aswell as the file descriptor of a UDP or L2TPIP socket. Seestructpppol2tp_addr ininclude/linux/if_pppol2tp.h. For historical reasons,there are unfortunately slightly different address structures forL2TPv2/L2TPv3 IPv4/IPv6 tunnels and userspace must use the appropriatestructure that matches the tunnel socket type.

Userspace may control behavior of the tunnel or session usingsetsockopt and ioctl on the PPPoX socket. The following socketoptions are supported:-

In addition to the standard PPP ioctls, a PPPIOCGL2TPSTATS is providedto retrieve tunnel and session statistics from the kernel using thePPPoX socket of the appropriate tunnel or session.

Sample userspace code:

• Create session PPPoX data socket:

  /* Input: the L2TP tunnel UDP socket `tunnel_fd`, which needs to be * bound already (both sockname and peername), otherwise it will not be * ready. */struct sockaddr_pppol2tp sax;int session_fd;int ret;session_fd = socket(AF_PPPOX, SOCK_DGRAM, PX_PROTO_OL2TP);if (session_fd < 0)        return -errno;sax.sa_family = AF_PPPOX;sax.sa_protocol = PX_PROTO_OL2TP;sax.pppol2tp.fd = tunnel_fd;sax.pppol2tp.addr.sin_addr.s_addr = addr->sin_addr.s_addr;sax.pppol2tp.addr.sin_port = addr->sin_port;sax.pppol2tp.addr.sin_family = AF_INET;sax.pppol2tp.s_tunnel  = tunnel_id;sax.pppol2tp.s_session = session_id;sax.pppol2tp.d_tunnel  = peer_tunnel_id;sax.pppol2tp.d_session = peer_session_id;/* session_fd is the fd of the session's PPPoL2TP socket. * tunnel_fd is the fd of the tunnel UDP / L2TPIP socket. */ret = connect(session_fd, (struct sockaddr *)&sax, sizeof(sax));if (ret < 0 ) {        close(session_fd);        return -errno;}return session_fd;

L2TP control packets will still be available for read ontunnel_fd.

• Create PPP channel:

  /* Input: the session PPPoX data socket `session_fd` which was created * as described above. */int ppp_chan_fd;int chindx;int ret;ret = ioctl(session_fd, PPPIOCGCHAN, &chindx);if (ret < 0)        return -errno;ppp_chan_fd = open("/dev/ppp", O_RDWR);if (ppp_chan_fd < 0)        return -errno;ret = ioctl(ppp_chan_fd, PPPIOCATTCHAN, &chindx);if (ret < 0) {        close(ppp_chan_fd);        return -errno;}return ppp_chan_fd;

LCP PPP frames will be available for read onppp_chan_fd.

• Create PPP interface:

  /* Input: the PPP channel `ppp_chan_fd` which was created as described * above. */int ifunit = -1;int ppp_if_fd;int ret;ppp_if_fd = open("/dev/ppp", O_RDWR);if (ppp_if_fd < 0)        return -errno;ret = ioctl(ppp_if_fd, PPPIOCNEWUNIT, &ifunit);if (ret < 0) {        close(ppp_if_fd);        return -errno;}ret = ioctl(ppp_chan_fd, PPPIOCCONNECT, &ifunit);if (ret < 0) {        close(ppp_if_fd);        return -errno;}return ppp_if_fd;

IPCP/IPv6CP PPP frames will be available for read onppp_if_fd.

The ppp<ifunit> interface can then be configured as usual with netlink’sRTM_NEWLINK, RTM_NEWADDR, RTM_NEWROUTE, or ioctl’s SIOCSIFMTU, SIOCSIFADDR,SIOCSIFDSTADDR, SIOCSIFNETMASK, SIOCSIFFLAGS, or with theip command.

• Bridging L2TP sessions which have PPP pseudowire types (this is also calledL2TP tunnel switching or L2TP multihop) is supported by bridging the PPPchannels of the two L2TP sessions to be bridged:

  /* Input: the session PPPoX data sockets `session_fd1` and `session_fd2` * which were created as described further above. */int ppp_chan_fd;int chindx1;int chindx2;int ret;ret = ioctl(session_fd1, PPPIOCGCHAN, &chindx1);if (ret < 0)        return -errno;ret = ioctl(session_fd2, PPPIOCGCHAN, &chindx2);if (ret < 0)        return -errno;ppp_chan_fd = open("/dev/ppp", O_RDWR);if (ppp_chan_fd < 0)        return -errno;ret = ioctl(ppp_chan_fd, PPPIOCATTCHAN, &chindx1);if (ret < 0) {        close(ppp_chan_fd);        return -errno;}ret = ioctl(ppp_chan_fd, PPPIOCBRIDGECHAN, &chindx2);close(ppp_chan_fd);if (ret < 0)        return -errno;return 0;

It can be noted that when bridging PPP channels, the PPP session is not locallyterminated, and no local PPP interface is created. PPP frames arriving on onechannel are directly passed to the other channel, and vice versa.

The PPP channel does not need to be kept open. Only the session PPPoX datasockets need to be kept open.

More generally, it is also possible in the same way to e.g. bridge a PPPoL2TPPPP channel with other types of PPP channels, such as PPPoE.

See more details for the PPP side inPPP Generic Driver and Channel Interface.

Old L2TPv2-only API¶

When L2TP was first added to the Linux kernel in 2.6.23, itimplemented only L2TPv2 and did not include a netlink API. Instead,tunnel and session instances in the kernel were managed directly usingonly PPPoL2TP sockets. The PPPoL2TP socket is used as described insection “PPPoL2TP Session Socket API” but tunnel and session instancesare automatically created on aconnect() of the socket instead ofbeing created by a separate netlink request:

• Tunnels are managed using a tunnel management socket which is adedicated PPPoL2TP socket, connected to (invalid) sessionid 0. The L2TP tunnel instance is created when the PPPoL2TPtunnel management socket is connected and is destroyed when thesocket is closed.

• Session instances are created in the kernel when a PPPoL2TPsocket is connected to a non-zero session id. Session parametersare set using setsockopt. The L2TP session instance is destroyedwhen the socket is closed.

This API is still supported but its use is discouraged. Instead, newL2TPv2 applications should use netlink to first create the tunnel andsession, then create a PPPoL2TP socket for the session.

Unmanaged L2TPv3 tunnels¶

The kernel L2TP subsystem also supports static (unmanaged) L2TPv3tunnels. Unmanaged tunnels have no userspace tunnel socket, andexchange no control messages with the peer to set up the tunnel; thetunnel is configured manually at each end of the tunnel. Allconfiguration is done using netlink. There is no need for an L2TPuserspace application in this case -- the tunnel socket is created bythe kernel and configured using parameters sent in theL2TP_CMD_TUNNEL_CREATE netlink request. Theip utility ofiproute2 has commands for managing static L2TPv3 tunnels; doipl2tphelp for more information.

Debugging¶

The L2TP subsystem offers a range of debugging interfaces through thedebugfs filesystem.

To access these interfaces, the debugfs filesystem must first be mounted:

# mount -t debugfs debugfs /debug

Files under the l2tp directory can then be accessed, providing a summaryof the current population of tunnel and session contexts existing in thekernel:

# cat /debug/l2tp/tunnels

The debugfs files should not be used by applications to obtain L2TPstate information because the file format is subject to change. It isimplemented to provide extra debug information to help diagnoseproblems. Applications should instead use the netlink API.

In addition the L2TP subsystem implements tracepoints using the standardkernel event tracing API. The available L2TP events can be reviewed asfollows:

# find /debug/tracing/events/l2tp

Finally, /proc/net/pppol2tp is also provided for backwards compatibilitywith the original pppol2tp code. It lists information about L2TPv2tunnels and sessions only. Its use is discouraged.

Sockets¶

UDP sockets are implemented by the networking core. When an L2TPtunnel is created using a UDP socket, the socket is set up as anencapsulated UDP socket by setting encap_rcv and encap_destroycallbacks on the UDP socket. l2tp_udp_encap_recv is called whenpackets are received on the socket. l2tp_udp_encap_destroy is calledwhen userspace closes the socket.

L2TPIP sockets are implemented innet/l2tp/l2tp_ip.c andnet/l2tp/l2tp_ip6.c.

Tunnels¶

The kernel keeps astructl2tp_tunnel context per L2TP tunnel. Thel2tp_tunnel is always associated with a UDP or L2TP/IP socket andkeeps a list of sessions in the tunnel. When a tunnel is firstregistered with L2TP core, the reference count on the socket isincreased. This ensures that the socket cannot be removed while L2TP’sdata structures reference it.

Tunnels are identified by a unique tunnel id. The id is 16-bit forL2TPv2 and 32-bit for L2TPv3. Internally, the id is stored as a 32-bitvalue.

Tunnels are kept in a per-net list, indexed by tunnel id. Thetunnel id namespace is shared by L2TPv2 and L2TPv3.

Handling tunnel socket close is perhaps the most tricky part of theL2TP implementation. If userspace closes a tunnel socket, the L2TPtunnel and all of its sessions must be closed and destroyed. Since thetunnel context holds a ref on the tunnel socket, the socket’ssk_destruct won’t be called until the tunnel sock_put’s itssocket. For UDP sockets, when userspace closes the tunnel socket, thesocket’s encap_destroy handler is invoked, which L2TP uses to initiateits tunnel close actions. For L2TPIP sockets, the socket’s closehandler initiates the same tunnel close actions. All sessions arefirst closed. Each session drops its tunnel ref. When the tunnel refreaches zero, the tunnel drops its socket ref.

Sessions¶

The kernel keeps astructl2tp_session context for each session. Eachsession has private data which is used for data specific to thesession type. With L2TPv2, the session always carries PPPtraffic. With L2TPv3, the session can carry Ethernet frames (Ethernetpseudowire) or other data types such as PPP, ATM, HDLC or FrameRelay. Linux currently implements only Ethernet and PPP session types.

Some L2TP session types also have a socket (PPP pseudowires) whileothers do not (Ethernet pseudowires).

Like tunnels, L2TP sessions are identified by a uniquesession id. Just as with tunnel ids, the session id is 16-bit forL2TPv2 and 32-bit for L2TPv3. Internally, the id is stored as a 32-bitvalue.

Sessions hold a ref on their parent tunnel to ensure that the tunnelstays extant while one or more sessions references it.

Sessions are kept in a per-net list. L2TPv2 sessions and L2TPv3sessions are stored in separate lists. L2TPv2 sessions are keyedby a 32-bit key made up of the 16-bit tunnel ID and 16-bitsession ID. L2TPv3 sessions are keyed by the 32-bit session ID, sinceL2TPv3 session ids are unique across all tunnels.

Although the L2TPv3 RFC specifies that L2TPv3 session ids are notscoped by the tunnel, the Linux implementation has historicallyallowed this. Such session id collisions are supported using a per-nethash table keyed by sk and session ID. When looking up L2TPv3sessions, the list entry may link to multiple sessions with thatsession ID, in which case the session matching the given sk (tunnel)is used.

PPP¶

net/l2tp/l2tp_ppp.c implements the PPPoL2TP socket family. Each PPPsession has a PPPoL2TP socket.

The PPPoL2TP socket’s sk_user_data references the l2tp_session.

Userspace sends and receives PPP packets over L2TP using a PPPoL2TPsocket. Only PPP control frames pass over this socket: PPP datapackets are handled entirely by the kernel, passing between the L2TPsession and its associatedpppN netdev through the PPP channelinterface of the kernel PPP subsystem.

The L2TP PPP implementation handles the closing of a PPPoL2TP socketby closing its corresponding L2TP session. This is complicated becauseit must consider racing with netlink session create/destroy requestsand pppol2tp_connect trying to reconnect with a session that is in theprocess of being closed. PPP sessions hold a ref on their associatedsocket in order that the socket remains extants while the sessionreferences it.

Ethernet¶

net/l2tp/l2tp_eth.c implements L2TPv3 Ethernet pseudowires. Itmanages a netdev for each session.

L2TP Ethernet sessions are created and destroyed by netlink request,or are destroyed when the tunnel is destroyed. Unlike PPP sessions,Ethernet sessions do not have an associated socket.

RFCs¶

The kernel code implements the datapath features specified in thefollowing RFCs:

Implementations¶

A number of open source applications use the L2TP kernel subsystem:

Limitations¶

The current implementation has a number of limitations:

1. Interfacing with openvswitch is not yet implemented. It may beuseful to map OVS Ethernet and VLAN ports into L2TPv3 tunnels.

2. VLAN pseudowires are implemented using anl2tpethN interfaceconfigured with a VLAN sub-interface. Since L2TPv3 VLANpseudowires carry one and only one VLAN, it may be better to usea single netdevice rather than anl2tpethN andl2tpethN:Mpair per VLAN session. The netlink attributeL2TP_ATTR_VLAN_ID was added for this, but it was neverimplemented.

Testing¶

Unmanaged L2TPv3 Ethernet features are tested by the kernel’s built-inselftests. Seetools/testing/selftests/net/l2tp.sh.

Another test suite,l2tp-ktest, covers allof the L2TP APIs and tunnel/session types. This may be integrated intothe kernel’s built-in L2TP selftests in the future.