Introduction¶

The NFSv4 protocol uses “lease-based file locking”. Leases helpNFSv4 servers provide file lock guarantees and manage theirresources.

Simply put, an NFSv4 server creates a lease for each NFSv4 client.The server collects each client’s file open and lock state underthe lease for that client.

The client is responsible for periodically renewing its leases.While a lease remains valid, the server holding that leaseguarantees the file locks the client has created remain in place.

If a client stops renewing its lease (for example, if it crashes),the NFSv4 protocol allows the server to remove the client’s openand lock state after a certain period of time. When a clientrestarts, it indicates to servers that open and lock stateassociated with its previous leases is no longer valid and can bedestroyed immediately.

In addition, each NFSv4 server manages a persistent list of clientleases. When the server restarts and clients attempt to recovertheir state, the server uses this list to distinguish amongstclients that held state before the server restarted and clientssending fresh OPEN and LOCK requests. This enables file locks topersist safely across server restarts.

NFSv4 client identifiers¶

Each NFSv4 client presents an identifier to NFSv4 servers so thatthey can associate the client with its lease. Each client’sidentifier consists of two elements:

• co_ownerid: An arbitrary but fixed string.

• boot verifier: A 64-bit incarnation verifier that enables aserver to distinguish successive boot epochs of the same client.

The NFSv4.0 specification refers to these two items as an“nfs_client_id4”. The NFSv4.1 specification refers to these twoitems as a “client_owner4”.

NFSv4 servers tie this identifier to the principal and securityflavor that the client used when presenting it. Servers use thisprincipal to authorize subsequent lease modification operationssent by the client. Effectively this principal is a third element ofthe identifier.

As part of the identity presented to servers, a good“co_ownerid” string has several important properties:

• The “co_ownerid” string identifies the client during rebootrecovery, therefore the string is persistent across clientreboots.

• The “co_ownerid” string helps servers distinguish the clientfrom others, therefore the string is globally unique. Notethat there is no central authority that assigns “co_ownerid”strings.

• Because it often appears on the network in the clear, the“co_ownerid” string does not reveal private information aboutthe client itself.

• The content of the “co_ownerid” string is set and unchangingbefore the client attempts NFSv4 mounts after a restart.

• The NFSv4 protocol places a 1024-byte limit on the size of the“co_ownerid” string.

Protecting NFSv4 lease state¶

NFSv4 servers utilize the “client_owner4” as described above toassign a unique lease to each client. Under this scheme, there arecircumstances where clients can interfere with each other. This isreferred to as “lease stealing”.

If distinct clients present the same “co_ownerid” string and usethe same principal (for example, AUTH_SYS and UID 0), a server isunable to tell that the clients are not the same. Each distinctclient presents a different boot verifier, so it appears to theserver as if there is one client that is rebooting frequently.Neither client can maintain open or lock state in this scenario.

If distinct clients present the same “co_ownerid” string and usedistinct principals, the server is likely to allow the first clientto operate normally but reject subsequent clients with the same“co_ownerid” string.

If a client’s “co_ownerid” string or principal are not stable,state recovery after a server or client reboot is not guaranteed.If a client unexpectedly restarts but presents a different“co_ownerid” string or principal to the server, the server orphansthe client’s previous open and lock state. This blocks access tolocked files until the server removes the orphaned state.

If the server restarts and a client presents a changed “co_ownerid”string or principal to the server, the server will not allow theclient to reclaim its open and lock state, and may give those locksto other clients in the meantime. This is referred to as “lockstealing”.

Lease stealing and lock stealing increase the potential for denialof service and in rare cases even data corruption.

Selecting an appropriate client identifier¶

By default, the Linux NFSv4 client implementation constructs its“co_ownerid” string starting with the words “Linux NFS” followed bythe client’s UTS node name (the same node name, incidentally, thatis used as the “machine name” in an AUTH_SYS credential). In smalldeployments, this construction is usually adequate. Often, however,the node name by itself is not adequately unique, and can changeunexpectedly. Problematic situations include:

• NFS-root (diskless) clients, where the local DHCP server (orequivalent) does not provide a unique host name.

• “Containers” within a single Linux host. If each container hasa separate network namespace, but does not use the UTS namespaceto provide a unique host name, then there can be multiple NFSclient instances with the same host name.

• Clients across multiple administrative domains that access acommon NFS server. If hostnames are not assigned centrallythen uniqueness cannot be guaranteed unless a domain name isincluded in the hostname.

Linux provides two mechanisms to add uniqueness to its “co_ownerid”string:

nfs.nfs4_unique_id

This module parameter can set an arbitrary uniquifier stringvia the kernel command line, or when the “nfs” module isloaded.

/sys/fs/nfs/net/nfs_client/identifier

This virtual file, available since Linux 5.3, is local to thenetwork namespace in which it is accessed and so can providedistinction between network namespaces (containers) when thehostname remains uniform.

Note that this file is empty on name-space creation. If thecontainer system has access to some sort of per-container identitythen that uniquifier can be used. For example, a uniquifier mightbe formed at boot using the container’s internal identifier:

sha256sum /etc/machine-id | awk ‘{print $1}’ \

> /sys/fs/nfs/net/nfs_client/identifier

Security considerations¶

The use of cryptographic security for lease management operationsis strongly encouraged.

If NFS with Kerberos is not configured, a Linux NFSv4 client usesAUTH_SYS and UID 0 as the principal part of its client identity.This configuration is not only insecure, it increases the risk oflease and lock stealing. However, it might be the only choice forclient configurations that have no local persistent storage.“co_ownerid” string uniqueness and persistence is critical in thiscase.

When a Kerberos keytab is present on a Linux NFS client, the clientattempts to use one of the principals in that keytab whenidentifying itself to servers. The “sec=” mount option does notcontrol this behavior. Alternately, a single-user client with aKerberos principal can use that principal in place of the client’shost principal.

Using Kerberos for this purpose enables the client and server touse the same lease for operations covered by all “sec=” settings.Additionally, the Linux NFS client uses the RPCSEC_GSS securityflavor with Kerberos and the integrity QOS to prevent in-transitmodification of lease modification requests.

Additional notes¶

The Linux NFSv4 client establishes a single lease on each NFSv4server it accesses. NFSv4 mounts from a Linux NFSv4 client of aparticular server then share that lease.

Once a client establishes open and lock state, the NFSv4 protocolenables lease state to transition to other servers, following datathat has been migrated. This hides data migration completely fromrunning applications. The Linux NFSv4 client facilitates statemigration by presenting the same “client_owner4” to all servers itencounters.