Reference counting in pnfs¶
The are several inter-related caches. We have layouts which canreference multiple devices, each of which can reference multiple data servers.Each data server can be referenced by multiple devices. Each devicecan be referenced by multiple layouts. To keep all of this straight,we need to reference count.
struct pnfs_layout_hdr¶
The on-the-wire command LAYOUTGET corresponds to structpnfs_layout_segment, usually referred to by the variable name lseg.Each nfs_inode may hold a pointer to a cache of these layoutsegments in nfsi->layout, of type struct pnfs_layout_hdr.
We reference the header for the inode pointing to it, across eachoutstanding RPC call that references it (LAYOUTGET, LAYOUTRETURN,LAYOUTCOMMIT), and for each lseg held within.
Each header is also (when non-empty) put on a list associated withstruct nfs_client (cl_layouts). Being put on this list does not bumpthe reference count, as the layout is kept around by the lseg thatkeeps it in the list.
deviceid_cache¶
lsegs reference device ids, which are resolved per nfs_client andlayout driver type. The device ids are held in a RCU cache (structnfs4_deviceid_cache). The cache itself is referenced across eachmount. The entries (struct nfs4_deviceid) themselves are held acrossthe lifetime of each lseg referencing them.
RCU is used because the deviceid is basically a write once, read manydata structure. The hlist size of 32 buckets needs betterjustification, but seems reasonable given that we can have multipledeviceid’s per filesystem, and multiple filesystems per nfs_client.
The hash code is copied from the nfsd code base. A discussion ofhashing and variations of this algorithm can be foundhere.
data server cache¶
file driver devices refer to data servers, which are kept in a modulelevel cache. Its reference is held over the lifetime of the deviceidpointing to it.
lseg¶
lseg maintains an extra reference corresponding to the NFS_LSEG_VALIDbit which holds it in the pnfs_layout_hdr’s list. When the final lsegis removed from the pnfs_layout_hdr’s list, the NFS_LAYOUT_DESTROYEDbit is set, preventing any new lsegs from being added.
layout drivers¶
PNFS utilizes what is called layout drivers. The STD defines 4 basiclayout types: “files”, “objects”, “blocks”, and “flexfiles”. For eachof these types there is a layout-driver with a common function-vectorstable which are called by the nfs-client pnfs-core to implement thedifferent layout types.
Files-layout-driver code is in: fs/nfs/filelayout/.. directoryBlocks-layout-driver code is in: fs/nfs/blocklayout/.. directoryFlexfiles-layout-driver code is in: fs/nfs/flexfilelayout/.. directory
blocks-layout setup¶
TODO: Document the setup needs of the blocks layout driver