The Bufmap¶

At startup userspace allocates two page-size-aligned (posix_memalign)mlocked memory buffers, one is used for IO and one is used for readdiroperations. The IO buffer is 41943040 bytes and the readdir buffer is4194304 bytes. Each buffer contains logical chunks, or partitions, anda pointer to each buffer is added to its own PVFS_dev_map_desc structurewhich also describes its total size, as well as the size and number ofthe partitions.

A pointer to the IO buffer’s PVFS_dev_map_desc structure is sent to amapping routine in the kernel module with an ioctl. The structure iscopied from user space to kernel space with copy_from_user and is usedto initialize the kernel module’s “bufmap” (structorangefs_bufmap), whichthen contains:

• refcnt- a reference counter

• desc_size - PVFS2_BUFMAP_DEFAULT_DESC_SIZE (4194304) - the IO buffer’spartition size, which represents the filesystem’s block size andis used for s_blocksize in super blocks.

• desc_count - PVFS2_BUFMAP_DEFAULT_DESC_COUNT (10) - the number ofpartitions in the IO buffer.

• desc_shift - log2(desc_size), used for s_blocksize_bits in super blocks.

• total_size - the total size of the IO buffer.

• page_count - the number of 4096 byte pages in the IO buffer.

• page_array - a pointer topage_count*(sizeof(structpage*)) bytesof kcalloced memory. This memory is used as an array of pointersto each of the pages in the IO buffer through a call to get_user_pages.

• desc_array - a pointer todesc_count*(sizeof(structorangefs_bufmap_desc))bytes of kcalloced memory. This memory is further initialized:

  user_desc is the kernel’s copy of the IO buffer’s ORANGEFS_dev_map_descstructure. user_desc->ptr points to the IO buffer.

  pages_per_desc = bufmap->desc_size / PAGE_SIZEoffset = 0bufmap->desc_array[0].page_array = &bufmap->page_array[offset]bufmap->desc_array[0].array_count = pages_per_desc = 1024bufmap->desc_array[0].uaddr = (user_desc->ptr) + (0 * 1024 * 4096)offset += 1024                   .                   .                   .bufmap->desc_array[9].page_array = &bufmap->page_array[offset]bufmap->desc_array[9].array_count = pages_per_desc = 1024bufmap->desc_array[9].uaddr = (user_desc->ptr) +                                       (9 * 1024 * 4096)offset += 1024

• buffer_index_array - a desc_count sized array of ints, used toindicate which of the IO buffer’s partitions are available to use.

• buffer_index_lock - a spinlock to protect buffer_index_array during update.

• readdir_index_array - a five (ORANGEFS_READDIR_DEFAULT_DESC_COUNT) elementint array used to indicate which of the readdir buffer’s partitions areavailable to use.

• readdir_index_lock - a spinlock to protect readdir_index_array duringupdate.

Operations¶

The kernel module builds an “op” (structorangefs_kernel_op_s) when itneeds to communicate with userspace. Part of the op contains the “upcall”which expresses the request to userspace. Part of the op eventuallycontains the “downcall” which expresses the results of the request.

The slab allocator is used to keep a cache of op structures handy.

At init time the kernel module defines and initializes a request listand an in_progress hash table to keep track of all the ops that arein flight at any given time.

Ops are stateful:

• unknown

  • op was just initialized

• waiting

  • op is on request_list (upward bound)

• inprogr

  • op is in progress (waiting for downcall)

• serviced

  • op has matching downcall; ok

• purged

  • op has to start a timer since client-coreexited uncleanly before servicing op

• given up

  • submitter has given up waiting for it

When some arbitrary userspace program needs to perform afilesystem operation on Orangefs (readdir, I/O, create, whatever)an op structure is initialized and tagged with a distinguishing IDnumber. The upcall part of the op is filled out, and the op ispassed to the “service_operation” function.

Service_operation changes the op’s state to “waiting”, putsit on the request list, and signals the Orangefs file_operations.pollfunction through a wait queue. Userspace is polling the pseudo-deviceand thus becomes aware of the upcall request that needs to be read.

When the Orangefs file_operations.read function is triggered, therequest list is searched for an op that seems ready-to-process.The op is removed from the request list. The tag from the op andthe filled-out upcallstructare copy_to_user’ed back to userspace.

If any of these (and some additional protocol) copy_to_users fail,the op’s state is set to “waiting” and the op is added back tothe request list. Otherwise, the op’s state is changed to “in progress”,and the op is hashed on its tag and put onto the end of a list in thein_progress hash table at the index the tag hashed to.

When userspace has assembled the response to the upcall, itwrites the response, which includes the distinguishing tag, back tothe pseudo device in a series of io_vecs. This triggers the Orangefsfile_operations.write_iter function to find the op with the associatedtag and remove it from the in_progress hash table. As long as the op’sstate is not “canceled” or “given up”, its state is set to “serviced”.The file_operations.write_iter function returns to the waiting vfs,and back to service_operation through wait_for_matching_downcall.

Service operation returns to its caller with the op’s downcallpart (the response to the upcall) filled out.

The “client-core” is the bridge between the kernel module anduserspace. The client-core is a daemon. The client-core has anassociated watchdog daemon. If the client-core is ever signaledto die, the watchdog daemon restarts the client-core. Even thoughthe client-core is restarted “right away”, there is a period oftime during such an event that the client-core is dead. A dead client-corecan’t be triggered by the Orangefs file_operations.poll function.Ops that pass through service_operation during a “dead spell” can timeouton the wait queue and one attempt is made to recycle them. Obviously,if the client-core stays dead too long, the arbitrary userspace processestrying to use Orangefs will be negatively affected. Waiting opsthat can’t be serviced will be removed from the request list andhave their states set to “given up”. In-progress ops that can’tbe serviced will be removed from the in_progress hash table andhave their states set to “given up”.

Readdir and I/O ops are atypical with respect to their payloads.

• readdir ops use the smaller of the two pre-allocated pre-partitionedmemory buffers. The readdir buffer is only available to userspace.The kernel module obtains an index to a free partition before launchinga readdir op. Userspace deposits the results into the indexed partitionand then writes them to back to the pvfs device.

• io (read and write) ops use the larger of the two pre-allocatedpre-partitioned memory buffers. The IO buffer is accessible fromboth userspace and the kernel module. The kernel module obtains anindex to a free partition before launching an io op. The kernel moduledeposits write data into the indexed partition, to be consumeddirectly by userspace. Userspace deposits the results of readrequests into the indexed partition, to be consumed directlyby the kernel module.

Responses to kernel requests are all packaged in pvfs2_downcall_tstructs. Besides a few other members, pvfs2_downcall_t contains aunionof structs, each of which is associated with a particularresponse type.

The several members outside of theunionare:

int32_ttype

• type of operation.

int32_tstatus

• return code for the operation.

int64_ttrailer_size

• 0 unless readdir operation.

char*trailer_buf

• initialized to NULL, used during readdir operations.

The appropriate member inside theunionis filled out for anyparticular response.

PVFS2_VFS_OP_FILE_IO

fill a pvfs2_io_response_t

PVFS2_VFS_OP_LOOKUP

fill a PVFS_object_kref

PVFS2_VFS_OP_CREATE

fill a PVFS_object_kref

PVFS2_VFS_OP_SYMLINK

fill a PVFS_object_kref

PVFS2_VFS_OP_GETATTR

fill in a PVFS_sys_attr_s (tons of stuff the kernel doesn’t need)fill in a string with the link target when the object is a symlink.

PVFS2_VFS_OP_MKDIR

fill a PVFS_object_kref

PVFS2_VFS_OP_STATFS

fill a pvfs2_statfs_response_t with useless info <g>. It is hard forus to know, in a timely fashion, these statistics about ourdistributed network filesystem.

PVFS2_VFS_OP_FS_MOUNT

fill a pvfs2_fs_mount_response_t which is just like a PVFS_object_krefexcept its members are in a different order and “__pad1” is replacedwith “id”.

PVFS2_VFS_OP_GETXATTR

fill a pvfs2_getxattr_response_t

PVFS2_VFS_OP_LISTXATTR

fill a pvfs2_listxattr_response_t

PVFS2_VFS_OP_PARAM

fill a pvfs2_param_response_t

PVFS2_VFS_OP_PERF_COUNT

fill a pvfs2_perf_count_response_t

PVFS2_VFS_OP_FSKEY

file a pvfs2_fs_key_response_t

PVFS2_VFS_OP_READDIR

jamb everything needed to represent a pvfs2_readdir_response_t intothe readdir buffer descriptor specified in the upcall.

Userspace useswritev() on /dev/pvfs2-req to pass responses to the requestsmade by the kernel side.

A buffer_list containing:

• a pointer to the prepared response to the request from thekernel (structpvfs2_downcall_t).

• and also, in the case of a readdir request, a pointer to abuffer containing descriptors for the objects in the targetdirectory.

... is sent to the function (PINT_dev_write_list) which performsthe writev.

PINT_dev_write_list has a local iovec array:structiovec io_array[10];

The first four elements of io_array are initialized like this for allresponses:

io_array[0].iov_base = address of local variable "proto_ver" (int32_t)io_array[0].iov_len = sizeof(int32_t)io_array[1].iov_base = address of global variable "pdev_magic" (int32_t)io_array[1].iov_len = sizeof(int32_t)io_array[2].iov_base = address of parameter "tag" (PVFS_id_gen_t)io_array[2].iov_len = sizeof(int64_t)io_array[3].iov_base = address of out_downcall member (pvfs2_downcall_t)                       of global variable vfs_request (vfs_request_t)io_array[3].iov_len = sizeof(pvfs2_downcall_t)

Readdir responses initialize the fifth element io_array like this:

io_array[4].iov_base = contents of member trailer_buf (char *)                       from out_downcall member of global variable                       vfs_requestio_array[4].iov_len = contents of member trailer_size (PVFS_size)                      from out_downcall member of global variable                      vfs_request

Orangefs exploits the dcache in order to avoid sending redundantrequests to userspace. We keep object inode attributes up-to-date withorangefs_inode_getattr. Orangefs_inode_getattr uses two arguments tohelp it decide whether or not to update an inode: “new” and “bypass”.Orangefs keeps private data in an object’s inode that includes a shorttimeout value, getattr_time, which allows any iteration oforangefs_inode_getattr to know how long it has been since the inode wasupdated. When the object is not new (new == 0) and the bypass flag is notset (bypass == 0) orangefs_inode_getattr returns without updating the inodeif getattr_time has not timed out. Getattr_time is updated each time theinode is updated.

Creation of a new object (file, dir, sym-link) includes the evaluation ofits pathname, resulting in a negative directory entry for the object.A new inode is allocated and associated with the dentry, turning it froma negative dentry into a “productive full member of society”. Orangefsobtains the new inode from Linux withnew_inode() and associatesthe inode with the dentry by sending the pair back to Linux withd_instantiate().

The evaluation of a pathname for an object resolves to its correspondingdentry. If there is no corresponding dentry, one is created for it inthe dcache. Whenever a dentry is modified or verified Orangefs stores ashort timeout value in the dentry’s d_time, and the dentry will be trustedfor that amount of time. Orangefs is a network filesystem, and objectscan potentially change out-of-band with any particular Orangefs kernel moduleinstance, so trusting a dentry is risky. The alternative to trustingdentries is to always obtain the needed information from userspace - atleast a trip to the client-core, maybe to the servers. Obtaining informationfrom a dentry is cheap, obtaining it from userspace is relatively expensive,hence the motivation to use the dentry when possible.

The timeout values d_time and getattr_time are jiffy based, and thecode is designed to avoid the jiffy-wrap problem:

"In general, if the clock may have wrapped around more than once, thereis no way to tell how much time has elapsed. However, if the times t1and t2 are known to be fairly close, we can reliably compute thedifference in a way that takes into account the possibility that theclock may have wrapped between times."

from course notes by instructor Andy Wang