Userspace MAD access¶
Device files¶
Each port of each InfiniBand device has a “umad” device and an“issm” device attached. For example, a two-port HCA will have twoumad devices and two issm devices, while a switch will have onedevice of each type (for switch port 0).
Creating MAD agents¶
A MAD agent can be created by filling in a struct ib_user_mad_reg_reqand then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a filedescriptor for the appropriate device file. If the registrationrequest succeeds, a 32-bit id will be returned in the structure.For example:
struct ib_user_mad_reg_req req = { /* ... */ };ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req);if (!ret) my_agent = req.id;else perror("agent register");Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENTioctl. Also, all agents registered through a file descriptor willbe unregistered when the descriptor is closed.
- 2014
- a new registration ioctl is now provided which allows additionalfields to be provided during registration.Users of this registration call are implicitly setting the use ofpkey_index (see below).
Receiving MADs¶
MADs are received using read(). The receive side now supportsRMPP. The buffer passed to read() must be at least onestruct ib_user_mad + 256 bytes. For example:
If the buffer passed is not large enough to hold the receivedMAD (RMPP), the errno is set to ENOSPC and the length of thebuffer needed is set in mad.length.
Example for normal MAD (non RMPP) reads:
struct ib_user_mad *mad;mad = malloc(sizeof *mad + 256);ret = read(fd, mad, sizeof *mad + 256);if (ret != sizeof mad + 256) { perror("read"); free(mad);}Example for RMPP reads:
struct ib_user_mad *mad;mad = malloc(sizeof *mad + 256);ret = read(fd, mad, sizeof *mad + 256);if (ret == -ENOSPC)) { length = mad.length; free(mad); mad = malloc(sizeof *mad + length); ret = read(fd, mad, sizeof *mad + length);}if (ret < 0) { perror("read"); free(mad);}In addition to the actual MAD contents, the other struct ib_user_madfields will be filled in with information on the received MAD. Forexample, the remote LID will be in mad.lid.
If a send times out, a receive will be generated with mad.status setto ETIMEDOUT. Otherwise when a MAD has been successfully received,mad.status will be 0.
poll()/select() may be used to wait until a MAD can be read.
Sending MADs¶
MADs are sent using write(). The agent ID for sending should befilled into the id field of the MAD, the destination LID should befilled into the lid field, and so on. The send side does supportRMPP so arbitrary length MAD can be sent. For example:
struct ib_user_mad *mad;mad = malloc(sizeof *mad + mad_length);/* fill in mad->data */mad->hdr.id = my_agent; /* req.id from agent registration */mad->hdr.lid = my_dest; /* in network byte order... *//* etc. */ret = write(fd, &mad, sizeof *mad + mad_length);if (ret != sizeof *mad + mad_length) perror("write");
Transaction IDs¶
Users of the umad devices can use the lower 32 bits of thetransaction ID field (that is, the least significant half of thefield in network byte order) in MADs being sent to matchrequest/response pairs. The upper 32 bits are reserved for use bythe kernel and will be overwritten before a MAD is sent.
P_Key Index Handling¶
The old ib_umad interface did not allow setting the P_Key index forMADs that are sent and did not provide a way for obtaining the P_Keyindex of received MADs. A new layout for struct ib_user_mad_hdrwith a pkey_index member has been defined; however, to preserve binarycompatibility with older applications, this new layout will not be usedunless one of IB_USER_MAD_ENABLE_PKEY or IB_USER_MAD_REGISTER_AGENT2 ioctl’sare called before a file descriptor is used for anything else.
In September 2008, the IB_USER_MAD_ABI_VERSION will be incrementedto 6, the new layout of struct ib_user_mad_hdr will be used bydefault, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed.
Setting IsSM Capability Bit¶
To set the IsSM capability bit for a port, simply open thecorresponding issm device file. If the IsSM bit is already set,then the open call will block until the bit is cleared (or returnimmediately with errno set to EAGAIN if the O_NONBLOCK flag ispassed to open()). The IsSM bit will be cleared when the issm fileis closed. No read, write or other operations can be performed onthe issm file.
/dev files¶
To create the appropriate character device files automatically withudev, a rule like:
KERNEL=="umad*", NAME="infiniband/%k"KERNEL=="issm*", NAME="infiniband/%k"can be used. This will create device nodes named:
/dev/infiniband/umad0/dev/infiniband/issm0for the first port, and so on. The InfiniBand device and portassociated with these devices can be determined from the files:
/sys/class/infiniband_mad/umad0/ibdev/sys/class/infiniband_mad/umad0/portand:
/sys/class/infiniband_mad/issm0/ibdev/sys/class/infiniband_mad/issm0/port