SMB Direct - SMB3 over RDMA

This document describes how to set up the Linux SMB client and server touse RDMA.

Overview

The Linux SMB kernel client supports SMB Direct, which is a transportscheme for SMB3 that uses RDMA (Remote Direct Memory Access) to providehigh throughput and low latencies by bypassing the traditional TCP/IPstack.SMB Direct on the Linux SMB client can be tested against KSMBD - akernel-space SMB server.

Installation

  • Install an RDMA device. As long as the RDMA device driver is supportedby the kernel, it should work. This includes both software emulators (softRoCE, soft iWARP) and hardware devices (InfiniBand, RoCE, iWARP).

  • Install a kernel with SMB Direct support. The first kernel release tosupport SMB Direct on both the client and server side is 5.15. Therefore,a distribution compatible with kernel 5.15 or later is required.

  • Install cifs-utils, which provides themount.cifs command to mount SMBshares.

  • Configure the RDMA stack

    Make sure that your kernel configuration has RDMA support enabled. UnderDevice Drivers -> Infiniband support, update the kernel configuration toenable Infiniband support.

    Enable the appropriate IB HCA support or iWARP adapter support,depending on your hardware.

    If you are using InfiniBand, enable IP-over-InfiniBand support.

    For soft RDMA, enable either the soft iWARP (RDMA _SIW) or soft RoCE(RDMA_RXE) module. Install theiproute2 package and use therdma link add command to load the module and create anRDMA interface.

    e.g. if your local ethernet interface iseth0, you can use:

    sudordmalinkaddsiw0typesiwnetdeveth0

  • Enable SMB Direct support for both the server and the client in the kernelconfiguration.

    Server Setup

    Network File Systems  --->    <M> SMB3 server support        [*] Support for SMB Direct protocol

    Client Setup

    Network File Systems  --->    <M> SMB3 and CIFS support (advanced network filesystem)        [*] SMB Direct support

  • Build and install the kernel. SMB Direct support will be enabled in thecifs.ko and ksmbd.ko modules.

Setup and Usage

  • Set up and start a KSMBD server as described in theKSMBD documentation.Also add the “server multi channel support = yes” parameter to ksmbd.conf.

  • On the client, mount the share withrdma mount option to use SMB Direct(specify a SMB version 3.0 or higher usingvers).

    For example:

    mount-tcifs//server/share/mnt/point-overs=3.1.1,rdma

  • To verify that the mount is using SMB Direct, you can check dmesg for thefollowing log line after mounting:

    CIFS: VFS: RDMA transport established

    Or, verifyrdma mount option for the share in/proc/mounts:

    cat/proc/mounts|grepcifs