Creating and Viewing Paravirtualized IPoIB Datalinks in Kernel Zones

Paravirtual (PV) drivers are high-performance network and disk drivers that significantly reduce the overhead of the traditional implementation of I/O device emulation. These drivers provide improved network performance, disk throughput, and system efficiency because these drivers do no not emulate other devices such as physical NICs. The paravirtualized network driverZVNET for Oracle Solaris Kernel Zones, interact with the hypervisor in the host OS through hypercall to achieve low-delay and high-throughput network performance.

About Paravirtualized IPoIB Datalinks in Oracle Solaris

Starting with Oracle Solaris 11.3, the paravirtualized IPoIB datalink is created as ananet resource in Oracle Solaris Kernel Zone and you can configure this datalink by using thezonecfg command. Theanet resource creates an IPoIB VNIC when the kernel zone boots up. The IPoIB VNIC is created over a partition of the lower link InfiniBand host channel adapter (IB HCA) and the port tuple in the global zone. Each IPoIB VNIC has one-to-one match and communicates with paravirtualized IPoIB datalink in the kernel zone. Each of these VNICs have a unique MAC address and can have a unique or different partition key (pkey). For eachanet resource, you can configure the mode over which the IPoIB datalinks are run. Connected mode (CM) and unreliable datagram (UD) mode are supported and you can configure these modes by using thezonecfg command. For more information, seeZone Resource Types and Their Properties inOracle Solaris Zones Configuration Resources.

To display the configured IPoIB datalinks within the kernel zone, use thedladm command.

Example 35  Creating a Paravirtualized IPoIB Datalink

You create a paravirtualized IPoIB datalink by creating an automatic network (anet) in the kernel zone and specifying the mandatory propertieslower-link andpkey. Set thelower-link property to one of the valid IB partitions and setpkey to one of the partition keys provided by that partition. The propertylinkmode, which can be eithercm orud, is optional. If you do not specify a value, the value is set tocm by default.

$zonecfg -z kzone0zonecfg:kzone0>add anetzonecfg:kzone0:anet>set lower-link=net1zonecfg:kzone0:anet>set pkey=0x8001zonecfg:kzone0:anet>set linkmode=cmzonecfg:kzone0:anet>end$zoneadm -z kzone0 boot

Example 36  Displaying Physical Device Information in Kernel Zones

The following example displays the physical device and attributes of all physical datalinks in a kernel zone including the InfiniBand devices.

solariskzone0:~$dladm show-physLINK              MEDIA                STATE      SPEED DUPLEX    DEVICEnet0              Ethernet             up         1000  full      zvnet0net1              Infiniband           up         32000 full      zvnet1

The following example displays the physical device and all the key attributes of physical links in a kernel zone.

solariszone1:~$dladm show-phys -o allLINK   MEDIA        STATE    SPEED DUPLEX    DEVICE    VFS-AVAIL   VFS-INUSE   FLAGSnet0   Ethernet       up     1000  full      zvnet0     --         --          -----net1   Infiniband     up     32000 full      zvnet1     --         --          -----

Example 37  Displaying MAC Addresses for the Physical Device

The following example displays the MAC addresses for the physical device in a kernel zone.

solariszone1:~$dladm show-phys -mLINK                SLOT     ADDRESS             INUSE  CLIENTnet0                primary  2:8:20:5:32:5a      yes    net0net1                primary  80:0:0:4a:fe:80:... yes    net1

Example 38  Displaying the IPoIB VNIC in the Host

The following example displays the IPoIB VNIC in the host.

solariszone1:~$dladm show-vnicLINK            OVER     SPEED  MACADDRESS      MACADDRTYPE IDSkzone1/net0     net0     1000   2:8:20:5:32:5a  random      VID:0kzone1/net1     net4     32000  80:0:0:4a:fe:.. fixed       PKEY:0x8001

In this example, the notationPKEY in theIDS field indicates that the VNIC is an IPoIB VNIC.

The following example displays the MAC addresses of the IPoIB VNICs.

solariszone1:~$dladm show-vnic -o macaddressMACADDRESS2:8:20:5:32:5a80:0:0:4a:fe:80:0:0:0:0:0:0:0:21:28:0:1:a0:e5:55

Example 39  Displaying Datalinks in the Host

The following example displays the data links in the host including the IPoIB VNIC created on the kernel zone.

solariszone1:~$dladm show-link  LINK                CLASS     MTU    STATE    OVERnet0                phys      1500   up       --net1                phys      1500   unknown  --kzone1/net0         vnic      1500   up       net0kzone1/net1         vnic      65520  up       net1

For more information, see thedladm(8) man page.

Creating VNICs Over Paravirtualized IPoIB Datalink

You can create an IPoIB VNIC over a paravirtualized IPoIB datalink in an Oracle Solaris Kernel Zone. The IPoIB VNIC is assigned the primary MAC address of the IPoIB datalink instance if it is not already used by another MAC client. As a result, you can create a non-global zone inside a kernel zone by using the paravirtualized IPoIB datalink as the lower link. The IPoIB VNIC inherits the partition key stored in the paravirtualized IPoIB link, which is again inherited from thepkey of the IB partition in the global zone.

To support multiple Oracle Solaris zones in a kernel zone, multiple IPoIBanet instances need to be assigned to the kernel zone from the global zone, and the kernel zone needs to assign a differentanet to each of its zones.

You can create the IPoIB VNIC over paravirtualized IPoIB datalink similar to the way in which you create IPoIB VNICs over the IB partition in the global zone. For information, seeConfiguring IPoIB VNICs.

You can specify the partition key while creating the IPoIB VNIC. A paravirtualized IPoIB datalink stores thepkey that is inherited from the partition created in the host. You can specify thispkey value when you create the VNIC. You can use thedladm show-phys -v command to see thepkey value of the paravirtualized datalink. However, you can also create a VNIC without specifying thepkey. In this case, the inheritedpkey is assigned by default.

Example 40  Creating an IPoIB VNIC Over Paravirtualized IPoIB Datalink by Specifying thepkey

The following example shows how to create the IPoIB VNIC by specifying thepkey over the IPoIB datalinknet1.

$dladm show-phys -vLINK    IDS           INUSE CLIENTnet0    VIDS:23       yes   --net1    PKEYS:ffff    no    --$dladm show-physLINK          MEDIA             STATE      SPEED  DUPLEX    DEVICEnet0          Ethernet          up         1000   full      zvnet0net1          Infiniband        up         32000  full      zvnet1$dladm create-vnic -l net1 -P 0xffff vnic1$dladmLINK                CLASS     MTU    STATE    OVERnet0                phys      1500   up       --net1                phys      65520  up       --vnic1               vnic      65520  up       net1$dladm show-vnic vnic1LINK       OVER        SPEED  MACADDRESS        MACADDRTYPE IDSvnic1      net1        32000  80:0:0:4a:fe:..   fixed       PKEY:0xffff$dladm show-phys -vLINK    IDS           INUSE CLIENTnet0    VIDS:23       yes   --net1    PKEYS:ffff    yes   vnic1

Example 41  Creating an IPoIB VNIC Over Paravirtualized IPoIB Datalink Without Specifying thepkey

The following example shows how to create IPoIB VNICvnic1 without specifying thepkey over the IPoIB datalinknet1.

$dladm show-phys -vLINK    ID           INUSE CLIENTnet0    VIDS:23       yes   --net1    PKEYS:ffff    no    --

$dladm create-vnic -l net1 vnic1$dladmLINK                CLASS     MTU    STATE    OVERnet0                phys      1500   up       --net1                phys      65520  up       --vnic1               vnic      65520  up       net1

In this example, thepkey value inherited by the IPoIB datalink is assigned to the VNIC.

Example 42  Displaying the Partition Key

The following example displays the partition keypkey for an IPoIB datalink.

$dladm show-phys -v -o link,vids,pkeysLINK    VIDS       PKEYSnet0    23         --net1    --         ffff