Repository files navigation

• Overview
• Package Content
• Hardware Support
• OS Support
• Software Compatibility
• Legal Disclaimer

This software package provides basic support forIntel(R) Resource Director Technology (Intel(R) RDT) andIntel(R) I/O Resource Director Technology (Intel(R) I/O RDT) including:Cache Monitoring Technology (CMT), Memory Bandwidth Monitoring (MBM),Cache Allocation Technology (CAT), Code and Data Prioritization (CDP)and Memory Bandwidth Allocation (MBA).

In principle, the software programs the technologies viaModel Specific Registers (MSR) on a hardware thread basis.MSR access is arranged via a standard operating system driver:msr on Linux and cpuctl on FreeBSD. In the most common architecturalimplementations, presence of the technologies is detected via theCPUID instruction.

In a limited number of special cases where CAT is not architecturallysupported on a particular SKU (but instead a non-architectural(model-specific) implementation exists) it can be detected via brandstring. This brand string is read from CPUID and compared to a tableof known-supported SKUs. If needed, a final check is to probe thespecific MSR’s to discover hardware capabilities, however it isrecommended that CPUID enumeration should be used wherever possible.

From software version v1.0.0 the library adds option to use Intel(R) RDT viaavailable OS interfaces (perf and resctrl on Linux). The library detectspresence of these interfaces and allows to select the preferred one througha configuration option.As the result, existing tools like 'pqos' or 'rdtset' can also be usedto manage Intel(R) RDT in an OS compatible way. As of release v4.3.0, OSinterface became the default option. 'pqos' tool wrappers have been added tohelp with the interface selection. 'pqos-os' and 'pqos-msr' for OS and MSRinterface operations respectively.

PID API compile time option has been removed and the APIs are always available.Note that proper operation of these APIs depends on availability andselection of OS interface.

This software package is maintained, updated and developed onhttps://github.com/intel/intel-cmt-cat

https://github.com/intel/intel-cmt-cat/wiki provides FAQ, usage examples anduseful links.

Please refer to INSTALL file for package installation instructions.

"lib" directory:
Includes software library files providing API's fortechnology detection, monitoring and allocation.Please refer to the library README for more details (lib/README).

“lib/perl” directory:
Includes PQoS library Perl wrapper.Please refer to the interface README for more details (lib/perl/README).

“lib/python” directory:
Includes PQoS library Python 3.x wrapper.Please refer to the interface README for more details (lib/python/README.md).

"pqos" directory:
Includes source files for a utility that provides command line access toIntel(R) RDT. The utility links against the library and programsthe technologies via its API's.Please refer to the utility README for more details "pqos/README".Manual page of "pqos" utility also provides information about tool usage:$ man pqos

"rdtset" directory:
Includes source files for a utility that provides "taskset"-like functionalityfor RDT configuration.The utility links against the library and programs the technologiesvia its API's.Please refer to the utility README for more details "rdtset/README".Manual page of "rdtset" utility also provides information about tool usage:$ man rdtset

"appqos" directory:
Includes source files for an application that allows to group apps intopriority based pools. Each pool is assigned an Intel(R) RDT and Intel(R) SSTconfiguration that can be set on startup or at runtime through a REST API.Please refer to the application README for more details "appqos/README".

"appqos_client" directory:
Includes source files for an App QoS client web application. The appprovides a simple user interface to remotely configure Intel(R) RDT andIntel(R) SST on systems where App QoS is running.Please refer to the application README for more details "appqos_client/README".

"examples" directory:
Includes C and Perl examples of Intel(R) RDT usage via the library API's.Please refer to README file for more details "examples/README".

"snmp" directory:
Includes Net-SNMP AgentX subagent written in Perl to demonstrate the use ofthe PQoS library Perl wrapper API.Please refer to README file for more details "snmp/README".

"tools" directory:
Includes membw tool for stressing memory bandwidth with different operations.

"srpm" directory:
Includes *.src *.rpm and *.spec files for the software package.

"ChangeLog" file:
Brief description of changes between releases.

"INSTALL" file:
Installation instructions.

"LICENSE" file:
License of the package.

"unit-test" directory:
Unit tests

Supported products can be found in Addendum A of the Intel® Resource Director Technology (Intel® RDT) Architecture Specification:https://www.intel.com/content/www/us/en/content-details/789566/intel-resource-director-technology-intel-rdt-architecture-specification.html

Addendum B contains a list of processors with model-specific Intel® RDT Features.
Note: Detection of model-specific features requires theRDT_PROBE_MSR environment variable to be set when using the library and utilities.These features are only available when using the MSR interface. See the "Interfaces" section below for more information.See thewiki for usage examples.

For additional Intel(R) RDT details please refer to the Intel(R)Architecture Software Development Manuals available at:https://www.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.htmlSpecific information can be found in volume 3a, Chapters 17.18 and 17.19.

Linux is the primary supported operating system at the moment. There is aFreeBSD port of the software but due to limited validation scope it is ratherexperimental at this stage. Although most modern Linux kernels include supportfor Intel(R) RDT, the Intel(R) RDT software package predates these extensionsand can operate with and without kernel support. The Intel(R) RDT software candetect and leverage these kernel extensions when available to add functionality,but is also compatible with legacy kernels.

Linux kernel support for Intel(R) RDT was originally introduced with Linux perfsystem call extensions for CMT and MBM. More recently, the Resctrl interfaceadded support for CAT, CDP and MBA. On modern Linux kernels, it is advised touse the kernel/OS interface when available. Details about these interfaces canbe found in resctrl_ui.txt. This software package, Intel(R) RDT, remains towork seamlessly in all Linux kernel versions.

The Intel(R) RDT software library and utilities offer two interfaces to programIntel(R) RDT technologies, these are the MSR & OS interfaces.

The MSR interface is used to configure the platform by programming the hardware(MSR's) directly. This is the legacy interface and requires no kernel supportfor Intel(R) RDT but is limited to monitoring and managing resources on a percore basis.

The OS interface was later added to the package and when selected, the librarywill leverage Linux kernel extensions to program these technologies. This allowsmonitoring and managing resources on a per core/process basis and should be usedwhen available.

Please see the tables below for more information on when Intel(R) RDT feature(MSR & OS) support was added to the package.

References:

1. Monitoring with Perf on a per core basis is not supported and returns invalid results.
2. The MSR and OS interfaces are not compatible. MSR interface is recommended if monitoring and allocation is to be used.

The only dependencies of Intel(R) RDT is access to C and pthreads libraries and:

• without kernel extensions - 'msr' kernel module
• with kernel extensions - Intel(R) RDT extended Perf system call and Resctrl filesystem

Enable Intel(R) RDT support in:

• kernel v4.10 - v4.13 with kernel configuration option CONFIG_INTEL_RDT_A
• kernel v4.14+ with kernel configuration option CONFIG_INTEL_RDT
• kernel v5.0+ with kernel configuration option CONFIG_X86_RESCTRL

Note: No kernel configuration options required before v4.10.

In short, using Intel(R) RDT or PCM software together withLinux perf and cgroup frameworks is not allowed at the moment.

As disappointing as it is, use of Linux perf for CMT & MBM andIntel(R) RDT for CAT & CDP is not allowed. This is becauseLinux perf overrides existing CAT configuration during its operations.

There are a number of options to choose from in order to make use of CAT:

• Intel(R) RDT software for CMT/MBM/CAT and CDP (core granularity only)
• use Linux resctrl for CAT and Linux perf for monitoring (kernel 4.10+)
• patch kernel with an out of tree cgroup patch (CAT) andonly use perf for monitoring (CMT kernels 4.1+, MBM kernels 4.6+)

Table 4. Software interoperability matrix

References:

1. pqos monitoring from Intel(R) RDT can detect otherpqos monitoring processes in the system.rdtset from Intel(R) RDT detects other processes started with rdtset andit will not use their CAT/CDP resources.

2. pqos from Intel(R) RDT can detect that PCM monitors cores andit will not attempt to hijack the cores unless forced.However, if pqos monitoring is started first and thenPCM is started then the latter one will hijack monitoringinfrastructure from pqos for its use.

3. Linux cgroup kernel patchhttps://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt

4. Linux kernel version 4.10 and newer.A wiki for Intel resctrl is available at:https://github.com/intel/intel-cmt-cat/wiki/resctrl

5. Only with Linux kernel version 4.10 (and newer),Intel(R) RDT version 1.0.0 (and newer) with selected OS interfaceSee '-I' option in 'man pqos' or 'pqos-os'.

PCM is available at:https://github.com/opcm/pcm

Table 5. Intel(R) RDT software enabling status.

Legend:

• Core - use of technology with core granularity
• Task - use of technology per task or group of tasks

References:

1. Linux kernel version 4.1 and newer
2. Linux kernel version 4.6 and newer
3. Linux perf corrupts CAT and CDP configuration even thoughit doesn't enable it
4. This is patch and relies on Linux perf enabling
5. Linux kernel version 4.10 and newer
6. perf API allows for CMT/MBM core monitoring but returned values are incorrect
7. Intel(R) RDT version 1.0.0 monitoring only and depends on kernel support
8. Linux kernel version 4.14 and newer
9. Linux kernel version 4.12 and newer

THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS ORIMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTSARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS ANDCONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTELDISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/ORUSE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TOFITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENTOF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.