4.x86 Topology¶

This documents and clarifies the main aspects of x86 topology modelling andrepresentation in the kernel. Update/change when doing changes to therespective code.

The architecture-agnostic topology definitions are inHow CPU topology info is exported via sysfs. This file holds x86-specificdifferences/specialities which must not necessarily apply to the genericdefinitions. Thus, the way to read up on Linux topology on x86 is to startwith the generic one and look at this one in parallel for the x86 specifics.

Needless to say, code should use the generic functions - this file isonlyhere todocument the inner workings of x86 topology.

Started by Thomas Gleixner <tglx@kernel.org> and Borislav Petkov <bp@alien8.de>.

The main aim of the topology facilities is to present adequate interfaces tocode which needs to know/query/use the structure of the running system wrtthreads, cores, packages, etc.

The kernel does not care about the concept of physical sockets because asocket has no relevance to software. It’s an electromechanical component. Inthe past a socket always contained a single package (see below), but with theadvent of Multi Chip Modules (MCM) a socket can hold more than one package. Sothere might be still references to sockets in the code, but they are ofhistorical nature and should be cleaned up.

The topology of a system is described in the units of:

packages
cores
threads

4.1.Package¶

Packages contain a number of cores plus shared resources, e.g. DRAMcontroller, shared caches etc.

Modern systems may also use the term ‘Die’ for package.

AMD nomenclature for package is ‘Node’.

Package-related topology information in the kernel:

topology_num_threads_per_package()
The number of threads in a package.
topology_num_cores_per_package()
The number of cores in a package.
topology_max_dies_per_package()
The maximum number of dies in a package.
cpuinfo_x86.topo.die_id:
The physical ID of the die.
cpuinfo_x86.topo.pkg_id:
The physical ID of the package. This information is retrieved via CPUIDand deduced from the APIC IDs of the cores in the package.
Modern systems use this value for the socket. There may be multiplepackages within a socket. This value may differ from topo.die_id.
cpuinfo_x86.topo.logical_pkg_id:
The logical ID of the package. As we do not trust BIOSes to enumerate thepackages in a consistent way, we introduced the concept of logical packageID so we can sanely calculate the number of maximum possible packages inthe system and have the packages enumerated linearly.
topology_max_packages():
The maximum possible number of packages in the system. Helpful for perpackage facilities to preallocate per package information.
cpuinfo_x86.topo.llc_id:
On Intel, the first APIC ID of the list of CPUs sharing the Last LevelCache
On AMD, the Node ID or Core Complex ID containing the Last LevelCache. In general, it is a number identifying an LLC uniquely on thesystem.

4.2.Cores¶

A core consists of 1 or more threads. It does not matter whether the threadsare SMT- or CMT-type threads.

AMDs nomenclature for a CMT core is “Compute Unit”. The kernel always uses“core”.

4.3.Threads¶

A thread is a single scheduling unit. It’s the equivalent to a logical LinuxCPU.

AMDs nomenclature for CMT threads is “Compute Unit Core”. The kernel alwaysuses “thread”.

Thread-related topology information in the kernel:

topology_core_cpumask():
The cpumask contains all online threads in the package to which a threadbelongs.
The number of online threads is also printed in /proc/cpuinfo “siblings.”
topology_sibling_cpumask():
The cpumask contains all online threads in the core to which a threadbelongs.
topology_logical_package_id():
The logical package ID to which a thread belongs.
topology_physical_package_id():
The physical package ID to which a thread belongs.
topology_core_id();
The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo“core_id.”
topology_logical_core_id();
The logical core ID to which a thread belongs.

4.4.System topology enumeration¶

The topology on x86 systems can be discovered using a combination of vendorspecific CPUID leaves which enumerate the processor topology and the cachehierarchy.

The CPUID leaves in their preferred order of parsing for each x86 vendor is asfollows:

AMD
1. CPUID leaf 0x80000026 [Extended CPU Topology] (Core::X86::Cpuid::ExCpuTopology)
  The extended CPUID leaf 0x80000026 is the extension of the CPUID leaf 0xBand provides the topology information of Core, Complex, CCD (Die), andSocket in each level.
  Support for the leaf is discovered by checking if the maximum extendedCPUID level is >= 0x80000026 and then checking ifLogProcAtThisLevelinEBX[15:0] at a particular level (starting from 0) is non-zero.
  TheLevelType inECX[15:8] at the level provides the topology domainthe level describes - Core, Complex, CCD(Die), or the Socket.
  The kernel uses theCoreMaskWidth fromEAX[4:0] to discover thenumber of bits that need to be right-shifted fromExtendedLocalApicIdinEDX[31:0] in order to get a unique Topology ID for the topologylevel. CPUs with the same Topology ID share the resources at that level.
  CPUID leaf 0x80000026 also provides more information regarding the powerand efficiency rankings, and about the core type on AMD processors withheterogeneous characteristics.
  If CPUID leaf 0x80000026 is supported, further parsing is not required.
2. CPUID leaf 0x0000000B [Extended Topology Enumeration] (Core::X86::Cpuid::ExtTopEnum)
  The extended CPUID leaf 0x0000000B is the predecessor on the extendedCPUID leaf 0x80000026 and only describes the core, and the socket domainsof the processor topology.
  The support for the leaf is discovered by checking if the maximum supportedCPUID level is >= 0xB and then ifEBX[31:0] at a particular level(starting from 0) is non-zero.
  TheLevelType inECX[15:8] at the level provides the topology domainthat the level describes - Thread, or Processor (Socket).
  The kernel uses theCoreMaskWidth fromEAX[4:0] to discover thenumber of bits that need to be right-shifted from theExtendedLocalApicIdinEDX[31:0] to get a unique Topology ID for that topology level. CPUssharing the Topology ID share the resources at that level.
  If CPUID leaf 0xB is supported, further parsing is not required.
3. CPUID leaf 0x80000008 ECX [Size Identifiers] (Core::X86::Cpuid::SizeId)
  If neither the CPUID leaf 0x80000026 nor 0xB is supported, the number ofCPUs on the package is detected using the Size Identifier leaf0x80000008 ECX.
  The support for the leaf is discovered by checking if the supportedextended CPUID level is >= 0x80000008.
  The shifts from the APIC ID for the Socket ID is calculated from theApicIdSize field inECX[15:12] if it is non-zero.
  IfApicIdSize is reported to be zero, the shift is calculated as theorder of thenumber of threads calculated fromNC field inECX[7:0] which describes thenumber of threads - 1 on the package.
  Unless Extended APIC ID is supported, the APIC ID used to find theSocket ID is from theLocalApicId field of CPUID leaf 0x00000001EBX[31:24].
  The topology parsing continues to detect if Extended APIC ID issupported or not.
4. CPUID leaf 0x8000001E [Extended APIC ID, Core Identifiers, Node Identifiers](Core::X86::Cpuid::{ExtApicId,CoreId,NodeId})
  The support for Extended APIC ID can be detected by checking for thepresence ofTopologyExtensions inECX[22] of CPUID leaf 0x80000001[Feature Identifiers] (Core::X86::Cpuid::FeatureExtIdEcx).
  If Topology Extensions is supported, the APIC ID fromExtendedApicIdfrom CPUID leaf 0x8000001EEAX[31:0] should be preferred over that fromLocalApicId field of CPUID leaf 0x00000001EBX[31:24] for topologyenumeration.
  On processors of Family 0x17 and above that do not support CPUID leaf0x80000026 or CPUID leaf 0xB, the shifts from the APIC ID for the CoreID is calculated using the order ofnumber of threads per corecalculated using theThreadsPerCore field inEBX[15:8] whichdescribesnumber of threads per core - 1.
  On Processors of Family 0x15, the Core ID fromEBX[7:0] is used as thecu_id (Compute Unit ID) to detect CPUs that share the compute units.
All AMD processors that support theTopologyExtensions feature store theNodeId from theECX[7:0] of CPUID leaf 0x8000001E(Core::X86::Cpuid::NodeId) as the per-CPUnode_id. On older processors,thenode_id was discovered using MSR_FAM10H_NODE_ID MSR (MSR0x0xc001_100c). The presence of the NODE_ID MSR was detected by checkingECX[19] of CPUID leaf 0x80000001 [Feature Identifiers](Core::X86::Cpuid::FeatureExtIdEcx).
Intel
On Intel platforms, the CPUID leaves that enumerate the processortopology are as follows:
1. CPUID leaf 0x1F (V2 Extended Topology Enumeration Leaf)
  The CPUID leaf 0x1F is the extension of the CPUID leaf 0xB and providesthe topology information of Core, Module, Tile, Die, DieGrp, and Socketin each level.
  The support for the leaf is discovered by checking if the supportedCPUID level is >= 0x1F and thenEBX[31:0] at a particular level(starting from 0) is non-zero.
  TheDomain Type inECX[15:8] of the sub-leaf provides the topologydomain that the level describes - Core, Module, Tile, Die, DieGrp, andSocket.
  The kernel uses the value fromEAX[4:0] to discover the number ofbits that need to be right shifted from thex2APIC ID inEDX[31:0]to get a unique Topology ID for the topology level. CPUs with the sameTopology ID share the resources at that level.
  If CPUID leaf 0x1F is supported, further parsing is not required.
2. CPUID leaf 0x0000000B (Extended Topology Enumeration Leaf)
  The extended CPUID leaf 0x0000000B is the predecessor of the V2 ExtendedTopology Enumeration Leaf 0x1F and only describes the core, and thesocket domains of the processor topology.
  The support for the leaf is iscovered by checking if the supported CPUIDlevel is >= 0xB and then checking ifEBX[31:0] at a particular level(starting from 0) is non-zero.
  CPUID leaf 0x0000000B shares the same layout as CPUID leaf 0x1F andshould be enumerated in a similar manner.
  If CPUID leaf 0xB is supported, further parsing is not required.
3. CPUID leaf 0x00000004 (Deterministic Cache Parameters Leaf)
  On Intel processors that support neither CPUID leaf 0x1F, nor CPUID leaf0xB, the shifts for the SMT domains is calculated using the number ofCPUs sharing the L1 cache.
  Processors that feature Hyper-Threading is detected usingEDX[28] ofCPUID leaf 0x1 (Basic CPUID Information).
  The order ofMaximum number of addressable IDs for logical processorssharing this cache fromEAX[25:14] of level-0 of CPUID 0x4 providesthe shifts from the APIC ID required to compute the Core ID.
  The APIC ID and Package information is computed using the data fromCPUID leaf 0x1.
4. CPUID leaf 0x00000001 (Basic CPUID Information)
  The mask and shifts to derive the Physical Package (socket) ID iscomputed using theMaximum number of addressable IDs for logicalprocessors in this physical package fromEBX[23:16] of CPUID leaf0x1.
The APIC ID on the legacy platforms is derived from theInitial APICID field fromEBX[31:24] of CPUID leaf 0x1.
Centaur and Zhaoxin
Similar to Intel, Centaur and Zhaoxin use a combination of CPUID leaf0x00000004 (Deterministic Cache Parameters Leaf) and CPUID leaf 0x00000001(Basic CPUID Information) to derive the topology information.

4.5.System topology examples¶

Note

The alternative Linux CPU enumeration depends on how the BIOS enumerates thethreads. Many BIOSes enumerate all threads 0 first and then all threads 1.That has the “advantage” that the logical Linux CPU numbers of threads 0 staythe same whether threads are enabled or not. That’s merely an implementationdetail and has no practical impact.

Single Package, Single Core:

[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0

Single Package, Dual Core

One thread per core:

[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0            -> [core 1] -> [thread 0] -> Linux CPU 1

Two threads per core:

[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0                        -> [thread 1] -> Linux CPU 1            -> [core 1] -> [thread 0] -> Linux CPU 2                        -> [thread 1] -> Linux CPU 3

Alternative enumeration:

[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0                        -> [thread 1] -> Linux CPU 2            -> [core 1] -> [thread 0] -> Linux CPU 1                        -> [thread 1] -> Linux CPU 3

AMD nomenclature for CMT systems:

[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0                             -> [Compute Unit Core 1] -> Linux CPU 1         -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2                             -> [Compute Unit Core 1] -> Linux CPU 3

Dual Package, Dual Core