The Concept of CPU Performance Scaling¶

The majority of modern processors are capable of operating in a number ofdifferent clock frequency and voltage configurations, often referred to asOperating Performance Points or P-states (in ACPI terminology). As a rule,the higher the clock frequency and the higher the voltage, the more instructionscan be retired by the CPU over a unit of time, but also the higher the clockfrequency and the higher the voltage, the more energy is consumed over a unit oftime (or the more power is drawn) by the CPU in the given P-state. Thereforethere is a natural tradeoff between the CPU capacity (the number of instructionsthat can be executed over a unit of time) and the power drawn by the CPU.

In some situations it is desirable or even necessary to run the program as fastas possible and then there is no reason to use any P-states different from thehighest one (i.e. the highest-performance frequency/voltage configurationavailable). In some other cases, however, it may not be necessary to executeinstructions so quickly and maintaining the highest available CPU capacity for arelatively long time without utilizing it entirely may be regarded as wasteful.It also may not be physically possible to maintain maximum CPU capacity for toolong for thermal or power supply capacity reasons or similar. To cover thosecases, there are hardware interfaces allowing CPUs to be switched betweendifferent frequency/voltage configurations or (in the ACPI terminology) to beput into different P-states.

Typically, they are used along with algorithms to estimate the required CPUcapacity, so as to decide which P-states to put the CPUs into. Of course, sincethe utilization of the system generally changes over time, that has to be donerepeatedly on a regular basis. The activity by which this happens is referredto as CPU performance scaling or CPU frequency scaling (because it involvesadjusting the CPU clock frequency).

CPU Performance Scaling in Linux¶

The Linux kernel supports CPU performance scaling by means of theCPUFreq(CPU Frequency scaling) subsystem that consists of three layers of code: thecore, scaling governors and scaling drivers.

TheCPUFreq core provides the common code infrastructure and user spaceinterfaces for all platforms that support CPU performance scaling. It definesthe basic framework in which the other components operate.

Scaling governors implement algorithms to estimate the required CPU capacity.As a rule, each governor implements one, possibly parametrized, scalingalgorithm.

Scaling drivers talk to the hardware. They provide scaling governors withinformation on the available P-states (or P-state ranges in some cases) andaccess platform-specific hardware interfaces to change CPU P-states as requestedby scaling governors.

In principle, all available scaling governors can be used with every scalingdriver. That design is based on the observation that the information used byperformance scaling algorithms for P-state selection can be represented in aplatform-independent form in the majority of cases, so it should be possibleto use the same performance scaling algorithm implemented in exactly the sameway regardless of which scaling driver is used. Consequently, the same set ofscaling governors should be suitable for every supported platform.

However, that observation may not hold for performance scaling algorithmsbased on information provided by the hardware itself, for example throughfeedback registers, as that information is typically specific to the hardwareinterface it comes from and may not be easily represented in an abstract,platform-independent way. For this reason,CPUFreq allows scaling driversto bypass the governor layer and implement their own performance scalingalgorithms. That is done by theintel_pstate scaling driver.

CPUFreq Policy Objects¶

In some cases the hardware interface for P-state control is shared by multipleCPUs. That is, for example, the same register (or set of registers) is used tocontrol the P-state of multiple CPUs at the same time and writing to it affectsall of those CPUs simultaneously.

Sets of CPUs sharing hardware P-state control interfaces are represented byCPUFreq asstructcpufreq_policy objects. For consistency,structcpufreq_policy is also used when there is only one CPU in the givenset.

TheCPUFreq core maintains a pointer to astructcpufreq_policy object forevery CPU in the system, including CPUs that are currently offline. If multipleCPUs share the same hardware P-state control interface, all of the pointerscorresponding to them point to the samestructcpufreq_policy object.

CPUFreq usesstructcpufreq_policy as its basic data type and the designof its user space interface is based on the policy concept.

CPU Initialization¶

First of all, a scaling driver has to be registered forCPUFreq to work.It is only possible to register one scaling driver at a time, so the scalingdriver is expected to be able to handle all CPUs in the system.

The scaling driver may be registered before or after CPU registration. IfCPUs are registered earlier, the driver core invokes theCPUFreq core totake a note of all of the already registered CPUs during the registration of thescaling driver. In turn, if any CPUs are registered after the registration ofthe scaling driver, theCPUFreq core will be invoked to take note of themat their registration time.

In any case, theCPUFreq core is invoked to take note of any logical CPU ithas not seen so far as soon as it is ready to handle that CPU. [Note that thelogical CPU may be a physical single-core processor, or a single core in amulticore processor, or a hardware thread in a physical processor or processorcore. In what follows “CPU” always means “logical CPU” unless explicitly statedotherwise and the word “processor” is used to refer to the physical partpossibly including multiple logical CPUs.]

Once invoked, theCPUFreq core checks if the policy pointer is already setfor the given CPU and if so, it skips the policy object creation. Otherwise,a new policy object is created and initialized, which involves the creation ofa new policy directory insysfs, and the policy pointer corresponding tothe given CPU is set to the new policy object’s address in memory.

Next, the scaling driver’s->init() callback is invoked with the policypointer of the new CPU passed to it as the argument. That callback is expectedto initialize the performance scaling hardware interface for the given CPU (or,more precisely, for the set of CPUs sharing the hardware interface it belongsto, represented by its policy object) and, if the policy object it has beencalled for is new, to set parameters of the policy, like the minimum and maximumfrequencies supported by the hardware, the table of available frequencies (ifthe set of supported P-states is not a continuous range), and the mask of CPUsthat belong to the same policy (including both online and offline CPUs). Thatmask is then used by the core to populate the policy pointers for all of theCPUs in it.

The next major initialization step for a new policy object is to attach ascaling governor to it (to begin with, that is the default scaling governordetermined by the kernel command line or configuration, but it may be changedlater viasysfs). First, a pointer to the new policy object is passed tothe governor’s->init() callback which is expected to initialize all of thedata structures necessary to handle the given policy and, possibly, to adda governorsysfs interface to it. Next, the governor is started byinvoking its->start() callback.

That callback is expected to register per-CPU utilization update callbacks forall of the online CPUs belonging to the given policy with the CPU scheduler.The utilization update callbacks will be invoked by the CPU scheduler onimportant events, like task enqueue and dequeue, on every iteration of thescheduler tick or generally whenever the CPU utilization may change (from thescheduler’s perspective). They are expected to carry out computations neededto determine the P-state to use for the given policy going forward and toinvoke the scaling driver to make changes to the hardware in accordance withthe P-state selection. The scaling driver may be invoked directly fromscheduler context or asynchronously, via a kernel thread or workqueue, dependingon the configuration and capabilities of the scaling driver and the governor.

Similar steps are taken for policy objects that are not new, but were “inactive”previously, meaning that all of the CPUs belonging to them were offline. Theonly practical difference in that case is that theCPUFreq core will attemptto use the scaling governor previously used with the policy that became“inactive” (and is re-initialized now) instead of the default governor.

In turn, if a previously offline CPU is being brought back online, but someother CPUs sharing the policy object with it are online already, there is noneed to re-initialize the policy object at all. In that case, it only isnecessary to restart the scaling governor so that it can take the new online CPUinto account. That is achieved by invoking the governor’s->stop and->start() callbacks, in this order, for the entire policy.

As mentioned before, theintel_pstate scaling driver bypasses the scalinggovernor layer ofCPUFreq and provides its own P-state selection algorithms.Consequently, ifintel_pstate is used, scaling governors are not attached tonew policy objects. Instead, the driver’s->setpolicy() callback is invokedto register per-CPU utilization update callbacks for each policy. Thesecallbacks are invoked by the CPU scheduler in the same way as for scalinggovernors, but in theintel_pstate case they both determine the P-state touse and change the hardware configuration accordingly in one go from schedulercontext.

The policy objects created during CPU initialization and other data structuresassociated with them are torn down when the scaling driver is unregistered(which happens when the kernel module containing it is unloaded, for example) orwhen the last CPU belonging to the given policy in unregistered.

Policy Interface insysfs¶

During the initialization of the kernel, theCPUFreq core creates asysfs directory (kobject) calledcpufreq under/sys/devices/system/cpu/.

That directory contains apolicyX subdirectory (whereX represents aninteger number) for every policy object maintained by theCPUFreq core.EachpolicyX directory is pointed to bycpufreq symbolic linksunder/sys/devices/system/cpu/cpuY/ (whereY represents an integerthat may be different from the one represented byX) for all of the CPUsassociated with (or belonging to) the given policy. ThepolicyX directoriesin/sys/devices/system/cpu/cpufreq each contain policy-specificattributes (files) to controlCPUFreq behavior for the corresponding policyobjects (that is, for all of the CPUs associated with them).

Some of those attributes are generic. They are created by theCPUFreq coreand their behavior generally does not depend on what scaling driver is in useand what scaling governor is attached to the given policy. Some scaling driversalso add driver-specific attributes to the policy directories insysfs tocontrol policy-specific aspects of driver behavior.

The generic attributes under/sys/devices/system/cpu/cpufreq/policyX/are the following:

affected_cpus

List of online CPUs belonging to this policy (i.e. sharing the hardwareperformance scaling interface represented by thepolicyX policyobject).

bios_limit

If the platform firmware (BIOS) tells the OS to apply an upper limit toCPU frequencies, that limit will be reported through this attribute (ifpresent).

The existence of the limit may be a result of some (often unintentional)BIOS settings, restrictions coming from a service processor or otherBIOS/HW-based mechanisms.

This does not cover ACPI thermal limitations which can be discoveredthrough a generic thermal driver.

This attribute is not present if the scaling driver in use does notsupport it.

cpuinfo_cur_freq

Current frequency of the CPUs belonging to this policy as obtained fromthe hardware (in KHz).

This is expected to be the frequency the hardware actually runs at.If that frequency cannot be determined, this attribute should notbe present.

cpuinfo_avg_freq

An average frequency (in KHz) of all CPUs belonging to a given policy,derived from a hardware provided feedback and reported on a time framespanning at most few milliseconds.

This is expected to be based on the frequency the hardware actually runsat and, as such, might require specialised hardware support (such as AMUextension on ARM). If one cannot be determined, this attribute shouldnot be present.

Note that failed attempt to retrieve current frequency for a givenCPU(s) will result in an appropriate error, i.e.: EAGAIN for CPU thatremains idle (raised on ARM).

cpuinfo_max_freq

Maximum possible operating frequency the CPUs belonging to this policycan run at (in kHz).

cpuinfo_min_freq

Minimum possible operating frequency the CPUs belonging to this policycan run at (in kHz).

cpuinfo_transition_latency

The time it takes to switch the CPUs belonging to this policy from oneP-state to another, in nanoseconds.

related_cpus

List of all (online and offline) CPUs belonging to this policy.

scaling_available_frequencies

List of available frequencies of the CPUs belonging to this policy(in kHz).

scaling_available_governors

List ofCPUFreq scaling governors present in the kernel that canbe attached to this policy or (if theintel_pstate scaling driver isin use) list of scaling algorithms provided by the driver that can beapplied to this policy.

[Note that some governors are modular and it may be necessary to load akernel module for the governor held by it to become available and belisted by this attribute.]

scaling_cur_freq

Current frequency of all of the CPUs belonging to this policy (in kHz).

In the majority of cases, this is the frequency of the last P-staterequested by the scaling driver from the hardware using the scalinginterface provided by it, which may or may not reflect the frequencythe CPU is actually running at (due to hardware design and otherlimitations).

Some architectures (e.g.x86) may attempt to provide informationmore precisely reflecting the current CPU frequency through thisattribute, but that still may not be the exact current CPU frequency asseen by the hardware at the moment. This behavior though, is onlyavailable via c:macro:CPUFREQ_ARCH_CUR_FREQ option.

scaling_driver

The scaling driver currently in use.

scaling_governor

The scaling governor currently attached to this policy or (if theintel_pstate scaling driver is in use) the scaling algorithmprovided by the driver that is currently applied to this policy.

This attribute is read-write and writing to it will cause a new scalinggovernor to be attached to this policy or a new scaling algorithmprovided by the scaling driver to be applied to it (in theintel_pstate case), as indicated by the string written to thisattribute (which must be one of the names listed by thescaling_available_governors attribute described above).

scaling_max_freq

Maximum frequency the CPUs belonging to this policy are allowed to berunning at (in kHz).

This attribute is read-write and writing a string representing aninteger to it will cause a new limit to be set (it must not be lowerthan the value of thescaling_min_freq attribute).

scaling_min_freq

Minimum frequency the CPUs belonging to this policy are allowed to berunning at (in kHz).

This attribute is read-write and writing a string representing anon-negative integer to it will cause a new limit to be set (it must notbe higher than the value of thescaling_max_freq attribute).

scaling_setspeed

This attribute is functional only if theuserspace scaling governoris attached to the given policy.

It returns the last frequency requested by the governor (in kHz) or canbe written to in order to set a new frequency for the policy.

Generic Scaling Governors¶

CPUFreq provides generic scaling governors that can be used with allscaling drivers. As stated before, each of them implements a single, possiblyparametrized, performance scaling algorithm.

Scaling governors are attached to policy objects and different policy objectscan be handled by different scaling governors at the same time (although thatmay lead to suboptimal results in some cases).

The scaling governor for a given policy object can be changed at any time withthe help of thescaling_governor policy attribute insysfs.

Some governors exposesysfs attributes to control or fine-tune the scalingalgorithms implemented by them. Those attributes, referred to as governortunables, can be either global (system-wide) or per-policy, depending on thescaling driver in use. If the driver requires governor tunables to beper-policy, they are located in a subdirectory of each policy directory.Otherwise, they are located in a subdirectory under/sys/devices/system/cpu/cpufreq/. In either case the name of thesubdirectory containing the governor tunables is the name of the governorproviding them.

Frequency Boost Support¶

References¶

Jonathan Corbet,Per-entity load tracking,https://lwn.net/Articles/531853/