Purpose of local atomic operations¶

Local atomic operations are meant to provide fast and highly reentrant per CPUcounters. They minimize the performance cost of standard atomic operations byremoving the LOCK prefix and memory barriers normally required to synchronizeacross CPUs.

Having fast per CPU atomic counters is interesting in many cases: it does notrequire disabling interrupts to protect from interrupt handlers and it permitscoherent counters in NMI handlers. It is especially useful for tracing purposesand for various performance monitoring counters.

Local atomic operations only guarantee variable modification atomicity wrt theCPU which owns the data. Therefore, care must taken to make sure that only oneCPU writes to thelocal_t data. This is done by using per cpu data andmaking sure that we modify it from within a preemption safe context. It ishowever permitted to readlocal_t data from any CPU: it will then appear tobe written out of order wrt other memory writes by the owner CPU.

Implementation for a given architecture¶

It can be done by slightly modifying the standard atomic operations: onlytheir UP variant must be kept. It typically means removing LOCK prefix (oni386 and x86_64) and any SMP synchronization barrier. If the architecture doesnot have a different behavior between SMP and UP, includingasm-generic/local.h in your architecture’slocal.h is sufficient.

Thelocal_t type is defined as an opaquesignedlong by embedding anatomic_long_t inside a structure. This is made so a cast from this type toalong fails. The definition looks like:

typedef struct { atomic_long_t a; } local_t;

Rules to follow when using local atomic operations¶

• Variables touched by local ops must be per cpu variables.

• Only the CPU owner of these variables must write to them.

• This CPU can use local ops from any context (process, irq, softirq, nmi, ...)to update itslocal_t variables.

• Preemption (or interrupts) must be disabled when using local ops inprocess context to make sure the process won’t be migrated to adifferent CPU between getting the per-cpu variable and doing theactual local op.

• When using local ops in interrupt context, no special care must betaken on a mainline kernel, since they will run on the local CPU withpreemption already disabled. I suggest, however, to explicitlydisable preemption anyway to make sure it will still work correctly on-rt kernels.

• Reading the local cpu variable will provide the current copy of thevariable.

• Reads of these variables can be done from any CPU, because updates to“long”, aligned, variables are always atomic. Since no memorysynchronization is done by the writer CPU, an outdated copy of thevariable can be read when reading someother cpu’s variables.

How to use local atomic operations¶

#include <linux/percpu.h>#include <asm/local.h>static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0);

Counting¶

Counting is done on all the bits of a signed long.

In preemptible context, useget_cpu_var() andput_cpu_var() aroundlocal atomic operations: it makes sure that preemption is disabled around writeaccess to the per cpu variable. For instance:

local_inc(&get_cpu_var(counters));put_cpu_var(counters);

If you are already in a preemption-safe context, you can usethis_cpu_ptr() instead:

local_inc(this_cpu_ptr(&counters));

Reading the counters¶

Those local counters can be read from foreign CPUs to sum the count. Note thatthe data seen by local_read across CPUs must be considered to be out of orderrelatively to other memory writes happening on the CPU that owns the data:

long sum = 0;for_each_online_cpu(cpu)        sum += local_read(&per_cpu(counters, cpu));

If you want to use a remote local_read to synchronize access to a resourcebetween CPUs, explicitsmp_wmb() andsmp_rmb() memory barriers must be usedrespectively on the writer and the reader CPUs. It would be the case if you usethelocal_t variable as a counter of bytes written in a buffer: there shouldbe asmp_wmb() between the buffer write and the counter increment and also asmp_rmb() between the counter read and the buffer read.

Here is a sample module which implements a basic per cpu counter usinglocal.h:

/* test-local.c * * Sample module for local.h usage. */#include <asm/local.h>#include <linux/module.h>#include <linux/timer.h>static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0);static struct timer_list test_timer;/* IPI called on each CPU. */static void test_each(void *info){        /* Increment the counter from a non preemptible context */        printk("Increment on cpu %d\n", smp_processor_id());        local_inc(this_cpu_ptr(&counters));        /* This is what incrementing the variable would look like within a         * preemptible context (it disables preemption) :         *         * local_inc(&get_cpu_var(counters));         * put_cpu_var(counters);         */}static void do_test_timer(unsigned long data){        int cpu;        /* Increment the counters */        on_each_cpu(test_each, NULL, 1);        /* Read all the counters */        printk("Counters read from CPU %d\n", smp_processor_id());        for_each_online_cpu(cpu) {                printk("Read : CPU %d, count %ld\n", cpu,                        local_read(&per_cpu(counters, cpu)));        }        mod_timer(&test_timer, jiffies + 1000);}static int __init test_init(void){        /* initialize the timer that will increment the counter */        timer_setup(&test_timer, do_test_timer, 0);        mod_timer(&test_timer, jiffies + 1);        return 0;}static void __exit test_exit(void){        timer_shutdown_sync(&test_timer);}module_init(test_init);module_exit(test_exit);MODULE_LICENSE("GPL");MODULE_AUTHOR("Mathieu Desnoyers");MODULE_DESCRIPTION("Local Atomic Ops");