Dynamic DMA mapping using the generic device¶
| Author: | James E.J. Bottomley <James.Bottomley@HansenPartnership.com> |
|---|
This document describes the DMA API. For a more gentle introductionof the API (and actual examples), see Documentation/DMA-API-HOWTO.txt.
This API is split into two pieces. Part I describes the basic API.Part II describes extensions for supporting non-consistent memorymachines. Unless you know that your driver absolutely has to supportnon-consistent platforms (this is usually only legacy platforms) youshould only use the API described in part I.
Part I - dma_API¶
To get the dma_API, you must #include <linux/dma-mapping.h>. Thisprovides dma_addr_t and the interfaces described below.
A dma_addr_t can hold any valid DMA address for the platform. It can begiven to a device to use as a DMA source or target. A CPU cannot referencea dma_addr_t directly because there may be translation between its physicaladdress space and the DMA address space.
Part Ia - Using large DMA-coherent buffers¶
void *dma_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag)
Consistent memory is memory for which a write by either the device orthe processor can immediately be read by the processor or devicewithout having to worry about caching effects. (You may however needto make sure to flush the processor’s write buffers before tellingdevices to read that memory.)
This routine allocates a region of <size> bytes of consistent memory.
It returns a pointer to the allocated region (in the processor’s virtualaddress space) or NULL if the allocation failed.
It also returns a <dma_handle> which may be cast to an unsigned integer thesame width as the bus and given to the device as the DMA address base ofthe region.
Note: consistent memory can be expensive on some platforms, and theminimum allocation length may be as big as a page, so you shouldconsolidate your requests for consistent memory as much as possible.The simplest way to do that is to use the dma_pool calls (see below).
The flag parameter (dma_alloc_coherent() only) allows the caller tospecify theGFP_ flags (seekmalloc()) for the allocation (theimplementation may choose to ignore flags that affect the location ofthe returned memory, like GFP_DMA).
voiddma_free_coherent(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_handle)
Free a region of consistent memory you previously allocated. dev,size and dma_handle must all be the same as those passed intodma_alloc_coherent(). cpu_addr must be the virtual address returned bythe dma_alloc_coherent().
Note that unlike their sibling allocation calls, these routinesmay only be called with IRQs enabled.
Part Ib - Using small DMA-coherent buffers¶
To get this part of the dma_API, you must #include <linux/dmapool.h>
Many drivers need lots of small DMA-coherent memory regions for DMAdescriptors or I/O buffers. Rather than allocating in units of a pageor more using dma_alloc_coherent(), you can use DMA pools. These workmuch like a struct kmem_cache, except that they use the DMA-coherent allocator,not __get_free_pages(). Also, they understand common hardware constraintsfor alignment, like queue heads needing to be aligned on N-byte boundaries.
struct dma_pool *dma_pool_create(const char *name, struct device *dev, size_t size, size_t align, size_t alloc);
dma_pool_create() initializes a pool of DMA-coherent buffersfor use with a given device. It must be called in a context whichcan sleep.
The “name” is for diagnostics (like a struct kmem_cache name); dev and sizeare like what you’d pass to dma_alloc_coherent(). The device’s hardwarealignment requirement for this type of data is “align” (which is expressedin bytes, and must be a power of two). If your device has no boundarycrossing restrictions, pass 0 for alloc; passing 4096 says memory allocatedfrom this pool must not cross 4KByte boundaries.
void *dma_pool_zalloc(struct dma_pool *pool, gfp_t mem_flags, dma_addr_t *handle)
Wrapsdma_pool_alloc() and also zeroes the returned memory if theallocation attempt succeeded.
void *dma_pool_alloc(struct dma_pool *pool, gfp_t gfp_flags, dma_addr_t *dma_handle);
This allocates memory from the pool; the returned memory will meet thesize and alignment requirements specified at creation time. PassGFP_ATOMIC to prevent blocking, or if it’s permitted (notin_interrupt, not holding SMP locks), pass GFP_KERNEL to allowblocking. Like dma_alloc_coherent(), this returns two values: anaddress usable by the CPU, and the DMA address usable by the pool’sdevice.
voiddma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t addr);
This puts memory back into the pool. The pool is what was passed todma_pool_alloc(); the CPU (vaddr) and DMA addresses are whatwere returned when that routine allocated the memory being freed.
voiddma_pool_destroy(struct dma_pool *pool);
dma_pool_destroy() frees the resources of the pool. It must becalled in a context which can sleep. Make sure you’ve freed all allocatedmemory back to the pool before you destroy it.
Part Ic - DMA addressing limitations¶
intdma_set_mask_and_coherent(struct device *dev, u64 mask)
Checks to see if the mask is possible and updates the devicestreaming and coherent DMA mask parameters if it is.
Returns: 0 if successful and a negative error if not.
intdma_set_mask(struct device *dev, u64 mask)
Checks to see if the mask is possible and updates the deviceparameters if it is.
Returns: 0 if successful and a negative error if not.
intdma_set_coherent_mask(struct device *dev, u64 mask)
Checks to see if the mask is possible and updates the deviceparameters if it is.
Returns: 0 if successful and a negative error if not.
u64dma_get_required_mask(struct device *dev)
This API returns the mask that the platform requires tooperate efficiently. Usually this means the returned maskis the minimum required to cover all of memory. Examining therequired mask gives drivers with variable descriptor sizes theopportunity to use smaller descriptors as necessary.
Requesting the required mask does not alter the current mask. If youwish to take advantage of it, you should issue a dma_set_mask()call to set the mask to the value returned.
size_tdma_max_mapping_size(struct device *dev);
Returns the maximum size of a mapping for the device. The size parameterof the mapping functions like dma_map_single(), dma_map_page() andothers should not be larger than the returned value.
booldma_need_sync(struct device *dev, dma_addr_t dma_addr);
Returns %true if dma_sync_single_for_{device,cpu} calls are required totransfer memory ownership. Returns %false if those calls can be skipped.
unsigned longdma_get_merge_boundary(struct device *dev);
Returns the DMA merge boundary. If the device cannot merge any the DMA addresssegments, the function returns 0.
Part Id - Streaming DMA mappings¶
dma_addr_tdma_map_single(struct device *dev, void *cpu_addr, size_t size, enum dma_data_direction direction)
Maps a piece of processor virtual memory so it can be accessed by thedevice and returns the DMA address of the memory.
The direction for both APIs may be converted freely by casting.However the dma_API uses a strongly typed enumerator for itsdirection:
| DMA_NONE | no direction (used for debugging) |
| DMA_TO_DEVICE | data is going from the memory to the device |
| DMA_FROM_DEVICE | data is coming from the device to the memory |
| DMA_BIDIRECTIONAL | direction isn’t known |
Note
Not all memory regions in a machine can be mapped by this API.Further, contiguous kernel virtual space may not be contiguous asphysical memory. Since this API does not provide any scatter/gathercapability, it will fail if the user tries to map a non-physicallycontiguous piece of memory. For this reason, memory to be mapped bythis API should be obtained from sources which guarantee it to bephysically contiguous (like kmalloc).
Further, the DMA address of the memory must be within thedma_mask of the device (the dma_mask is a bit mask of theaddressable region for the device, i.e., if the DMA address ofthe memory ANDed with the dma_mask is still equal to the DMAaddress, then the device can perform DMA to the memory). Toensure that the memory allocated by kmalloc is within the dma_mask,the driver may specify various platform-dependent flags to restrictthe DMA address range of the allocation (e.g., on x86, GFP_DMAguarantees to be within the first 16MB of available DMA addresses,as required by ISA devices).
Note also that the above constraints on physical contiguity anddma_mask may not apply if the platform has an IOMMU (a device whichmaps an I/O DMA address to a physical memory address). However, to beportable, device driver writers maynot assume that such an IOMMUexists.
Warning
Memory coherency operates at a granularity called the cacheline width. In order for memory mapped by this API to operatecorrectly, the mapped region must begin exactly on a cache lineboundary and end exactly on one (to prevent two separately mappedregions from sharing a single cache line). Since the cache line sizemay not be known at compile time, the API will not enforce thisrequirement. Therefore, it is recommended that driver writers whodon’t take special care to determine the cache line size at run timeonly map virtual regions that begin and end on page boundaries (whichare guaranteed also to be cache line boundaries).
DMA_TO_DEVICE synchronisation must be done after the last modificationof the memory region by the software and before it is handed off tothe device. Once this primitive is used, memory covered by thisprimitive should be treated as read-only by the device. If the devicemay write to it at any point, it should be DMA_BIDIRECTIONAL (seebelow).
DMA_FROM_DEVICE synchronisation must be done before the driveraccesses data that may be changed by the device. This memory shouldbe treated as read-only by the driver. If the driver needs to writeto it at any point, it should be DMA_BIDIRECTIONAL (see below).
DMA_BIDIRECTIONAL requires special handling: it means that the driverisn’t sure if the memory was modified before being handed off to thedevice and also isn’t sure if the device will also modify it. Thus,you must always sync bidirectional memory twice: once before thememory is handed off to the device (to make sure all memory changesare flushed from the processor) and once before the data may beaccessed after being used by the device (to make sure any processorcache lines are updated with data that the device may have changed).
voiddma_unmap_single(struct device *dev, dma_addr_t dma_addr, size_t size, enum dma_data_direction direction)
Unmaps the region previously mapped. All the parameters passed inmust be identical to those passed in (and returned) by the mappingAPI.
dma_addr_tdma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction direction)voiddma_unmap_page(struct device *dev, dma_addr_t dma_address, size_t size, enum dma_data_direction direction)
API for mapping and unmapping for pages. All the notes and warningsfor the other mapping APIs apply here. Also, although the <offset>and <size> parameters are provided to do partial page mapping, it isrecommended that you never use these unless you really know what thecache width is.
dma_addr_tdma_map_resource(struct device *dev, phys_addr_t phys_addr, size_t size, enum dma_data_direction dir, unsigned long attrs)voiddma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs)
API for mapping and unmapping for MMIO resources. All the notes andwarnings for the other mapping APIs apply here. The API should only beused to map device MMIO resources, mapping of RAM is not permitted.
intdma_mapping_error(struct device *dev, dma_addr_t dma_addr)
In some circumstances dma_map_single(), dma_map_page() and dma_map_resource()will fail to create a mapping. A driver can check for these errors by testingthe returned DMA address with dma_mapping_error(). A non-zero return valuemeans the mapping could not be created and the driver should take appropriateaction (e.g. reduce current DMA mapping usage or delay and try again later).
intdma_map_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction direction)
Returns: the number of DMA address segments mapped (this may be shorterthan <nents> passed in if some elements of the scatter/gather list arephysically or virtually adjacent and an IOMMU maps them with a singleentry).
Please note that the sg cannot be mapped again if it has been mapped once.The mapping process is allowed to destroy information in the sg.
As with the other mapping interfaces, dma_map_sg() can fail. When itdoes, 0 is returned and a driver must take appropriate action. It iscritical that the driver do something, in the case of a block driveraborting the request or even oopsing is better than doing nothing andcorrupting the filesystem.
With scatterlists, you use the resulting mapping like this:
int i, count = dma_map_sg(dev, sglist, nents, direction);struct scatterlist *sg;for_each_sg(sglist, sg, count, i) { hw_address[i] = sg_dma_address(sg); hw_len[i] = sg_dma_len(sg);}where nents is the number of entries in the sglist.
The implementation is free to merge several consecutive sglist entriesinto one (e.g. with an IOMMU, or if several pages just happen to bephysically contiguous) and returns the actual number of sg entries itmapped them to. On failure 0, is returned.
Then you should loop count times (note: this can be less than nents times)and use sg_dma_address() and sg_dma_len() macros where you previouslyaccessed sg->address and sg->length as shown above.
voiddma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction direction)
Unmap the previously mapped scatter/gather list. All the parametersmust be the same as those and passed in to the scatter/gather mappingAPI.
Note: <nents> must be the number you passed in,not the number ofDMA address entries returned.
voiddma_sync_single_for_cpu(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction direction)voiddma_sync_single_for_device(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction direction)voiddma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction direction)voiddma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction direction)
Synchronise a single contiguous or scatter/gather mapping for the CPUand device. With the sync_sg API, all the parameters must be the sameas those passed into the single mapping API. With the sync_single API,you can use dma_handle and size parameters that aren’t identical tothose passed into the single mapping API to do a partial sync.
Note
You must do this:
- Before reading values that have been written by DMA from the device(use the DMA_FROM_DEVICE direction)
- After writing values that will be written to the device using DMA(use the DMA_TO_DEVICE) direction
- beforeand after handing memory to the device if the memory isDMA_BIDIRECTIONAL
See also dma_map_single().
dma_addr_tdma_map_single_attrs(struct device *dev, void *cpu_addr, size_t size, enum dma_data_direction dir, unsigned long attrs)voiddma_unmap_single_attrs(struct device *dev, dma_addr_t dma_addr, size_t size, enum dma_data_direction dir, unsigned long attrs)intdma_map_sg_attrs(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs)voiddma_unmap_sg_attrs(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs)
The four functions above are just like the counterpart functionswithout the _attrs suffixes, except that they pass an optionaldma_attrs.
The interpretation of DMA attributes is architecture-specific, andeach attribute should be documented in Documentation/DMA-attributes.txt.
If dma_attrs are 0, the semantics of each of these functionsis identical to those of the corresponding functionwithout the _attrs suffix. As a result dma_map_single_attrs()can generally replace dma_map_single(), etc.
As an example of the use of the*_attrs functions, here’s howyou could pass an attribute DMA_ATTR_FOO when mapping memoryfor DMA:
#include <linux/dma-mapping.h>/* DMA_ATTR_FOO should be defined in linux/dma-mapping.h and* documented in Documentation/DMA-attributes.txt */... unsigned long attr; attr |= DMA_ATTR_FOO; .... n = dma_map_sg_attrs(dev, sg, nents, DMA_TO_DEVICE, attr); ....
Architectures that care about DMA_ATTR_FOO would check for itspresence in their implementations of the mapping and unmappingroutines, e.g.::
void whizco_dma_map_sg_attrs(struct device *dev, dma_addr_t dma_addr, size_t size, enum dma_data_direction dir, unsigned long attrs){ .... if (attrs & DMA_ATTR_FOO) /* twizzle the frobnozzle */ ....}Part II - Advanced dma usage¶
Warning: These pieces of the DMA API should not be used in themajority of cases, since they cater for unlikely corner cases thatdon’t belong in usual drivers.
If you don’t understand how cache line coherency works between aprocessor and an I/O device, you should not be using this part of theAPI at all.
void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs)
Identical to dma_alloc_coherent() except that when theDMA_ATTR_NON_CONSISTENT flags is passed in the attrs argument, theplatform will choose to return either consistent or non-consistent memoryas it sees fit. By using this API, you are guaranteeing to the platformthat you have all the correct and necessary sync points for this memoryin the driver should it choose to return non-consistent memory.
Note: where the platform can return consistent memory, it willguarantee that the sync points become nops.
Warning: Handling non-consistent memory is a real pain. You shouldonly use this API if you positively know your driver will berequired to work on one of the rare (usually non-PCI) architecturesthat simply cannot make consistent memory.
voiddma_free_attrs(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_handle, unsigned long attrs)
Free memory allocated by the dma_alloc_attrs(). All commonparameters must be identical to those otherwise passed to dma_free_coherent,and the attrs argument must be identical to the attrs passed todma_alloc_attrs().
intdma_get_cache_alignment(void)
Returns the processor cache alignment. This is the absolute minimumalignmentand width that you must observe when either mappingmemory or doing partial flushes.
Note
This API may return a numberlarger than the actual cacheline, but it will guarantee that one or more cache lines fit exactlyinto the width returned by this call. It will also always be a powerof two for easy alignment.
voiddma_cache_sync(struct device *dev, void *vaddr, size_t size, enum dma_data_direction direction)
Do a partial sync of memory that was allocated by dma_alloc_attrs() withthe DMA_ATTR_NON_CONSISTENT flag starting at virtual address vaddr andcontinuing on for size. Again, youmust observe the cache lineboundaries when doing this.
intdma_declare_coherent_memory(struct device *dev, phys_addr_t phys_addr, dma_addr_t device_addr, size_t size);
Declare region of memory to be handed out by dma_alloc_coherent() whenit’s asked for coherent memory for this device.
phys_addr is the CPU physical address to which the memory is currentlyassigned (this will be ioremapped so the CPU can access the region).
device_addr is the DMA address the device needs to be programmedwith to actually address this memory (this will be handed out as thedma_addr_t in dma_alloc_coherent()).
size is the size of the area (must be multiples of PAGE_SIZE).
As a simplification for the platforms, onlyone such region ofmemory may be declared per device.
For reasons of efficiency, most platforms choose to track the declaredregion only at the granularity of a page. For smaller allocations,you should use the dma_pool() API.
Part III - Debug drivers use of the DMA-API¶
The DMA-API as described above has some constraints. DMA addresses must bereleased with the corresponding function with the same size for example. Withthe advent of hardware IOMMUs it becomes more and more important that driversdo not violate those constraints. In the worst case such a violation canresult in data corruption up to destroyed filesystems.
To debug drivers and find bugs in the usage of the DMA-API checking code canbe compiled into the kernel which will tell the developer about thoseviolations. If your architecture supports it you can select the “Enabledebugging of DMA-API usage” option in your kernel configuration. Enabling thisoption has a performance impact. Do not enable it in production kernels.
If you boot the resulting kernel will contain code which does some bookkeepingabout what DMA memory was allocated for which device. If this code detects anerror it prints a warning message with some details into your kernel log. Anexample warning message may look like this:
WARNING: at /data2/repos/linux-2.6-iommu/lib/dma-debug.c:448 check_unmap+0x203/0x490()Hardware name:forcedeth 0000:00:08.0: DMA-API: device driver frees DMA memory with wrong function [device address=0x00000000640444be] [size=66 bytes] [mapped assingle] [unmapped as page]Modules linked in: nfsd exportfs bridge stp llc r8169Pid: 0, comm: swapper Tainted: G W 2.6.28-dmatest-09289-g8bb99c0 #1Call Trace:<IRQ> [<ffffffff80240b22>] warn_slowpath+0xf2/0x130[<ffffffff80647b70>] _spin_unlock+0x10/0x30[<ffffffff80537e75>] usb_hcd_link_urb_to_ep+0x75/0xc0[<ffffffff80647c22>] _spin_unlock_irqrestore+0x12/0x40[<ffffffff8055347f>] ohci_urb_enqueue+0x19f/0x7c0[<ffffffff80252f96>] queue_work+0x56/0x60[<ffffffff80237e10>] enqueue_task_fair+0x20/0x50[<ffffffff80539279>] usb_hcd_submit_urb+0x379/0xbc0[<ffffffff803b78c3>] cpumask_next_and+0x23/0x40[<ffffffff80235177>] find_busiest_group+0x207/0x8a0[<ffffffff8064784f>] _spin_lock_irqsave+0x1f/0x50[<ffffffff803c7ea3>] check_unmap+0x203/0x490[<ffffffff803c8259>] debug_dma_unmap_page+0x49/0x50[<ffffffff80485f26>] nv_tx_done_optimized+0xc6/0x2c0[<ffffffff80486c13>] nv_nic_irq_optimized+0x73/0x2b0[<ffffffff8026df84>] handle_IRQ_event+0x34/0x70[<ffffffff8026ffe9>] handle_edge_irq+0xc9/0x150[<ffffffff8020e3ab>] do_IRQ+0xcb/0x1c0[<ffffffff8020c093>] ret_from_intr+0x0/0xa<EOI> <4>---[ end trace f6435a98e2a38c0e ]---
The driver developer can find the driver and the device including a stacktraceof the DMA-API call which caused this warning.
Per default only the first error will result in a warning message. All othererrors will only silently counted. This limitation exist to prevent the codefrom flooding your kernel log. To support debugging a device driver this canbe disabled via debugfs. See the debugfs interface documentation below fordetails.
The debugfs directory for the DMA-API debugging code is called dma-api/. Inthis directory the following files can currently be found:
| dma-api/all_errors | This file contains a numeric value. If thisvalue is not equal to zero the debugging codewill print a warning for every error it findsinto the kernel log. Be careful with thisoption, as it can easily flood your logs. |
| dma-api/disabled | This read-only file contains the character ‘Y’if the debugging code is disabled. This canhappen when it runs out of memory or if it wasdisabled at boot time |
| dma-api/dump | This read-only file contains current DMAmappings. |
| dma-api/error_count | This file is read-only and shows the totalnumbers of errors found. |
| dma-api/num_errors | The number in this file shows how manywarnings will be printed to the kernel logbefore it stops. This number is initialized toone at system boot and be set by writing intothis file |
| dma-api/min_free_entries | This read-only file can be read to get theminimum number of free dma_debug_entries theallocator has ever seen. If this value goesdown to zero the code will attempt to increasenr_total_entries to compensate. |
| dma-api/num_free_entries | The current number of free dma_debug_entriesin the allocator. |
| dma-api/nr_total_entries | The total number of dma_debug_entries in theallocator, both free and used. |
| dma-api/driver_filter | You can write a name of a driver into this fileto limit the debug output to requests from thatparticular driver. Write an empty string tothat file to disable the filter and seeall errors again. |
If you have this code compiled into your kernel it will be enabled by default.If you want to boot without the bookkeeping anyway you can provide‘dma_debug=off’ as a boot parameter. This will disable DMA-API debugging.Notice that you can not enable it again at runtime. You have to reboot to doso.
If you want to see debug messages only for a special device driver you canspecify the dma_debug_driver=<drivername> parameter. This will enable thedriver filter at boot time. The debug code will only print errors for thatdriver afterwards. This filter can be disabled or changed later using debugfs.
When the code disables itself at runtime this is most likely because it ranout of dma_debug_entries and was unable to allocate more on-demand. 65536entries are preallocated at boot - if this is too low for you boot with‘dma_debug_entries=<your_desired_number>’ to overwrite the default. Notethat the code allocates entries in batches, so the exact number ofpreallocated entries may be greater than the actual number requested. Thecode will print to the kernel log each time it has dynamically allocatedas many entries as were initially preallocated. This is to indicate that alarger preallocation size may be appropriate, or if it happens continuallythat a driver may be leaking mappings.
voiddebug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
dma-debug interface debug_dma_mapping_error() to debug drivers that failto check DMA mapping errors on addresses returned by dma_map_single() anddma_map_page() interfaces. This interface clears a flag set bydebug_dma_map_page() to indicate that dma_mapping_error() has been called bythe driver. When driver does unmap, debug_dma_unmap() checks the flag and ifthis flag is still set, prints warning message that includes call trace thatleads up to the unmap. This interface can be called from dma_mapping_error()routines to enable DMA mapping error check debugging.