Error Detection And Correction (EDAC) Devices¶

Main Concepts used at the EDAC subsystem¶

There are several things to be aware of that aren’t at all obvious, likesockets, *socket sets,banks,rows,chip-select rows,channels,etc...

These are some of the many terms that are thrown about that don’t alwaysmean what people think they mean (Inconceivable!). In the interest ofcreating a common ground for discussion, terms and their definitionswill be established.

Memory devices

The individual DRAM chips on a memory stick. These devices commonlyoutput 4 and 8 bits each (x4, x8). Grouping several of these in parallelprovides the number of bits that the memory controller expects:typically 72 bits, in order to provide 64 bits + 8 bits of ECC data.

Memory Stick

A printed circuit board that aggregates multiple memory devices inparallel. In general, this is the Field Replaceable Unit (FRU) whichgets replaced, in the case of excessive errors. Most often it is alsocalled DIMM (Dual Inline Memory Module).

Memory Socket

A physical connector on the motherboard that accepts a single memorystick. Also called as “slot” on several datasheets.

Channel

A memory controller channel, responsible to communicate with a group ofDIMMs. Each channel has its own independent control (command) and databus, and can be used independently or grouped with other channels.

Branch

It is typically the highest hierarchy on a Fully-Buffered DIMM memorycontroller. Typically, it contains two channels. Two channels at thesame branch can be used in single mode or in lockstep mode. Whenlockstep is enabled, the cacheline is doubled, but it generally bringssome performance penalty. Also, it is generally not possible to point tojust one memory stick when an error occurs, as the error correction codeis calculated using two DIMMs instead of one. Due to that, it is capableof correcting more errors than on single mode.

Single-channel

The data accessed by the memory controller is contained into one dimmonly. E. g. if the data is 64 bits-wide, the data flows to the CPU usingone 64 bits parallel access. Typically used with SDR, DDR, DDR2 and DDR3memories. FB-DIMM and RAMBUS use a different concept for channel, sothis concept doesn’t apply there.

Double-channel

The data size accessed by the memory controller is interlaced into twodimms, accessed at the same time. E. g. if the DIMM is 64 bits-wide (72bits with ECC), the data flows to the CPU using a 128 bits parallelaccess.

Chip-select row

This is the name of the DRAM signal used to select the DRAM ranks to beaccessed. Common chip-select rows for single channel are 64 bits, fordual channel 128 bits. It may not be visible by the memory controller,as some DIMM types have a memory buffer that can hide direct access toit from the Memory Controller.

Single-Ranked stick

A Single-ranked stick has 1 chip-select row of memory. Motherboardscommonly drive two chip-select pins to a memory stick. A single-rankedstick, will occupy only one of those rows. The other will be unused.

Double-Ranked stick

A double-ranked stick has two chip-select rows which access differentsets of memory devices. The two rows cannot be accessed concurrently.

Double-sided stick

DEPRECATED TERM, seeDouble-Ranked stick.

A double-sided stick has two chip-select rows which access different setsof memory devices. The two rows cannot be accessed concurrently.“Double-sided” is irrespective of the memory devices being mounted onboth sides of the memory stick.

Socket set

All of the memory sticks that are required for a single memory access orall of the memory sticks spanned by a chip-select row. A single socketset has two chip-select rows and if double-sided sticks are used thesewill occupy those chip-select rows.

Bank

This term is avoided because it is unclear when needing to distinguishbetween chip-select rows and socket sets.

High Bandwidth Memory (HBM)

HBM is a new memory type with low power consumption and ultra-widecommunication lanes. It uses vertically stacked memory chips (DRAM dies)interconnected by microscopic wires called “through-silicon vias,” orTSVs.

Several stacks of HBM chips connect to the CPU or GPU through an ultra-fastinterconnect called the “interposer”. Therefore, HBM’s characteristicsare nearly indistinguishable from on-chip integrated RAM.

Memory Controllers¶

Most of the EDAC core is focused on doing Memory Controller error detection.Theedac_mc_alloc(). It uses internally the structmem_ctl_infoto describe the memory controllers, with is an opaque struct for the EDACdrivers. Only the EDAC core is allowed to touch it.

enumdev_type¶: describe the type of memory DRAM chips used at the stick

Constants

DEV_UNKNOWN: Can’t be determined, or MC doesn’t support detect it
DEV_X1: 1 bit for data
DEV_X2: 2 bits for data
DEV_X4: 4 bits for data
DEV_X8: 8 bits for data
DEV_X16: 16 bits for data
DEV_X32: 32 bits for data
DEV_X64: 64 bits for data

Description

Typical values are x4 and x8.

enumhw_event_mc_err_type¶: type of the detected error

Constants

HW_EVENT_ERR_CORRECTED: Corrected Error - Indicates that an ECCcorrected error was detected
HW_EVENT_ERR_UNCORRECTED: Uncorrected Error - Indicates an error thatcan’t be corrected by ECC, but it is notfatal (maybe it is on an unused memory area,or the memory controller could recover fromit for example, by re-trying the operation).
HW_EVENT_ERR_DEFERRED: Deferred Error - Indicates an uncorrectableerror whose handling is not urgent. This couldbe due to hardware data poisoning where thesystem can continue operation until the poisoneddata is consumed. Preemptive measures may alsobe taken, e.g. offlining pages, etc.
HW_EVENT_ERR_FATAL: Fatal Error - Uncorrected error that could notbe recovered.
HW_EVENT_ERR_INFO: Informational - The CPER spec defines a forthtype of error: informational logs.

enummem_type¶: memory types. For a more detailed reference, please seehttp://en.wikipedia.org/wiki/DRAM

Constants

MEM_EMPTY: Empty csrow
MEM_RESERVED: Reserved csrow type
MEM_UNKNOWN: Unknown csrow type
MEM_FPM: FPM - Fast Page Mode, used on systems up to 1995.
MEM_EDO: EDO - Extended data out, used on systems up to 1998.
MEM_BEDO: BEDO - Burst Extended data out, an EDO variant.
MEM_SDR: SDR - Single data rate SDRAMhttp://en.wikipedia.org/wiki/Synchronous_dynamic_random-access_memoryThey use 3 pins for chip select: Pins 0 and 2 arefor rank 0; pins 1 and 3 are for rank 1, if the memoryis dual-rank.
MEM_RDR: Registered SDR SDRAM
MEM_DDR: Double data rate SDRAMhttp://en.wikipedia.org/wiki/DDR_SDRAM
MEM_RDDR: Registered Double data rate SDRAMThis is a variant of the DDR memories.A registered memory has a buffer inside it, hidingpart of the memory details to the memory controller.
MEM_RMBS: Rambus DRAM, used on a few Pentium III/IV controllers.
MEM_DDR2: DDR2 RAM, as described at JEDEC JESD79-2F.Those memories are labeled as “PC2-” instead of “PC” todifferentiate from DDR.
MEM_FB_DDR2: Fully-Buffered DDR2, as described at JEDEC Std No. 205and JESD206.Those memories are accessed per DIMM slot, and not bya chip select signal.
MEM_RDDR2: Registered DDR2 RAMThis is a variant of the DDR2 memories.
MEM_XDR: Rambus XDRIt is an evolution of the original RAMBUS memories,created to compete with DDR2. Weren’t used on anyx86 arch, but cell_edac PPC memory controller uses it.
MEM_DDR3: DDR3 RAM
MEM_RDDR3: Registered DDR3 RAMThis is a variant of the DDR3 memories.
MEM_LRDDR3: Load-Reduced DDR3 memory.
MEM_LPDDR3: Low-Power DDR3 memory.
MEM_DDR4: Unbuffered DDR4 RAM
MEM_RDDR4: Registered DDR4 RAMThis is a variant of the DDR4 memories.
MEM_LRDDR4: Load-Reduced DDR4 memory.
MEM_LPDDR4: Low-Power DDR4 memory.
MEM_DDR5: Unbuffered DDR5 RAM
MEM_RDDR5: Registered DDR5 RAM
MEM_LRDDR5: Load-Reduced DDR5 memory.
MEM_NVDIMM: Non-volatile RAM
MEM_WIO2: Wide I/O 2.
MEM_HBM2: High bandwidth Memory Gen 2.
MEM_HBM3: High bandwidth Memory Gen 3.

enumedac_type¶: Error Detection and Correction capabilities and mode

Constants

EDAC_UNKNOWN: Unknown if ECC is available
EDAC_NONE: Doesn’t support ECC
EDAC_RESERVED: Reserved ECC type
EDAC_PARITY: Detects parity errors
EDAC_EC: Error Checking - no correction
EDAC_SECDED: Single bit error correction, Double detection
EDAC_S2ECD2ED: Chipkill x2 devices - do these exist?
EDAC_S4ECD4ED: Chipkill x4 devices
EDAC_S8ECD8ED: Chipkill x8 devices
EDAC_S16ECD16ED: Chipkill x16 devices

enumscrub_type¶: scrubbing capabilities

Constants

SCRUB_UNKNOWN: Unknown if scrubber is available
SCRUB_NONE: No scrubber
SCRUB_SW_PROG: SW progressive (sequential) scrubbing
SCRUB_SW_SRC: Software scrub only errors
SCRUB_SW_PROG_SRC: Progressive software scrub from an error
SCRUB_SW_TUNABLE: Software scrub frequency is tunable
SCRUB_HW_PROG: HW progressive (sequential) scrubbing
SCRUB_HW_SRC: Hardware scrub only errors
SCRUB_HW_PROG_SRC: Progressive hardware scrub from an error
SCRUB_HW_TUNABLE: Hardware scrub frequency is tunable

enumedac_mc_layer_type¶: memory controller hierarchy layer

Constants

EDAC_MC_LAYER_BRANCH: memory layer is named “branch”
EDAC_MC_LAYER_CHANNEL: memory layer is named “channel”
EDAC_MC_LAYER_SLOT: memory layer is named “slot”
EDAC_MC_LAYER_CHIP_SELECT: memory layer is named “chip select”
EDAC_MC_LAYER_ALL_MEM: memory layout is unknown. All memory is mappedas a single memory area. This is used whenretrieving errors from a firmware driven driver.

Description

Thisenumis used by the drivers to tell edac_mc_sysfs what name shouldbe used when describing a memory stick location.

structedac_mc_layer¶: describes the memory controller hierarchy

Definition:

struct edac_mc_layer {    enum edac_mc_layer_type type;    unsigned size;    bool is_virt_csrow;};

Members

type: layer type
size: number of components per layer. For example,if the channel layer has two channels, size = 2
is_virt_csrow: This layer is part of the “csrow” when old APIcompatibility mode is enabled. Otherwise, it isa channel

structrank_info¶: contains the information for one DIMM rank

Definition:

struct rank_info {    int chan_idx;    struct csrow_info *csrow;    struct dimm_info *dimm;    u32 ce_count;};

Members

chan_idx: channel number where the rank is (typically, 0 or 1)
csrow: A pointer to the chip select row structure (the parentstructure). The location of the rank is given bythe (csrow->csrow_idx, chan_idx) vector.
dimm: A pointer to the DIMM structure, where the DIMM labelinformation is stored.
ce_count: number of correctable errors for this rank

Description

FIXME: Currently, the EDAC core model will assume one DIMM per rank.: This is a bad assumption, but it makes this patch easier. Laterpatches in this series will fix this issue.

structedac_raw_error_desc¶: Raw error report structure

Definition:

struct edac_raw_error_desc {    char location[LOCATION_SIZE];    char label[(EDAC_MC_LABEL_LEN + 1 + sizeof(OTHER_LABEL)) * EDAC_MAX_LABELS];    long grain;    u16 error_count;    enum hw_event_mc_err_type type;    int top_layer;    int mid_layer;    int low_layer;    unsigned long page_frame_number;    unsigned long offset_in_page;    unsigned long syndrome;    const char *msg;    const char *other_detail;};

Members

location: location of the error
label: label of the affected DIMM(s)
grain: minimum granularity for an error report, in bytes
error_count: number of errors of the same type
type: severity of the error (CE/UE/Fatal)
top_layer: top layer of the error (layer[0])
mid_layer: middle layer of the error (layer[1])
low_layer: low layer of the error (layer[2])
page_frame_number: page where the error happened
offset_in_page: page offset
syndrome: syndrome of the error (or 0 if unknown or ifthe syndrome is not applicable)
msg: error message
other_detail: other driver-specific detail about the error

structdimm_info*edac_get_dimm(structmem_ctl_info*mci,intlayer0,intlayer1,intlayer2)¶: Get DIMM info from a memory controller given by [layer0,layer1,layer2] position

Parameters

structmem_ctl_info*mci: MC descriptorstructmem_ctl_info
intlayer0: layer0 position
intlayer1: layer1 position. Unused if n_layers < 2
intlayer2: layer2 position. Unused if n_layers < 3

Description

For 1 layer, this function returns “dimms[layer0]”;

For 2 layers, this function is similar to allocating a two-dimensionalarray and returning “dimms[layer0][layer1]”;

For 3 layers, this function is similar to allocating a tri-dimensionalarray and returning “dimms[layer0][layer1][layer2]”;

structedac_scrub_ops¶: scrub device operations (all elements optional)

Definition:

struct edac_scrub_ops {    int (*read_addr)(struct device *dev, void *drv_data, u64 *base);    int (*read_size)(struct device *dev, void *drv_data, u64 *size);    int (*write_addr)(struct device *dev, void *drv_data, u64 base);    int (*write_size)(struct device *dev, void *drv_data, u64 size);    int (*get_enabled_bg)(struct device *dev, void *drv_data, bool *enable);    int (*set_enabled_bg)(struct device *dev, void *drv_data, bool enable);    int (*get_min_cycle)(struct device *dev, void *drv_data,  u32 *min);    int (*get_max_cycle)(struct device *dev, void *drv_data,  u32 *max);    int (*get_cycle_duration)(struct device *dev, void *drv_data, u32 *cycle);    int (*set_cycle_duration)(struct device *dev, void *drv_data, u32 cycle);};

Members

read_addr: read base address of scrubbing range.
read_size: read offset of scrubbing range.
write_addr: set base address of the scrubbing range.
write_size: set offset of the scrubbing range.
get_enabled_bg: check if currently performing background scrub.
set_enabled_bg: start or stop a bg-scrub.
get_min_cycle: get minimum supported scrub cycle duration in seconds.
get_max_cycle: get maximum supported scrub cycle duration in seconds.
get_cycle_duration: get current scrub cycle duration in seconds.
set_cycle_duration: set current scrub cycle duration in seconds.

structedac_ecs_ops¶: ECS device operations (all elements optional)

Definition:

struct edac_ecs_ops {    int (*get_log_entry_type)(struct device *dev, void *drv_data, int fru_id, u32 *val);    int (*set_log_entry_type)(struct device *dev, void *drv_data, int fru_id, u32 val);    int (*get_mode)(struct device *dev, void *drv_data, int fru_id, u32 *val);    int (*set_mode)(struct device *dev, void *drv_data, int fru_id, u32 val);    int (*reset)(struct device *dev, void *drv_data, int fru_id, u32 val);    int (*get_threshold)(struct device *dev, void *drv_data, int fru_id, u32 *threshold);    int (*set_threshold)(struct device *dev, void *drv_data, int fru_id, u32 threshold);};

Members

get_log_entry_type: read the log entry type value.
set_log_entry_type: set the log entry type value.
get_mode: read the mode value.
set_mode: set the mode value.
reset: reset the ECS counter.
get_threshold: read the threshold count per gigabits of memory cells.
set_threshold: set the threshold count per gigabits of memory cells.

structedac_mem_repair_ops¶: memory repair operations (all elements are optional except do_repair, set_hpa/set_dpa)

Definition:

struct edac_mem_repair_ops {    int (*get_repair_type)(struct device *dev, void *drv_data, const char **type);    int (*get_persist_mode)(struct device *dev, void *drv_data, bool *persist);    int (*set_persist_mode)(struct device *dev, void *drv_data, bool persist);    int (*get_repair_safe_when_in_use)(struct device *dev, void *drv_data, bool *safe);    int (*get_hpa)(struct device *dev, void *drv_data, u64 *hpa);    int (*set_hpa)(struct device *dev, void *drv_data, u64 hpa);    int (*get_min_hpa)(struct device *dev, void *drv_data, u64 *hpa);    int (*get_max_hpa)(struct device *dev, void *drv_data, u64 *hpa);    int (*get_dpa)(struct device *dev, void *drv_data, u64 *dpa);    int (*set_dpa)(struct device *dev, void *drv_data, u64 dpa);    int (*get_min_dpa)(struct device *dev, void *drv_data, u64 *dpa);    int (*get_max_dpa)(struct device *dev, void *drv_data, u64 *dpa);    int (*get_nibble_mask)(struct device *dev, void *drv_data, u32 *val);    int (*set_nibble_mask)(struct device *dev, void *drv_data, u32 val);    int (*get_bank_group)(struct device *dev, void *drv_data, u32 *val);    int (*set_bank_group)(struct device *dev, void *drv_data, u32 val);    int (*get_bank)(struct device *dev, void *drv_data, u32 *val);    int (*set_bank)(struct device *dev, void *drv_data, u32 val);    int (*get_rank)(struct device *dev, void *drv_data, u32 *val);    int (*set_rank)(struct device *dev, void *drv_data, u32 val);    int (*get_row)(struct device *dev, void *drv_data, u32 *val);    int (*set_row)(struct device *dev, void *drv_data, u32 val);    int (*get_column)(struct device *dev, void *drv_data, u32 *val);    int (*set_column)(struct device *dev, void *drv_data, u32 val);    int (*get_channel)(struct device *dev, void *drv_data, u32 *val);    int (*set_channel)(struct device *dev, void *drv_data, u32 val);    int (*get_sub_channel)(struct device *dev, void *drv_data, u32 *val);    int (*set_sub_channel)(struct device *dev, void *drv_data, u32 val);    int (*do_repair)(struct device *dev, void *drv_data, u32 val);};

Members

get_repair_type: get the memory repair type, listed inenumedac_mem_repair_function.
get_persist_mode: get the current persist mode.false - Soft repair type (temporary repair).true - Hard memory repair type (permanent repair).
set_persist_mode: set the persist mode of the memory repair instance.
get_repair_safe_when_in_use: get whether memory media is accessible anddata is retained during repair operation.
get_hpa: get current host physical address (HPA) of memory to repair.
set_hpa: set host physical address (HPA) of memory to repair.
get_min_hpa: get the minimum supported host physical address (HPA).
get_max_hpa: get the maximum supported host physical address (HPA).
get_dpa: get current device physical address (DPA) of memory to repair.
set_dpa: set device physical address (DPA) of memory to repair.In some states of system configuration (e.g. before address decodershave been configured), memory devices (e.g. CXL) may not have an activemapping in the host physical address map. As such, the memoryto repair must be identified by a device specific physical addressingscheme using a device physical address(DPA). The DPA and other controlattributes to use for the repair operations will be presented in relatederror records.
get_min_dpa: get the minimum supported device physical address (DPA).
get_max_dpa: get the maximum supported device physical address (DPA).
get_nibble_mask: get current nibble mask of memory to repair.
set_nibble_mask: set nibble mask of memory to repair.
get_bank_group: get current bank group of memory to repair.
set_bank_group: set bank group of memory to repair.
get_bank: get current bank of memory to repair.
set_bank: set bank of memory to repair.
get_rank: get current rank of memory to repair.
set_rank: set rank of memory to repair.
get_row: get current row of memory to repair.
set_row: set row of memory to repair.
get_column: get current column of memory to repair.
set_column: set column of memory to repair.
get_channel: get current channel of memory to repair.
set_channel: set channel of memory to repair.
get_sub_channel: get current subchannel of memory to repair.
set_sub_channel: set subchannel of memory to repair.
do_repair: Issue memory repair operation for the HPA/DPA andother control attributes set for the memory to repair.

Description

All elements are optional except do_repair and at least one of set_hpa/set_dpa.

structmem_ctl_info*edac_mc_alloc(unsignedintmc_num,unsignedintn_layers,structedac_mc_layer*layers,unsignedintsz_pvt)¶: Allocate and partially fill a structmem_ctl_info.

Parameters

unsignedintmc_num: Memory controller number
unsignedintn_layers: Number of MC hierarchy layers
structedac_mc_layer*layers: Describes each layer as seen by the Memory Controller
unsignedintsz_pvt: size of private storage needed

Description

Everything is kmalloc’ed as one big chunk - more efficient.Only can be used if all structures have the same lifetime - otherwiseyou have to allocate and initialize your own structures.

Useedac_mc_free() to free mc structures allocated by this function.

Note

drivers handle multi-rank memories in different ways: in somedrivers, one multi-rank memory stick is mapped as one entry, while, inothers, a single multi-rank memory stick would be mapped into severalentries. Currently, this function will allocate multiplestructdimm_infoon such scenarios, as grouping the multiple ranks require drivers change.

Return

On success, return a pointer tostructmem_ctl_info pointer;NULL otherwise

constchar*edac_get_owner(void)¶: Return the owner’s mod_name of EDAC MC

Parameters

void: no arguments

Return

Pointer to mod_name string when EDAC MC is owned. NULL otherwise.

voidedac_mc_free(structmem_ctl_info*mci)¶: Frees a previously allocatedmci structure

Parameters

structmem_ctl_info*mci: pointer to astructmem_ctl_info structure

booledac_has_mcs(void)¶: Check if any MCs have been allocated.

Parameters

void: no arguments

Return

True if MC instances have been registered successfully.False otherwise.

structmem_ctl_info*edac_mc_find(intidx)¶: Search for a mem_ctl_info structure whose index isidx.

Parameters

intidx: index to be seek

Description

If found, return a pointer to the structure.Else return NULL.

structmem_ctl_info*find_mci_by_dev(structdevice*dev)¶: Scan list of controllers looking for the one that manages thedev device.

Parameters

structdevice*dev: pointer to astructdevice related with the MCI

Return

on success, returns a pointer to structmem_ctl_info;NULL otherwise.

structmem_ctl_info*edac_mc_del_mc(structdevice*dev)¶: Remove sysfs entries for mci structure associated withdev and remove mci structure from global list.

Parameters

structdevice*dev: Pointer to structdevice representing mci structure to remove.

Return

pointer to removed mci structure, orNULL if device not found.

intedac_mc_find_csrow_by_page(structmem_ctl_info*mci,unsignedlongpage)¶: Ancillary routine to identify what csrow contains a memory page.

Parameters

structmem_ctl_info*mci: pointer to astructmem_ctl_info structure
unsignedlongpage: memory page to find

Return

on success, returns the csrow. -1 if not found.

voidedac_raw_mc_handle_error(structedac_raw_error_desc*e)¶: Reports a memory event to userspace without doing anything to discover the error location.

Parameters

structedac_raw_error_desc*e: error description

Description

This raw function is used internally byedac_mc_handle_error(). It shouldonly be called directly when the hardware error come directly from BIOS,like in the case of APEI GHES driver.

voidedac_mc_handle_error(constenumhw_event_mc_err_typetype,structmem_ctl_info*mci,constu16error_count,constunsignedlongpage_frame_number,constunsignedlongoffset_in_page,constunsignedlongsyndrome,constinttop_layer,constintmid_layer,constintlow_layer,constchar*msg,constchar*other_detail)¶: Reports a memory event to userspace.

Parameters

constenumhw_event_mc_err_typetype: severity of the error (CE/UE/Fatal)
structmem_ctl_info*mci: astructmem_ctl_info pointer
constu16error_count: Number of errors of the same type
constunsignedlongpage_frame_number: mem page where the error occurred
constunsignedlongoffset_in_page: offset of the error inside the page
constunsignedlongsyndrome: ECC syndrome
constinttop_layer: Memory layer[0] position
constintmid_layer: Memory layer[1] position
constintlow_layer: Memory layer[2] position
constchar*msg: Message meaningful to the end users thatexplains the event
constchar*other_detail: Technical details about the event thatmay help hardware manufacturers andEDAC developers to analyse the event

PCI Controllers¶

The EDAC subsystem provides a mechanism to handle PCI controllers by callingtheedac_pci_alloc_ctl_info(). It will use the structedac_pci_ctl_info to describe the PCI controllers.

structedac_pci_ctl_info*edac_pci_alloc_ctl_info(unsignedintsz_pvt,constchar*edac_pci_name)¶: Thealloc() function for the ‘edac_pci’ control info structure.

Parameters

unsignedintsz_pvt: size of the private info at structedac_pci_ctl_info
constchar*edac_pci_name: name of the PCI device

Description

The chip driver will allocate one of these for eachedac_pci it is going to control/register with the EDAC CORE.

Return

a pointer to structedac_pci_ctl_info on success;NULL otherwise.

voidedac_pci_free_ctl_info(structedac_pci_ctl_info*pci)¶: Last action on the pci control structure.

Parameters

structedac_pci_ctl_info*pci: pointer to structedac_pci_ctl_info

Description

Calls the remove sysfs information, which will unregisterthis control struct’s kobj. When that kobj’s ref countgoes to zero, its release function will be call and thenkfree() the memory.

intedac_pci_alloc_index(void)¶: Allocate a unique PCI index number

Parameters

void: no arguments

Return

allocated index number

intedac_pci_add_device(structedac_pci_ctl_info*pci,intedac_idx)¶: Insert the ‘edac_dev’ structure into the edac_pci global list and create sysfs entries associated with edac_pci structure.

Parameters

structedac_pci_ctl_info*pci: pointer to the edac_device structure to be added to the list
intedac_idx: A unique numeric identifier to be assigned to the‘edac_pci’ structure.

Return

0 on Success, or an error code on failure

structedac_pci_ctl_info*edac_pci_del_device(structdevice*dev)¶

Parameters

structdevice*dev: Pointer to ‘structdevice’ representing edac_pci structureto remove

Description

Remove sysfs entries for specified edac_pci structure andthen remove edac_pci structure from global list

Return

Pointer to removed edac_pci structure,orNULL if device not found

structedac_pci_ctl_info*edac_pci_create_generic_ctl(structdevice*dev,constchar*mod_name)¶

Parameters

structdevice*dev: pointer to structdevice;
constchar*mod_name: name of the PCI device

Description

A generic constructor for a PCI parity polling deviceSome systems have more than one domain of PCI busses.For systems with one domain, then this API willprovide for a generic poller.

This routine calls theedac_pci_alloc_ctl_info() forthe generic device, with default values

Return

Pointer to structedac_pci_ctl_info on success,NULL onfailure.

voidedac_pci_release_generic_ctl(structedac_pci_ctl_info*pci)¶

Parameters

structedac_pci_ctl_info*pci: pointer to structedac_pci_ctl_info

Description

The release function of a generic EDAC PCI polling device

intedac_pci_create_sysfs(structedac_pci_ctl_info*pci)¶

Parameters

structedac_pci_ctl_info*pci: pointer to structedac_pci_ctl_info

Description

Create the controls/attributes for the specified EDAC PCI device

voidedac_pci_remove_sysfs(structedac_pci_ctl_info*pci)¶

Parameters

structedac_pci_ctl_info*pci: pointer to structedac_pci_ctl_info

Description

remove the controls and attributes for this EDAC PCI device

EDAC Blocks¶

The EDAC subsystem also provides a generic mechanism to report errors onother parts of the hardware viaedac_device_alloc_ctl_info() function.

The structuresedac_dev_sysfs_block_attribute,edac_device_block,edac_device_instance andedac_device_ctl_info provide a generic or abstract ‘edac_device’representation at sysfs.

This set of structures and the code that implements the APIs for the same, provide for registering EDAC type devices which are NOT standard memory orPCI, like:

CPU caches (L1 and L2)
DMA engines
Core CPU switches
Fabric switch units
PCIe interface controllers
other EDAC/ECC type devices that can be monitored forerrors, etc.

It allows for a 2 level set of hierarchy.

For example, a cache could be composed of L1, L2 and L3 levels of cache.Each CPU core would have its own L1 cache, while sharing L2 and maybe L3caches. On such case, those can be represented via the following sysfsnodes:

/sys/devices/system/edac/..pci/            <existing pci directory (if available)>mc/             <existing memory device directory>cpu/cpu0/..     <L1 and L2 block directory>        /L1-cache/ce_count                 /ue_count        /L2-cache/ce_count                 /ue_countcpu/cpu1/..     <L1 and L2 block directory>        /L1-cache/ce_count                 /ue_count        /L2-cache/ce_count                 /ue_count...the L1 and L2 directories would be "edac_device_block's"

intedac_device_add_device(structedac_device_ctl_info*edac_dev)¶: Insert the ‘edac_dev’ structure into the edac_device global list and create sysfs entries associated with edac_device structure.

Parameters

structedac_device_ctl_info*edac_dev: pointer to edac_device structure to be added to the list‘edac_device’ structure.

Return

0 on Success, or an error code on failure

structedac_device_ctl_info*edac_device_del_device(structdevice*dev)¶: Remove sysfs entries for specified edac_device structure and then remove edac_device structure from global list

Parameters

structdevice*dev: Pointer to structdevice representing the edac devicestructure to remove.

Return

Pointer to removed edac_device structure,orNULL if device not found.

voidedac_device_handle_ce_count(structedac_device_ctl_info*edac_dev,unsignedintcount,intinst_nr,intblock_nr,constchar*msg)¶: Log correctable errors.

Parameters

structedac_device_ctl_info*edac_dev: pointer to structedac_device_ctl_info
unsignedintcount: Number of errors to log.
intinst_nr: number of the instance where the CE error happened
intblock_nr: number of the block where the CE error happened
constchar*msg: message to be printed

voidedac_device_handle_ue_count(structedac_device_ctl_info*edac_dev,unsignedintcount,intinst_nr,intblock_nr,constchar*msg)¶: Log uncorrectable errors.

Parameters

structedac_device_ctl_info*edac_dev: pointer to structedac_device_ctl_info
unsignedintcount: Number of errors to log.
intinst_nr: number of the instance where the CE error happened
intblock_nr: number of the block where the CE error happened
constchar*msg: message to be printed

voidedac_device_handle_ce(structedac_device_ctl_info*edac_dev,intinst_nr,intblock_nr,constchar*msg)¶: Log a single correctable error

Parameters

structedac_device_ctl_info*edac_dev: pointer to structedac_device_ctl_info
intinst_nr: number of the instance where the CE error happened
intblock_nr: number of the block where the CE error happened
constchar*msg: message to be printed

voidedac_device_handle_ue(structedac_device_ctl_info*edac_dev,intinst_nr,intblock_nr,constchar*msg)¶: Log a single uncorrectable error

Parameters

structedac_device_ctl_info*edac_dev: pointer to structedac_device_ctl_info
intinst_nr: number of the instance where the UE error happened
intblock_nr: number of the block where the UE error happened
constchar*msg: message to be printed

intedac_device_alloc_index(void)¶: Allocate a unique device index number

Parameters

void: no arguments

Return

allocated index number

Heterogeneous system support¶

An AMD heterogeneous system is built by connecting the data fabrics ofboth CPUs and GPUs via custom xGMI links. Thus, the data fabric on theGPU nodes can be accessed the same way as the data fabric on CPU nodes.

The MI200 accelerators are data center GPUs. They have 2 data fabrics,and each GPU data fabric contains four Unified Memory Controllers (UMC).Each UMC contains eight channels. Each UMC channel controls one 128-bitHBM2e (2GB) channel (equivalent to 8 X 2GB ranks). This creates a totalof 4096-bits of DRAM data bus.

While the UMC is interfacing a 16GB (8high X 2GB DRAM) HBM stack, each UMCchannel is interfacing 2GB of DRAM (represented as rank).

Memory controllers on AMD GPU nodes can be represented in EDAC thusly:

GPU DF / GPU Node -> EDAC MCGPU UMC -> EDAC CSROWGPU UMC channel -> EDAC CHANNEL

For example: a heterogeneous system with 1 AMD CPU is connected to4 MI200 (Aldebaran) GPUs using xGMI.

Some more heterogeneous hardware details:

The CPU UMC (Unified Memory Controller) is mostly the same as the GPU UMC.They have chip selects (csrows) and channels. However, the layouts are differentfor performance, physical layout, or other reasons.
CPU UMCs use 1 channel, In this case UMC = EDAC channel. This follows themarketing speak. CPU has X memory channels, etc.
CPU UMCs use up to 4 chip selects, So UMC chip select = EDAC CSROW.
GPU UMCs use 1 chip select, So UMC = EDAC CSROW.
GPU UMCs use 8 channels, So UMC channel = EDAC channel.

The EDAC subsystem provides a mechanism to handle AMD heterogeneoussystems by calling system specific ops for both CPUs and GPUs.

AMD GPU nodes are enumerated in sequential order based on the PCIhierarchy, and the first GPU node is assumed to have a Node ID valuefollowing those of the CPU nodes after latter are fully populated:

$ ls /sys/devices/system/edac/mc/        mc0   - CPU MC node 0        mc1  |        mc2  |- GPU card[0] => node 0(mc1), node 1(mc2)        mc3  |        mc4  |- GPU card[1] => node 0(mc3), node 1(mc4)        mc5  |        mc6  |- GPU card[2] => node 0(mc5), node 1(mc6)        mc7  |        mc8  |- GPU card[3] => node 0(mc7), node 1(mc8)

For example, a heterogeneous system with one AMD CPU is connected tofour MI200 (Aldebaran) GPUs using xGMI. This topology can be representedvia the following sysfs entries:

/sys/devices/system/edac/mc/..CPU                     # CPU node├── mc 0GPU Nodes are enumerated sequentially after CPU nodes have been populatedGPU card 1              # Each MI200 GPU has 2 nodes/mcs├── mc 1                # GPU node 0 == mc1, Each MC node has 4 UMCs/CSROWs│   ├── csrow 0         # UMC 0│   │   ├── channel 0   # Each UMC has 8 channels│   │   ├── channel 1   # size of each channel is 2 GB, so each UMC has 16 GB│   │   ├── channel 2│   │   ├── channel 3│   │   ├── channel 4│   │   ├── channel 5│   │   ├── channel 6│   │   ├── channel 7│   ├── csrow 1         # UMC 1│   │   ├── channel 0│   │   ├── ..│   │   ├── channel 7│   ├── ..              ..│   ├── csrow 3         # UMC 3│   │   ├── channel 0│   │   ├── ..│   │   ├── channel 7│   ├── rank 0│   ├── ..              ..│   ├── rank 31         # total 32 ranks/dimms from 4 UMCs├├── mc 2                # GPU node 1 == mc2│   ├── ..              # each GPU has total 64 GBGPU card 2├── mc 3│   ├── ..├── mc 4│   ├── ..GPU card 3├── mc 5│   ├── ..├── mc 6│   ├── ..GPU card 4├── mc 7│   ├── ..├── mc 8│   ├── ..

Movatterモバイル変換

Error Detection And Correction (EDAC) Devices¶

Main Concepts used at the EDAC subsystem¶

Memory Controllers¶

PCI Controllers¶

EDAC Blocks¶

Heterogeneous system support¶