US8886881B2 - Implementing storage adapter performance optimization with parity update footprint mirroring - Google Patents

Abstract

A method and controller for implementing storage adapter performance optimization with parity update footprint mirroring between dual adapters minimizing firmware operations, and a design structure on which the subject controller circuit resides are provided. Each of a first controller and a second controller includes a plurality of hardware engines, a control store configured to store parity update footprint (PUFP) data; a data store; and a nonvolatile random access memory (NVRAM). One controller operates in a first initiator mode for transferring PUFP data to the other controller operating in a target mode. Respective initiator hardware engines transfers PUFP data from the initiator control store, selectively updating PUFP data, and writing PUFP data to the initiator data store and to the initiator NVRAM, and simultaneously transmitting PUFP data to the other controller. Respective target hardware engines write PUFP data to the target data store and the target NVRAM, eliminating firmware operations.

Description

FIELD OF THE INVENTION

The present invention relates generally to the data processing field, and more particularly, relates to a method and controller for implementing storage adapter performance optimization with parity update footprint mirroring between dual adapters minimizing firmware operations, and a design structure on which the subject controller circuit resides.

DESCRIPTION OF THE RELATED ART

Storage adapters are used to connect a host computer system to peripheral storage I/O devices such as hard disk drives, solid state drives, tape drives, compact disk drives, and the like. Currently various high speed system interconnects are to connect the host computer system to the storage adapter and to connect the storage adapter to the storage I/O devices, such as, Peripheral Component Interconnect Express (PCIe), Serial Attach SCSI (SAS), Fibre Channel, and InfiniBand.

For many years now, hard disk drives (HDDs) or spinning drives have been the dominant storage I/O device used for the persistent storage of computer data which requires online access. Recently, solid state drives (SSDs) have become more popular due to their superior performance. Specifically, SSDs are typically capable of performing more I/Os per seconds (IOPS) than HDDs, even if their maximum data rates are not always higher than HDDs.

From a performance point of view, an ideal storage adapter would never be a performance bottleneck to the system. However, in reality storage adapters are often a performance bottleneck to the computer system. One effect of the increasing popularity of SSDs is that the storage adapter is more often the performance bottleneck in the computer system.

A need exists for an effective method and controller for implementing storage adapter performance optimization. A need exists for such method and controller for use with either HDDs or SSDs and that significantly reduces the time required for an I/O operation, while efficiently and effectively maintaining needed functions of the storage adapter for various arrangements of the storage adapter and the storage I/O devices, such as utilizing Write Caching, and Dual Controllers configurations, and redundant array of inexpensive drives (RAID) read and write operations.

As used in the following description and claims, the terms controller and controller circuit should be broadly understood to include an input/output (IO) adapter (IOA) and includes an IO RAID adapter connecting various arrangements of a host computer system and peripheral storage I/O devices including hard disk drives, solid state drives, tape drives, compact disk drives, and the like.

SUMMARY OF THE INVENTION

Principal aspects of the present invention are to provide a method and controller for implementing storage adapter performance optimization with parity update footprint mirroring between dual adapters minimizing firmware operations, and a design structure on which the subject controller circuit resides. Other important aspects of the present invention are to provide such method, controller, and design structure substantially without negative effects and that overcome many of the disadvantages of prior art arrangements.

In brief, a method and controller for implementing storage adapter performance optimization with parity update footprint mirroring between dual adapters minimizing firmware operations, and a design structure on which the subject controller circuit resides are provided. Each of a first controller and a second controller includes a plurality of hardware engines, a control store configured to store parity update footprint (PUFP) data; a data store; and a nonvolatile random access memory (NVRAM). One of the first controller or the second controller operates in a first initiator mode for transferring PUFP data to the other of the first controller or the second controller operating in a target mode. Respective initiator hardware engines transfers PUFP data from the initiator control store, selectively updating PUFP data, and writing PUFP data to the initiator data store and to the initiator NVRAM, and simultaneously transmitting a PUFP data frame to the other of the first controller or the second controller operating in the target mode. Respective target hardware engines write PUFP data to the target data store and the target NVRAM, eliminating firmware operations.

In accordance with features of the invention, the PUFP data mirroring between the first controller and second controller involves minimal firmware operation for the controller operating in the first initiator mode, while eliminating firmware operations of the controller operating in the target mode.

In accordance with features of the invention, the target hardware engines write PUFP data to the target data store addressed by PUFP DS Base address. The target hardware engines write PUFP data to the target NVRAM addressed by PUFP NVRAM Base address.

In accordance with features of the invention, the controller operating in the target mode transmits a response frame after processing the PUFP data frame at the end of the mirrored PUFP data operation.

In accordance with features of the invention, the controller operating in the first initiator mode does not transmit a command frame for the mirrored PUFP data operation. The controller operating in the target mode does not transmit any transfer ready frames.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention together with the above and other objects and advantages may best be understood from the following detailed description of the preferred embodiments of the invention illustrated in the drawings, wherein:

FIG. 1 is a schematic and block diagram illustrating an exemplary system for implementing storage adapter performance optimization with chained hardware operations minimizing hardware and firmware interactions in accordance with the preferred embodiment;

FIG. 2A illustrates example chained hardware operations minimizing hardware and firmware interactions in accordance with the preferred embodiment;

FIG. 2B illustrates conventional prior art storage adapter hardware and firmware interactions;

FIG. 3A illustrates an example control store (CS) structure including a plurality of sequential control blocks in accordance with the preferred embodiment;

FIG. 3B illustrates an enhanced hardware (HW) and firmware (FW) interface including a plurality of example hardware (HW) Work Queues and a HW Event Queue stored in the control store (CS) in accordance with the preferred embodiment;

FIG. 4A illustrates an example common header of a control block in accordance with the preferred embodiment;

FIG. 4B illustrates a plurality of example control blocks in accordance with the preferred embodiment;

FIGS. 5A and 5B are hardware logic operations flow and flow chart illustrating exemplary operations performed by a predefined chain of a plurality of the control blocks selectively arranged to implement an example RAID-5 normal parity update in accordance with the preferred embodiment;

FIGS. 6A and 6B are hardware logic operations flow and flow chart illustrating exemplary operations performed by a predefined chain of a plurality of the control blocks selectively arranged to implement an example RAID-6 normal parity update in accordance with the preferred embodiment;

FIGS. 7A and 7B are hardware logic operations flow and flow chart illustrating exemplary operations performed by a pair of predefined chains of a plurality of the control blocks selectively arranged to implement an example RAID-5/6 stripe write with cache in accordance with the preferred embodiment;

FIGS. 8A and 8B are hardware logic operations flow and flow chart illustrating exemplary operations for mirroring parity update footprints between dual adapters in accordance with the preferred embodiment; and

FIG. 9 is a flow diagram of a design process used in semiconductor design, manufacturing, and/or test.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following detailed description of embodiments of the invention, reference is made to the accompanying drawings, which illustrate example embodiments by which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

In accordance with features of the invention, a method and controller implement enhanced storage adapter performance and performance optimization with chained hardware operations for cache data and cache directory mirroring write and mirroring delete of minimizing firmware use, and a design structure on which the subject controller circuit resides is provided.

Having reference now to the drawings, inFIG. 1, there is shown an input/output adapter (IOA) or controller in accordance with the preferred embodiment generally designated by thereference character100.Controller100 includes asemiconductor chip102 coupled to at least oneprocessor complex104 including one or more processors or central processor units (CPUs)106.Controller100 includes a control store (CS)108, such as a dynamic random access memory (DRAM) proximate to theCPU106 providing control block, work queue and event queue storage.Controller100 includes a non-volatile (NV)backup memory110 and a data store (DS)112 providing data and scratch buffers for control block set up and processing, for example, performed by hardware.Controller100 includes a non-volatile random access memory (NVRAM)114, and aflash memory116.

In accordance with features of the invention, controller100 implements methods that uniquely chains together hardware operations in order to minimize hardware/firmware interactions in order to maximize performance. The hardware (HW) chaining is completely heterogeneous; asynchronous, not requiring synchronization or defined timed slots for operations; fully free form with any HW engine chained to any HW engine, and operational policy in FW dispatching at HW speeds.

Controller semiconductor chip102 includes a plurality ofhardware engines120, such as, a hardware direct memory access (HDMA)engine120, aSIS engine120, an allocate andde-allocate engine120, an XOR or sum of products (SOP)engine120, a Serial Attach SCSI (SAS)engine120, a set/update/clear/mirror footprint (S/U/C/M FP)engine120, and a compression/decompression (COMP/DECOMP)engine120.

In accordance with features of the invention, substantial conventional firmware function is moved to HW operations performed by thehardware engines120. Thehardware engines120 are completely heterogeneous, and are fully extensible with chaining any engine to any other engine enabled.

As shown,controller semiconductor chip102 includes a respective Peripheral Component Interconnect Express (PCIe)interface128 with a PCIe high speed system interconnect between thecontroller semiconductor chip102 and theprocessor complex104, and a Serial Attach SCSI (SAS)controller130 with a SAS high speed system interconnect between thecontroller semiconductor chip102 and each of a plurality ofstorage devices132, such as hard disk drives (HDDs) or spinning drives132, and solid state drives (SSDs)132. Ahost system134 is connected to thecontroller100 with a PCIe high speed system interconnect.

DS112, for example, 8 GB of DRAM, stores volatile or non-volatile pages of Data, such as 4 KB page of Data or 8*528-bytes usable data or 64 CAS access (66-bytes), 32-byte cache line (CL) with one CL for each non-volatile page of the write cache in a contiguous area of DS and 32-byte parity update footprint (PUFP) in a contiguous area of DS after the CL area.

The control store (CS)108 stores other structures and control blocks, such as illustrated and described with respect toFIGS. 3A and 3B, andFIGS. 4A and 4B. The control store (CS)108 includes a control block (CB) buffer area, such as 8 MB size and 8 MB alignment, a HW Event Queue, such as 4 MB size and 4 MB alignment, providing 1M entries of 4 B each, SIS SEND Queue, such as 64 KB size and 64 KB alignment, providing 4K entries of 16 B each, Index Free List Volatile and Index Free List Non-Volatile, each such as 4 MB size and 4 MB alignment, providing 1M entries of 4 B each, HW Work Queues (WQ), such as 512 KB size and 512 KB alignment, providing 32 WQ of 16 KB each. Other structures in theCS108 include Page Table Index Lists, such as 4 B,1-N entries of 4 B each, which can be anywhere in the 256 MB space and are often within the 8 MB CS buffer area, CS target Buffers of 128 B alignment, where each buffer is 1 KB, and can be anywhere in the 256 MB space, and HW CB of 64 B alignment, which are within the 8 MB CS buffer area, such as illustrated inFIG. 3A.

Referring toFIG. 2A, there are shown example chained hardware operations minimizing hardware and firmware interactions in accordance with the preferred embodiment generally designated by thereference character200. The chainedhardware operations200 include achain202 of a plurality of sequential operations by hardware (HW)204 with an initial interaction with code or firmware (FW)206 at the initial setup and another interaction withFW208 at the completion of the series orchain202 of operations byHW204.

In accordance with features of the invention, the types of chained operations include Buffer Allocate, Buffer Deallocate, SAS Read-XOR, SAS Write, and Setting Parity Update Footprint (PUFP). Clearing PUFP, Mirrored write of a PUFP to a remote adapter, Mirrored write of cache data to remote adapter, and the like. For example, the following is an example of chained operations for a RAID-5 write: a) Buffer allocate, b) Read-XOR of data, c) Setting of PUFP, d) Write of data, e) Update parity footprint, f) Read-XOR of parity, g) Write of parity, h) Clearing of PUFP, and i) Buffer deallocate.

FIG. 2B illustrates conventional prior art storage adapter hardware and firmware interactions that includes a code or firmware (FW) and hardware interaction with each of multiple IOA operations. As shown inFIG. 2A, the chainedhardware operations200 of the invention, significantly reduces the firmware path length required for an I/O operation. The chainedhardware operations200 of the invention are arranged to minimize hardware/firmware interactions in order to maximize performance.

Referring toFIG. 3A, there is shown an example control store (CS) structure generally designated by thereference character300 in accordance with the preferred embodiment.CS structure300 includes predefined fields including an offset302,size304, anddefinition306.CS structure300 includes a plurality of sequential control blocks (HW CB) #1-17,308, for example, which are selectively arranged in a predefined chain to minimize hardware and firmware interaction, such as to minimize thehardware engines120 writing event queue entries to theprocessor complex104.

In accordance with features of the invention, each predefined chain includes sequential control blocks308 stored within contiguous memory inCS108, as illustrated inFIG. 3A. Each predefined chain defines controls applied torespective hardware engines120. Each control block308 can be linked to any other control block308 defining a predefined chain of operations. For example, each buffer inCS structure300 is 2 KB in size. FW gives these buffers to HW by writing CS Indices to the Global Hardware (HW) Work Queue. HW returns to FW by writing to the HW Event Queue, as illustrated and described with respect toFIG. 3B.

Referring toFIG. 3B, there is shown an enhanced hardware (HW) and firmware (FW) interface generally designated by thereference character350 in accordance with the preferred embodiment. The HW/FW interface350 includes a HW block352 including the plurality ofHW engines120 in thecontroller chip102 and afirmware block354 provided with theCPU106 in theprocessor complex104. The HW/FW interface350 includes a global hardware (HW)Work Queue356, such as a small embedded array in thecontroller chip102. The globalHW Work Queue356 is coupled to each of a plurality of hardware (HW)Work Queues358.

Each of the plurality of hardware (HW)Work Queues358 is applied to respective hardware engines1-N,120 within thechip102. AHW Event Queue360 is coupled to firmware (FW)354 providing completion results to theprocessor complex104. AWork Queue Manager362 in thecontroller chip102 is coupled to each of the plurality of hardware (HW)Work Queues358 and hardware engines1-N,120, and to theHW Event Queue360. The globalHW work queue356 includes a queue input coupled toFW354 in theprocessor complex104 and a queue input coupled to theWork Queue Manager362 in thecontroller chip102. TheWork Queue Manager362 and the globalHW work queue356 provide an input to theHW Event Queue360. TheHW Work Queues358, and theHW Event Queue360 are stored in the control store (CS)108.

Thehardware engines120 are arranged to DMA data from thehost system134 to thecontroller100. TheHDMA engine120 DMAs the data fromhost system134 to theCS108 orDS112, then notifies FW via theHW Event Queue360. Thehardware engines120 are arranged to run some functions in parallel, such as 8 or 12SAS engines120, 4host DMA engines120, and the like. Thehardware engines120 are arranged to run multiple operations on different steps of the same function, such as anHDMA engine120 fetches data from thehost system134 at the same time that anotherHDMA engine120 is DMAing other data to thehost system134.

In accordance with features of the invention, each control block308 includes a common header including a control block ID, a chain position, and a next control block ID. The control block chain position identifies a first in chain, a last in chain, middle in linked chain, or stand alone. The common header includes a predefined hardware event queue entry selectively written when the control block completes. The predefined hardware event queue entry is written when a stand alone control block completes and the last in chain control block completes. The predefined hardware event queue entry is written when control block fails with an error.

Referring also toFIG. 4A, there is shown an example common header generally designated by thereference character400 of thecontrol block308 in accordance with the preferred embodiment. Eachcontrol block header400 includes abyte0,402, for example, reserved or drive tag.

Eachcontrol block header400 includes abyte1,404 including for example, a selective write HW Event Queue entry. The predefined hardwareevent queue entry404 is selectively written when the control block completes. The predefined hardwareevent queue entry404 is written when a stand alone control block completes or a last in chain control block completes. The predefined hardwareevent queue entry404 is written when control block fails with an error.

Eachcontrol block header400 includes abyte2,406 including an update HW Event Queue entry and a next control block engine identification (ID)406. TheHW Event Queue360 shown inFIG. 3B is a circular first-in first-out (FIFO) in theCS108. TheHW Event Queue360 is aligned on a 4M-byte address boundary, and is 4M-bytes in size. This size allows the queue to be a history of the last 1M events. HW writes 4-byte entries406 to the HW Event Queue for each event. FW periodically reads and removes the entries from the HW Event Queue.

Eachcontrol block header400 includes abyte3,408, including a control block engine ID and achain position408, and includes a header address (ADR)410. The controlblock chain position408 identifies a first in chain, a last in chain, middle in linked chain, or stand alone control block chain position.

Chained or stand alone CB execution begins when an entry is removed from the GlobalHW Work Queue356 and dispatched by theWork Queue Manager362 to one of theHW Work Queues358 coupled to one of theHardware Engines120.Hardware Engines120 inFIG. 3B can execute a chain of control blocks, HW CB #1-17,308, as shown inFIG. 3A and further illustrated inFIGS. 4A, and4B. TheHW CB308 links to the next operation in the predefined chain when thecurrent engine120 completes execution of its operation in the predefined chain. The mechanism for thenext HW CB308 in a respective predefined chain to eventually start execution is initiated by therespective hardware engine120. Thehardware engine120 when completing execution of itsHW CB308 in the chain, adds 64 to its current CB address inCS108, which then forms a new CB address inCS108 that maps directly to the next 64 byte Offset302 in the chain shown inFIG. 3A. This new CB address, together with the CB ID Next Linkedfield406, is given to theWork Queue Manager362 byhardware engine120. TheWork Queue Manager362 then adds a new entry toGlobal HW WQ356. The next CB in the predefined chain will then execute when this entry is removed from theGlobal HW WQ356 and dispatched to one of theHW Work Queues358.

Referring toFIG. 4B, there are shown a plurality of example control blocks in accordance with the preferred embodiment. The control blocks308 include:

• Set/Update/Clear/Mirror FP (Footprint)—F,
• Set/Clear/Mirror CL—M,
• Send SAS Op—S,
• Free Allocated Pages—D,
• Run SOP Engine—X,
• Allocate Pages—A,
• Send HDMA Op—H, and
• Comp/Decompression—C.

With the Set/Update/Clear/Mirror FP (Footprint)—F control block308, CS actions performed by HW or S/U/C/M FP engine120 include for example,Read 32 Bytes fromCS108, for Set, for each 4K,Read 32 Bytes, Write 32 Bytes toDS112 and Write 32 Bytes toNVRAM114, and optionally mirror to remote controller; for Update,Read 32 Bytes fromCS108 orDS112, Write 32 Bytes toDS112 and Write 32 Bytes toNVRAM114, and optionally mirror to remote controller; and for Clear, Write 32 Bytes toDS112 and Write 32 Bytes toNVRAM114, and optionally mirror to remote controller.

With the Set/Clear/Mirror CL—M control block308, CS actions performed by HW or S/C/M CL engine120 include for example,Read 32 Bytes fromCS108, for Set, for each 4K,Read 32 Bytes, Write 32 Bytes toDS112 and For each 4K,Read 4 byte index, and may read 4K fromDS112 and optionally mirror to remote controller; and for Clear, For each 4K,Read 4 byte index, and Write 32 Bytes toDS112 and optionally mirror to remote controller.

With the Send SAS Op—S control block308 and the Send HDMA Op—H, CS actions performed by HW or therespective SAS engine120 and theHDMA engine120 include for example, For each 4K,SAS engine120 and theHDMA engine120Read 4 byte index, andHDMA engine120 will Read or Write 4K toDS112, andSAS engine120 may read and write 4K toDS112. TheHDMA engine120 moves data betweenDS112 and thehost system134, and theSAS engine120 moves data betweenDS112, and thestorage devices132.

With the Free Allocated Pages—D and the Allocate pages—A control blocks308, CS actions performed by HW or the Alloc/Dealloc engine120 include for example, for each 4K,Read 4 Bytes, andWrite 4 Bytes.

With the Run SOP Engine—X control block308, CS actions performed by HW or theXOR engine120 include for example, For each 4K of Source (for each source),Read 4 Bytes, and Read 4K ofDS112; and For each 4K of Destination (for each destination),Read 4 Bytes, and Write 4K ofDS112. The sum-of-products (SOP)engine120 takes an input of 0-N source page lists and 0-M destination page lists as well as an N×M array of multipliers. For example, N=18 and M=2. For each 4K, the first source page is read from DRAM and the first set of M multipliers are applied to each byte. The resulting data is put into M on chip accumulation buffers. Each subsequent source page is multiplied by its associated M multipliers and the product XORed with the corresponding accumulation buffers. When every source has been processed, the accumulation buffers are written out to the corresponding M destination buffers. Then, the next 4K is started. This allows computing an N input XOR to compute RAID-5 parity or N input multiply XOR of M equations simultaneously for Reed-Solomon based RAID-6 P & Q redundancy data.

With the Comp/Decompression—C control block308, CS actions performed by HW or the Comp/Decomp engine120 include for example, For each logical 4K (compressed data may be <4K),Read 4 Bytes, and Read 4K of DS112 (or less if doing decompression),Read 4 Bytes, and Write 4K of DS112 (or less if doing compression), and optionally other operations may be performed.

A respective example chain of control blocks308 is illustrated and described with respect to each ofFIGS. 5A,6A, and7A in accordance with the preferred embodiment.

Referring toFIGS. 5A and 5B, there are shown hardware logic operations flow generally designated by thereference character500 and a flow chart inFIG. 5B illustrating exemplary operations performed by a predefined chain generally designated by thereference character520 of a plurality of the control blocks selectively arranged to implement an example RAID-5 normal parity update in accordance with the preferred embodiment. InFIG. 5A, thechain520 ofcontrol block308 include control blocks A1, S2, F3, S4, F5, S6, S7, F8, D9, M10, and D11, as defined inFIG. 4B together with the respective steps1-11 shown inFIGS. 5A and 5B.

FIG. 5A includes a local CS502 of a first orlocal controller100A coupled by a HW engine505 to aremote DS504 and to aremote NVRAM506 of a second orremote controller100B. The local CS502 is coupled by the HW engine505 to alocal NVRAM508, and alocal DS510 of thefirst controller100A. A plurality of buffers of afirst controller100A including buffer A,512, buffer B,514, and buffer C,516, are coupled to adisk P518 and adisk X520.

Instep1, A and B lists for Buffer A,512, and Buffer B,514 are allocated or populated at control block A1 ofchain520, in CS local502 inFIG. 5A, and as indicated at ablock540 inFIG. 5B. Next inStep2, Data is read fromDisk X520, and XORed with Buffer C,516 and the result is placed in Buffer B,514 at control block S2 ofchain520, at 2 XOR inFIG. 5A, and as indicated at ablock542 inFIG. 5B. Instep3, set footprint is performed at control block F3 ofchain520, read by HW engine505,line3 from HW engine505 toDS510 andNVRAM508 on thelocal controller100A and set footprint on theremote controller100B from HW engine505 toDS504 andNVRAM506 inFIG. 5A, and as indicated at ablock544 inFIG. 5B.

Instep4, Write data from Buffer C,516 toDisk X520 is performed at control block S4 ofchain520,line4 from Buffer C,516 toDisk X520 inFIG. 5A, and as indicated at ablock546 inFIG. 5B. Next inStep5, update footprint is performed at control block F5 ofchain520, read by HW engine505,line5 from HW engine505 toDS510 andNVRAM508 on thelocal controller100A and update footprint on theremote controller100B from HW engine505 toDS504 andNVRAM506 inFIG. 5A, and as indicated at ablock547 inFIG. 5B. Next inStep6, Data is read fromDisk P518, and XORed with Buffer B,514 and the result is placed in Buffer A,512 at control block S6 ofchain520, at 6 XOR inFIG. 5A, and as indicated at ablock548 inFIG. 5B. Next inStep7, Write data from Buffer A,512 toDisk P518 is performed at control block S7 ofchain520, atline7 from Buffer A,512 toDisk P518 inFIG. 5A, and as indicated at ablock550 inFIG. 5B.

Instep8, Clear footprint is performed by HW engine505 writing zeros at control block F8 ofchain520, atline8 from HW engine505 toNVRAM508 and theDS510 on thelocal controller100A and clear footprint on theremote controller100B atline8 from HW engine505 toDS504 andNVRAM506 inFIG. 5A, and as indicated at ablock552 inFIG. 5B. Instep9, A and B lists for Buffer A,512, and Buffer B,514 are deallocated or depopulated at control block D9 ofchain520, at CS local502 inFIG. 5A, and as indicated at ablock554 inFIG. 5B. Instep10, Send mirrored delete for cache by HW engine505 writing zeros to clear CL onlocal DS510 and to clear CL onremote DS504 at control block M10 ofchain520, indicated atline10 from HW engine505 tolocal DS510 and toremote DS504 inFIG. 5A, and as indicated at ablock556 inFIG. 5B. Instep11, Page lists for Buffer C,516 are de-allocated or depopulated at control block D11 ofchain520, at CS local502 inFIG. 5A, and as indicated at ablock558 inFIG. 5B.

Referring toFIGS. 6A and 6B, there are shown hardware logic operations flow generally designated by thereference character600 and a flow chart inFIG. 6B illustrating exemplary operations performed by a predefined chain generally designated by thereference character630 of a plurality of the control blocks selectively arranged to implement an example RAID-6 normal parity update in accordance with the preferred embodiment. InFIG. 6A, thechain630 ofcontrol block308 include control blocks A1, S2, F3, S4, S5, S6, S7, F8, S9, S10, F11, D12, M13, and D14, as defined inFIG. 4B together with the respective steps1-14 shown inFIGS. 6A and 6B.

FIG. 6A includes alocal CS602 of a first orlocal controller100A coupled by a hardware engine605 to aremote DS604 and to aremote NVRAM606 of a second orremote controller100B. Thelocal CS602 is coupled by the hardware engine605 to alocal NVRAM608, and alocal DS610 of thefirst controller100A. A plurality of buffers of afirst controller100A including buffer A,612, buffer B,614, and buffer C,616, are coupled to adisk P618, adisk X620 and adisk Q622.

Instep1, A and B lists for Buffer A,612, and Buffer B,614 are allocated or populated at control block A1 ofchain630, in CS local602 inFIG. 6A, and as indicated at ablock640 inFIG. 6B. Next inStep2, Data is read fromDisk X620, and XORed with Buffer C,616 and the result is placed in Buffer B,614 at control block S2 ofchain630, at 2 XOR inFIG. 6A, and as indicated at ablock642 inFIG. 6B. Instep3, set footprint is performed at control block F3 ofchain630, read by HW engine605,line3 from HW engine605 toDS610 andNVRAM608 on thelocal controller100A and set footprint on theremote controller100B from HW engine605 toDS604 andNVRAM606 inFIG. 6A, and as indicated at ablock644 inFIG. 6B.

Instep4, Write data from Buffer C,616 toDisk X630 is performed control block S4 ofchain630,line4 from Buffer C,616 toDisk X630 inFIG. 6A, and as indicated at ablock646 inFIG. 6B. Next inStep5, Data is read fromDisk P618, and XORed with multiplied data from Buffer B,614 and the result is placed in Buffer A,612 at control block S5 ofchain630, at 5 XOR inFIG. 6A, and Multiply-Read-XOR B to A as indicated at ablock648 inFIG. 6B. Instep6, update footprint is performed at control block F6 ofchain630, read by HW engine605,line6 from HW engine605 toDS610 andNVRAM608 on thelocal controller100A and update footprint on theremotecontroller100B line6 from HW engine605 toDS604 andNVRAM606 inFIG. 6A, and as indicated at ablock650 inFIG. 6B.

Next inStep7, Write data from Buffer A,612 toDisk P618 is performed at control block S7 ofchain630, atline7 from Buffer A,612 toDisk P618 inFIG. 6A, and as indicated at ablock652 inFIG. 6B. Instep8, update footprint is performed at control block F8 ofchain630, read by HW engine605,line8 from HW engine605 toDS610 andNVRAM608 on thelocal controller100A and update footprint on theremotecontroller100B line8 from HW engine605 toremote DS604 andremote NVRAM606 inFIG. 6A, and as indicated at ablock654 inFIG. 6B. Next inStep9, Data is read fromDisk Q622, and XORed with multiplied data from Buffer B,614 and the result is placed in Buffer A,612 at control block S9 ofchain630, at9 XOR inFIG. 6A, and Multiply-Read-XOR B to A as indicated at ablock656 inFIG. 6B. Instep10, Write data from Buffer A,612 toDisk Q622 is performed at control block S10 ofchain630, atline10 from Buffer A,612 toDisk Q622 inFIG. 6A, and as indicated at ablock658 inFIG. 5B.

Instep11, Clear footprint is performed at control block F11 ofchain630, zeros written by HW engine605, atline11 from HW engine605 toDS610 andNVRAM608 on thelocal controller100A and clear footprint on theremote controller100B atline11 from HW engine605 toremote DS604 andremote NVRAM606 inFIG. 6A, and as indicated at ablock660 inFIG. 6B. Instep12, A and B lists for Buffer A,612, and Buffer B,614 are deallocated or depopulated at control block D12 ofchain630, in CS local602 inFIG. 6A, and as indicated at ablock662 inFIG. 6B. Instep13, Send mirrored delete for cache by HW engine605 writing zeros to clear CL onlocal DS610 and to clear CL onremote DS604 at control block M13 ofchain630, atline13 from HW engine605 tolocal DS610 and toremote DS604 inFIG. 6A, and as indicated at ablock664 inFIG. 6B. Instep14, Page lists for Buffer C,616 are de-allocated or depopulated at control block D14 ofchain630, at DS local610 inFIG. 6A, and as indicated at ablock666 inFIG. 6B.

Referring toFIGS. 7A and 7B, there are shown hardware logic operations flow generally designated by thereference character700 and a flow chart inFIG. 7B illustrating exemplary operations performed by apredefined chain pair720 of a plurality of the control blocks selectively arranged to implement an example RAID-5/6 stripe write with cache in accordance with the preferred embodiment. InFIG. 7A, thechain pair720 ofcontrol block308 include control blocks A1, X2, F3, S4, and control blocks F6, D7, M8, and D9, separated by an interaction of firmware (FW)5, with the control blocks308 as defined inFIG. 4B together with the respective steps1-9 shown inFIGS. 7A and 7B.

FIG. 7A includes alocal CS702 of a first orlocal controller100A coupled by ahardware engine705 to aremote DS704 and to aremote NVRAM706 of a second orremote controller100B. Thelocal CS702 is coupled by theHW engine705 to alocal NVRAM708, and to alocal DS710 of thefirst controller100A.Cache Data712 of the first controller are coupled to a plurality ofDrives714 and a sum of products (SOP)engine716 coupled byParity Buffers718 to a pair of theDrives714. For RAID-6, there are twoParity Buffers718 and twoDrives714, while for RAID-5, oneParity Buffer718 and oneDrive714 can be used.

Instep1, Page lists are allocated or populated if needed at control block A1 ofchain pair720, at CS local702 inFIG. 7A, and as indicated at ablock730 inFIG. 7B. Next inStep2,Run SOP engine716 is performed generating parity or P and Q redundancy data at control block X2 ofchain pair720, at2SOP716 inFIG. 7A, and as indicated at ablock732 inFIG. 7B.

Instep3, set footprint is performed at control block F3 ofchain pair720, read byHW engine705,line3 fromHW engine705 toDS710 andNVRAM708 on thelocal controller100A and set footprint on theremotecontroller100B line3 fromHW engine705 toremote DS704 andNVRAM706 inFIG. 7A, and as indicated at ablock734 inFIG. 7B.

Instep4, performing overlapped Write data tomultiple Drives714 is provided as indicated at multiple parallel control blocks S4 ofchain pair720,lines4 fromCache Data712 tomultiple Drives714 inFIG. 7A, and as indicated at ablock736 inFIG. 7B. Firmware optionally takes care of gathering completions of the multiple SAS ops as indicated at ablock FW5 between thechain pair720, and as indicated at ablock738 inFIG. 7B. The firmware operation atFW5 could be implemented with anotherhardware engine120.

Instep6, Clear footprint is performed writing zeros byHW engine705 at control block F6 ofchain720, atline6 fromHW engine705 toDS710 andNVRAM708 on thelocal controller100A and clear footprint on theremote controller100B atline6 fromHW engine705 toremote DS704 andremote NVRAM706 inFIG. 7A, and as indicated at ablock740 inFIG. 7B. Instep7, Page lists are de-allocated or depopulated if needed at control block D7 ofchain720, at CS local702 inFIG. 7A, and as indicated at ablock742 inFIG. 7B. Instep8, Cache update to clear CL writing zeros byhardware engine705 onlocal DS710 and to clear CL onremote DS704 at control block M8 ofchain pair720, atline8 fromhardware engine705 tolocal DS710 and toremote DS704 inFIG. 7A, and as indicated at ablock744 inFIG. 7B. Instep9, Cache page lists are de-allocated or depopulated at control block D9 ofchain pair720, at DS local710 inFIG. 7A, and as indicated at ablock746 inFIG. 7B.

Referring toFIGS. 8A and 8B, there are shown hardware logic operations flow generally designated by thereference character800 and a flow chart inFIG. 8B illustrating exemplary operations for mirroring parity update footprints between dual adapters orcontrollers100 in accordance with the preferred embodiment.

Hardware logic operations flow800 ofFIG. 8A includes afirst controller100A and asecond controller100B. As shown, thecontroller100B includes aninitiator802 for a mirrored write to atarget802 in thecontroller100A. Each of the first controller110A and the second controller110B includes a plurality ofhardware engines120, control store (CS)108 configured to store parity update footprints (PUFP) data, thedata store112 configured to store parity update footprints (PUFP) data, andNVRAM114 configured to store parity update footprints (PUFP) data as shown inFIG. 1.

In accordance with features of the invention, thesecond controller100B operates in the first initiator mode includes firmware (FW) to set PUFP, which is a 32 byte data transfer fromCS108B, addressed by PUFP CS base address and an offset. Theinitiator802 performs PUFP update as follows, write the PUFP data to theinitiator DS112B, addressed by PUFP DS Base address, and write the PUFP data to theinitiator NVRAM114B, addressed by PUFP NVRAM Base address. Theinitiator802 transmits the PUFP update to thetarget802 based upon a configuration bit in an IO control block (IOCB).Target804 writes the PUFP data to the target DS108A, addressed by PUFP DS Base address, and write the PUFP data to thetarget NVRAM114A, addressed by PUFP NVRAM Base address.Target804 transmits a response frame at the end of the operation.

Referring also toFIG. 8B, as indicated at ablock810, Set PUFP, which is a 32 byte data transfer fromCS108B, addressed by PUFP CS base address and an offset, as indicated atline1 from theCS108B to theinitiator802 inFIG. 8A. As indicated at ablock812, theinitiator802 performs PUFP update as follows, writes the PUFP data to theinitiator DS112B, addressed by PUFP DS Base address, writes the PUFP data to theinitiator NVRAM114B, addressed by PUFP NVRAM Base address, andinitiator802 transmits the PUFP update to thetarget802 based upon a configuration bit in an IO control block (IOCB) as indicated atrespective lines2A,2B,2C respectively frominitiator802 to theDS112B,NVRAM114B, and to target804 inFIG. 8A. Theinitiator802 builds the PUFP Data frame with a predefined format including a data store (DS) Base address allowing direct store in DS110A and a NVRAM Base address allowing direct store inNVRAM114A.

As indicated at ablock814, thetarget804 writes the PUFP data to the target DS108A, addressed by PUFP DS Base address, and as indicated at ablock816, thetarget804 writes the PUFP data to thetarget NVRAM114A, addressed by PUFP NVRAM Base address, as indicated bylines3A,3B from thetarget804 to theDS112A, and to theNVRAM114A inFIG. 8A. As indicated at ablock818, thetarget804 transmits a response frame at the end of the operation, as indicated byline4 from thetarget804 to theinitiator802 inFIG. 8A.

FIG. 9 shows a block diagram of anexample design flow900.Design flow900 may vary depending on the type of IC being designed. For example, adesign flow900 for building an application specific IC (ASIC) may differ from adesign flow900 for designing a standard component.Design structure902 is preferably an input to adesign process904 and may come from an IP provider, a core developer, or other design company or may be generated by the operator of the design flow, or from other sources.Design structure902 comprisescircuits100,200,300,308,350,400,500,600,700,800 in the form of schematics or HDL, a hardware-description language, for example, Verilog, VHDL, C, and the like.Design structure902 may be contained on one or more machine readable medium. For example,design structure902 may be a text file or a graphical representation ofcircuits100,200,300,308,350,400,500,600,700,800.Design process904 preferably synthesizes, or translates,circuit100 into anetlist906, wherenetlist906 is, for example, a list of wires, transistors, logic gates, control circuits, I/O, models, etc. that describes the connections to other elements and circuits in an integrated circuit design and recorded on at least one of machine readable medium. This may be an iterative process in which netlist906 is resynthesized one or more times depending on design specifications and parameters for the circuit.

Design process904 may include using a variety of inputs; for example, inputs fromlibrary elements908 which may house a set of commonly used elements, circuits, and devices, including models, layouts, and symbolic representations, for a given manufacturing technology, such as different technology nodes, 32 nm, 45 nm, 90 nm, and the like,design specifications910,characterization data912,verification data914,design rules916, and test data files918, which may include test patterns and other testing information.Design process904 may further include, for example, standard circuit design processes such as timing analysis, verification, design rule checking, place and route operations, and the like. One of ordinary skill in the art of integrated circuit design can appreciate the extent of possible electronic design automation tools and applications used indesign process904 without deviating from the scope and spirit of the invention. The design structure of the invention is not limited to any specific design flow.

Design process904 preferably translates an embodiment of the invention as shown inFIGS. 1,2A,3A,3B,4A,4B,5A,5B,6A,6B,7A7B,8A and8B along with any additional integrated circuit design or data (if applicable), into asecond design structure920.Design structure920 resides on a storage medium in a data format used for the exchange of layout data of integrated circuits, for example, information stored in a GDSII (GDS2), GL1, OASIS, or any other suitable format for storing such design structures.Design structure920 may comprise information such as, for example, test data files, design content files, manufacturing data, layout parameters, wires, levels of metal, vias, shapes, data for routing through the manufacturing line, and any other data required by a semiconductor manufacturer to produce an embodiment of the invention as shown inFIGS. 1,2A,3A,3B,4A,4B,5A,5B,6A,6B,7A,7B,8A and8B.Design structure920 may then proceed to astage922 where, for example,design structure920 proceeds to tape-out, is released to manufacturing, is released to a mask house, is sent to another design house, is sent back to the customer, and the like.

While the present invention has been described with reference to the details of the embodiments of the invention shown in the drawing, these details are not intended to limit the scope of the invention as claimed in the appended claims.

Claims (20)

What is claimed is:

1. A data storage system comprising:

a first controller and a second controller, each of the first controller and the second controller comprising

a plurality of hardware engines;

a control store configured to store parity update footprint (PUFP) data;

a data store;

a nonvolatile random access memory (NVRAM);

one of the first controller or the second controller operating in a first initiator mode builds a PUFP data frame for transferring PUFP data to the other of the first controller or the second controller operating in a target mode;

respective initiator hardware engines transferring PUFP data from the initiator control store, and updating PUFP data, writing PUFP data to the initiator data store and the initiator NVRAM, and transmitting PUFP data to the other of the first controller or the second controller operating in the target mode; and

respective target hardware engines writing PUFP data to the target data store and the target NVRAM, eliminating firmware operations.

2. The data storage system as recited inclaim 1 wherein said respective target hardware engines write PUFP data to the target data store, addressed by PUFP DS Base address.

3. The data storage system as recited inclaim 1 wherein said respective target hardware engines write PUFP data to the target NVRAM, addressed by PUFP NVRAM Base address.

4. The data storage system as recited inclaim 1 wherein said controller operating in the target mode transmits a response frame at the end of the mirrored PUFP write operation.

5. The data storage system as recited inclaim 1 wherein said respective initiator hardware engines write zeros to the initiator data store and the initiator NVRAM to clear PUFP data.

6. The data storage system as recited inclaim 5 wherein said respective initiator hardware engines write zeros to the other of the first controller or the second controller operating in the target mode to clear PUFP data.

7. The data storage system as recited inclaim 5 wherein said respective target hardware engines write zeros to the target data store, addressed by PUFP DS Base address to clear PUFP data.

8. The data storage system as recited inclaim 5 wherein said respective target hardware engines write zeros to the target NVRAM, addressed by PUFP NVRAM Base address to clear PUFP data.

9. A method for implementing storage adapter performance optimization in a data storage system comprising:

providing a first controller and a second controller,

providing each of the first controller and the second controller comprising a plurality of hardware engines; a control store configured to store parity update footprint (PUFP) data; a data store; and a nonvolatile random access memory (NVRAM);

operating one of the first controller or the second controller in a first initiator mode builds a PUFP data frame for transferring PUFP data to the other of the first controller or the second controller operating in a target mode;

respective initiator hardware engines transferring PUFP data from the initiator control store, and updating PUFP data, writing PUFP data to the initiator data store and the initiator NVRAM, and transmitting PUFP data to the other of the first controller or the second controller operating in the target mode; and

respective target hardware engines writing PUFP data to the target data store and the target NVRAM, eliminating firmware operations.

10. The method as recited inclaim 9 wherein said respective target hardware engines write PUFP data to the target data store, addressed by PUFP DS Base address.

11. The method as recited inclaim 9 wherein said respective target hardware engines write PUFP data to the target NVRAM, addressed by PUFP NVRAM Base address.

12. The method as recited inclaim 9 includes said controller operating in the target mode transmitting a response frame at the end of the mirrored PUFP write operation.

13. The method as recited inclaim 9 wherein said respective initiator hardware engines write zeros to the initiator data store and the initiator NVRAM to clear PUFP data.

14. The method as recited inclaim 9 wherein said respective initiator hardware engines write zeros to the other of the first controller or the second controller operating in the target mode to clear PUFP data.

15. The method as recited inclaim 9 wherein said respective target hardware engines write zeros to the target NVRAM, addressed by PUFP NVRAM Base address to clear PUFP data and said respective target hardware engines write zeros to the target data store, addressed by PUFP DS Base address to clear PUFP data.

16. A design structure embodied in a non-transitory machine readable medium used in a design process, the design structure comprising:

a first controller circuit and a second controller circuit tangibly embodied in the machine readable medium used in the design process, said first controller circuit and said second controller circuit for implementing storage adapter performance

optimization in a data storage system, each said controller circuit comprising: a plurality of hardware engines; a control store configured to store parity update footprint (PUFP) data; a data store; a nonvolatile random access memory (NVRAM); one of the first controller or the second controller operating in a first initiator mode builds a PUFP data frame for transferring PUFP data to the other of the first controller or the second controller operating in a target mode;

respective initiator hardware engines transferring PUFP data from the initiator control store, and updating PUFP data, writing PUFP data to the initiator data store and the initiator NVRAM, and transmitting PUFP data to the other of the first controller or the second controller operating in the target mode; and

respective target hardware engines writing PUFP data to the target data store and the target NVRAM, eliminating firmware operations, wherein the design structure, when read and used in the manufacture of a semiconductor chip produces a chip comprising said controller circuit.

17. The design structure ofclaim 16, wherein the design structure comprises a netlist, which describes said controller circuit.

18. The design structure ofclaim 16, wherein the design structure resides on storage medium as a data format used for the exchange of layout data of integrated circuits.

19. The design structure ofclaim 16, wherein the design structure includes at least one of test data files, characterization data, verification data, or design specifications.

20. The design structure ofclaim 16, wherein said respective initiator hardware engines write and said respective target hardware engines write zeros to clear stored PUFP data.

Images