US20130047029A1 - Storage system, storage control apparatus, and storage control method - Google Patents

Abstract

A storage control apparatus manages a plurality of storage devices which belong to an RLU such that data is made redundant between different storage devices. If a first storage device fails, a rebuild controller executes a rebuild process to store the same data as recorded in the first storage device in a spare storage device. When the rebuild controller executing the rebuild process has failed in reading out data from a second storage device, a data recovery controller restarts the first storage device, and reads out data to be stored in the spare storage device from the restarted first storage device.

Description

CROSS-REFERENCE TO RELATED APPLICATION
• This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2011-178466, filed on Aug. 17, 2011, the entire contents of which are incorporated herein by reference.
FIELD
• The embodiments discussed herein are related to a storage system, a storage control apparatus, and a storage control method.
BACKGROUND
• In recent years, there has been widely used a storage system using a plurality of storage devices, such as HDDs (Hard Disk Drives). Many of such storage systems have redundant controllers for controlling access to the storage devices in order to improve reliability of data access to the storage devices. For example, there has been proposed a storage system configured such that if both of duplexed controllers are stopped due to a hardware error, the hardware is recovered from the error having temporarily occurred by executing processing for powering off and on the both systems.
• Further, in the storage systems, in general, record control in which data is redundantly stored in two or more storage devices is performed using a RAID (Redundant Arrays of Independent Disks) technique, whereby the safety of recorded data is improved.
• Furthermore, in the storage systems which redundantly control data, if one of storage devices suffers from a failure, data stored in the faulty storage device is rebuilt, and is stored in another storage device. This process is generally referred to as “the rebuild process”. By executing the rebuild process, redundancy of the data is recovered.
• Many of storage systems have a spare storage device provided therein, referred to as the “hot spare”, and often execute the rebuild process using the hot spare. On the other hand, there are also other storage systems that do not use a hot spare, but execute the rebuild process for a new storage device when a faulty storage device is replaced by the new storage device. For example, there has been proposed a storage system configured such that when an error has occurred in one of storage devices managed byRAID 5, parity is rebuilt for a storage device having replaced the faulty storage device, by copying data not from a storage device within the same parity group, but from a storage device at the same location in the mirror configuration for snap shot.
• See, for example, Japanese Laid-Open Patent Publication No. 2004-206239 and Japanese Laid-Open Patent Publication No. 2002-108571.
• By the way, in a storage system, if one of storage devices suffers from a failure in a state in which data has been duplexed, such as a state managed byRAID 5, the rebuild process is executed in a state in which the redundancy of the data is lost. During the rebuild process in the state in which the redundancy of the data is lost as mentioned above, there can further occur a failure in reading out data from still another storage device. This readout failure may be caused e.g. by a partial defect on the surface of a disk medium. If the readout of data from the still other storage device fails in the state in which the redundancy of the data is lost, the data is lost.
SUMMARY
• According to an aspect of the invention, there is provided a storage system including a plurality of storage devices, a storage control apparatus configured to control recording of data in the plurality of storage devices such that data recorded in the plurality of storage devices is made redundant between different storage devices, and a spare storage device configured to be used in place of any of the plurality of storage devices, wherein the storage control apparatus executes, when a first storage device in the plurality of storage devices fails, a rebuild process to store the same data as data recorded in the first storage device in the spare storage device, and reads out, when reading of data from a second storage device in the plurality of storage devices is unsuccessful in the rebuild process, data to be stored in the spare storage device from the first storage device.
• The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
• It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
BRIEF DESCRIPTION OF DRAWINGS
• FIG. 1 illustrates an example of the configuration of a storage system according to a first embodiment;
• FIG. 2 illustrates an example of the entire configuration of a storage system according to a second embodiment;
• FIG. 3 illustrates an example of the hardware configuration of a CM;
• FIG. 4 is a block diagram of an example of the configuration of processing functions of the CM;
• FIG. 5 illustrates an example of information registered in a RAID management table;
• FIG. 6 illustrates an example of information registered in a bad data management table;
• FIG. 7 illustrates an example of information registered in a non-redundant write management table;
• FIG. 8 illustrates an example of information registered in a salvage management table;
• FIG. 9 illustrates an example of a data recording format in a HDD;
• FIG. 10 illustrates an example of a state in which a salvage process is started;
• FIG. 11 illustrates preprocessing for execution of a salvaging method (1);
• FIG. 12 illustrates the salvaging method (1);
• FIG. 13 illustrates preprocessing for execution of salvaging methods (2) and (3);
• FIG. 14 illustrates a procedure of the salvaging method (2);
• FIG. 15 illustrates a procedure of the salvaging method (3);
• FIG. 16 is a flowchart of an example of a rebuild process;
• FIG. 17 is a flowchart of an example of an I/O process executed by an access controller;
• FIG. 18 illustrates an example of a salvage process;
• FIG. 19 is a flowchart (1) of an example of a host read process executed by a controller module according to a third embodiment;
• FIG. 20 is a flowchart (2) of an example of the host read process executed by the controller module according to the third embodiment;
• FIG. 21 is a block diagram of an example of the configuration of processing functions of a controller module according to a fourth embodiment;
• FIG. 22 illustrates an example of information registered in a write management table;
• FIG. 23 illustrates preprocessing for execution of a salvaging method (2a);
• FIG. 24 illustrates the salvaging method (2a);
• FIG. 25 is a flowchart of an example of an I/O process executed in the fourth embodiment; and
• FIG. 26 is a flowchart of an example of a salvage process executed in the fourth embodiment.
DESCRIPTION OF EMBODIMENTS
• Several embodiments will be described below with reference to the accompanying drawings, wherein like reference numerals refer to like elements throughout.
(a) First Embodiment
• FIG. 1 illustrates an example of the configuration of a storage system according to a first embodiment.
• The storage system, denoted byreference numeral1, includes astorage control apparatus10 and a plurality of storage devices. The storage devices included in thestorage system1 are nonvolatile storage devices, such as HDDs and SSDs (Solid State Drives).FIG. 1 illustrates the storage devices, denoted byreference numeral21,22, and31, as examples of the storage devices included in thestorage system1.
• Thestorage control apparatus10 controls access to the storage devices of thestorage system1 according to a request from a host apparatus, not illustrated. Further, thestorage control apparatus10 controls the access processing to the storage devices of which storage areas form logical storage areas, on a logical storage area-by-logical storage area basis. Hereinafter, each logical storage area formed as above is referred to as an RLU (RAID Logical Unit). InFIG. 1, the respective storage areas of thestorage devices21 and22 belong to one RLU, by way of example.
• Thestorage control apparatus10 controls a plurality of storage devices which belong to an RLU such that data recorded in the RLU is made redundant between respective storage areas of different storage devices. By controlling the data as above, even when one of the storage devices which belong to the RLU fails, the data recorded in the RLU is prevented from being lost. Methods used for the RLU management includeRAID 1,RAID 4, andRAID 5.
• Further, thestorage system1 has a spare storage device made ready beforehand for being used in place of any of the storage devices which belong to the RLU. InFIG. 1, it is assumed that thestorage device31 is a spare storage device, by way of example.
• Thestorage control apparatus10 includes arebuild controller11 and adata recovery controller12. Processes executed by therebuild controller11 and thedata recovery controller12 are realized e.g. through executing predetermined programs by a CPU (Central Processing Unit) included in thestorage control apparatus10.
• If one of the storage devices (thestorage device22 in the present example) which belong to the RLU fails (step S1), therebuild controller11 executes “the rebuild process” for storing the same data as recorded in thefaulty storage device22, in the spare storage device31 (step S2).
• When the RLU is managed byRAID 1, therebuild controller11 copies the RLU data read from theother storage device21 which belongs to the RLU into thespare storage device31. Further, when the RLU is managed byRAID 4 or RAID 5 (in this case, the RLU is formed by three or more storage devices), therebuild controller11 reads out data from all of the storage devices belonging to the RLU which are not faulty, calculates data recorded in the faulty storage device based on the read data, and stores the calculated data in the spare storage device.
• If therebuild controller11 has failed in reading data during the rebuild process (step S3), thedata recovery controller12 executes processing for salvaging (recovering) the data to be stored in thespare storage device31.FIG. 1 illustrates a case where therebuild controller11 has failed in reading the data from thestorage device21 during the rebuild process.
• Thedata recovery controller12 attempts to read the data from thefaulty storage device22. For example, thedata recovery controller12 restarts the faulty storage device22 (step S4), and reads out the data to be stored in thespare storage device31, which corresponds to the data which was not successfully read by the rebuild controller11 (step S5). Note that “to restart thefaulty storage device22” in the step S4 means processing for resetting thefaulty storage device22 by powering off and on thestorage device22. For example, when thestorage device22 is powered off after the failure of thestorage device22 has occurred (step S1), thestorage device22 is powered on again in the step S4. On the other hand, when the power of thestorage device22 remains on after the failure has occurred (step S1), thedata recovery controller12 executes reading of the data from thestorage device22 before restarting thestorage device22, for example, and if the reading of the data is unsuccessful, thedata recovery controller12 may restart thestorage device22 and execute reading of the data from the restartedstorage device22 again.
• If reading of the data from thestorage device22 is successful (i.e. if the data has been successfully salvaged), thedata recovery controller12 stores the read data in thespare storage device31.
• Even when data has not been successfully read out during the rebuild process, by executing the above-described salvage process, there is created the possibility of recovery of the data to be stored in thespare storage device31, corresponding to the data which has not been successfully read out. Therefore, it is possible to reduce the possibility of occurrence of data loss.
• Note that even after a failure has occurred in thestorage device22 which belongs to the RLU, thestorage control apparatus10 is capable of continuing the access processing to the RLU in response to a request from the host apparatus using thestorage device21 as the remaining storage device which belongs to the RLU. In the above-described salvage process executed by thedata recovery controller12, thefaulty storage device21, which is not being used for the access processing executed in response to a request from the host apparatus, is restarted. Therefore, the salvage process has little influence on the access processing executed in response to a request from the host apparatus, which makes it possible to prevent the speed of response to the host apparatus from being largely reduced.
(b) Second Embodiment
• FIG. 2 illustrates an example of the entire configuration of a storage system according to a second embodiment.
• The storage system illustrated inFIG. 2, denoted byreference numeral100, includes a CE (Controller Enclosure)200 and a DE (Drive Enclosure)300. Further, ahost apparatus400 is connected to theCE200.
• TheCE200 contains CMs (Controller Modules)201 and202. TheCMs201 and202 each perform reading and writing of data from and into storage devices within theDE300 in response to an I/O (In/Out) request from thehost apparatus400. TheCMs201 and202 manage physical storage areas realized by the storage devices within theDE300 by RAID, and controls access to these physical storage areas.
• TheCMs201 and202 may be connected to each other e.g. via a router. Further, theCE200 may be provided with only one CM, or three or more CMs. However, by providing the CMs in plurality, an access control system with respect to theDE300 is made redundant, which improves reliability of access control processing.
• TheDE300 is provided with a plurality of storage devices each of which is under access control by theCMs201 and202. In the present embodiment, theDE300 is a disk array device provided with HDDs as storage devices. Note that as the storage devices contained in theDE300, it is possible to use other types of nonvolatile storage devices, such as SSDs. Further, a plurality ofDEs300 may be connected to theCE200.
• Thehost apparatus400 requests theCMs201 and202 to access the HDDs in theDE300 according to a user's operation. Thehost apparatus400 is capable of reading out data from the HDDs in theDE300 and writing data into the HDDs in theDE300 via one of theCMs201 and202 e.g. according to a user's operation.
• Note that theCMs201 and202 in theCE200 each have the same configuration, and is capable of executing the same processing. Therefore, only theCM201 will be described hereinafter, and description of theCM202 is omitted.
• FIG. 3 illustrates an example of the hardware configuration of the CM.
• The overall operation of theCM201 is controlled by aCPU211. A RAM (Random Access Memory)212 and a plurality of peripheral devices are connected to theCPU211 via abus217. TheRAM212 is used as a main storage device for theCM201, and temporarily stores part of programs being executed by theCPU211 and various data for use in processing by the programs.
• AnSSD213, aninput interface214, a CA (Channel Adapter)215, and a DI (Drive Interface)216, as examples of the peripheral devices, are connected to theCPU211.
• TheSSD213 is used as a secondary storage device for theCM201, and stores programs executed by theCPU211 and various data for execution of the programs. Note that another type of nonvolatile storage device, such as a HDD, may be used as the secondary storage device.
• Aninput device214aincluding operation keys is connected to theinput interface214. Theinput interface214 outputs a signal indicative of an operation input to theinput device214ato theCPU211.
• TheCA215 executes interface processing for transmitting and receiving data between thehost apparatus400 and theCM201. TheCA215 and thehost apparatus400 communicates with each other e.g. according to the FC (Fibre Channel) standard.
• TheDI216 executes interface processing for transmitting and receiving data between theDE300 and theCM201. TheDI216 and theDE300 communicates with each other e.g. according to the SAS (Serial Attached SCSI, SCSI: Small Computer System Interface) standard.
• FIG. 4 is a block diagram of an example of the configuration of processing functions of the CM.
• TheCM201 includes anaccess controller220, arebuild controller230, and asalvage controller240. Processes executed by theaccess controller220, therebuild controller230, and thesalvage controller240 are realized e.g. by executing predetermined programs by theCPU211 of theCM201.
• Further, the storage device in theCM201 stores a RAID management table250, a bad data management table260, a non-redundant write management table270, and a salvage management table280. These tables are stored e.g. in theSSD213.
• Theaccess controller220 accesses the HDDs in theDE300 in response to an I/O request from thehost apparatus400. For example, when theaccess controller220 receives a data read request, from thehost apparatus400, theaccess controller220 reads out requested data from a predetermined HDD of theDE300, and transmits the read data to thehost apparatus400. On the other hand, when theaccess controller220 receives a data write request from thehost apparatus400, theaccess controller220 writes data to be written, which has been received from thehost apparatus400, into a predetermined HDD of theDE300.
• In the following description, the operation of theaccess controller220 for reading out data from a HDD in response to a read request from thehost apparatus400 is referred to as “host read”. Further, the operation of theaccess controller220 for writing data into a HDD in response to a write request from thehost apparatus400 is referred to as “host write”.
• Further, theaccess controller220 manages data recorded in the HDDs of theDE300 by RAID based on information set in the RAID management table250. Theaccess controller220 manages recorded data according to a predetermined RAID level for each RLU (RAID Logical Unit). The RLU is a logical storage area formed by a combination of logical storage areas in the plurality of HDDs installed in theDE300, and is also referred to as the RAID group.
• The RAID management table250 holds an RLU identification number, a RAID level which is applied to an associated RLU, information indicative of HDDs which belong to the RLU and a logical volume, information indicative of a controlled status of the RLU, and so on, on an RLU-by-RLU basis. Theaccess controller220 refers to the RAID management table250 to thereby determine a HDD into which data is to be host-written and a RAID level used in host-writing the data. Note that in the present embodiment, theaccess controller220 manages each RLU according to the associated RAID level at which data is duplexed. The RAID level at which data is duplexed includesRAID 1,RAID 4, andRAID 5.
• Further, during a time period after one of HDDs belonging to an RLU has failed, thereby causing the redundancy of data recorded in the RLU to be lost, and before the rebuild process for a hot spare HDD becomes ready for being started, when executing host-writing of data into the RLU which has lost the redundancy of the data stored therein, theaccess controller220 registers location information on where the data is host-written, in the non-redundant write management table270.
• When one of the HDDs belonging to the RLU has failed, therebuild controller230 executes “the rebuild process” for writing data recorded in the faulty HDD into a hot spare HDD. Therebuild controller230 refers to the RAID management table250 when executing the rebuild process to thereby determine how to generate the data recorded in the faulty HDD.
• Further, if therebuild controller230 has failed in reading out data from one of the HDDs belonging to the RLU other than the faulty HDD during execution of the rebuild process, therebuild controller230 registers information on a location of the data which has not been successfully read out, in the bad data management table260 and the salvage management table280, and requests thesalvage controller240 to salvage the data which has not been successfully read out. No that when theaccess controller220 receives a request from thehost apparatus400 for reading out data at the location the information on which is registered in the bad data management table260, theaccess controller220 sends an error response to thehost apparatus400.
• Thesalvage controller240 salvages data which has not been successfully read out in the rebuild process executed by therebuild controller230. In the salvage management table280, location information on data to be salvaged (i.e. data which has not been successfully read out in the rebuild process) is registered. Thesalvage controller240 sequentially executes the salvage process on data associated with the location information registered in the salvage management table280 whereby it is possible to execute the salvage process asynchronously with the rebuild process. Further, thesalvage controller240 attempts to salvage data according to several processing patterns, described hereinafter, while consulting the non-redundant write management table270, etc.
• FIG. 5 illustrates an example of information registered in the RAID management table.
• In the RAID management table250, arecord251 is registered on an RLU-by-RLU basis. An RLU number for identifying an RLU associated therewith is given to eachrecord251. In the following description, an RLU number xx is denoted as “RLU #xx”, and an RLU having an RLU number of “RLU #xx” is simply referred to as an “RLU #xx”.
• In each record251, “RAID level”, “disk number”, “logical unit number”, “HS disk number”, and “RAID status” are registered.
• The “RAID level” indicates a RAID level of an associated RLU when in normal operation. In the present embodiment, the “RAID level” is set to one ofRAID 1,RAID 4, andRAID 5.
• The “disk number” indicates an identification number of a HDD to which belongs a physical storage area constituting the associated RLU. Since each RLU is formed by a plurality of physical storage areas in a plurality of respective HDDs, the “disk number” is registered for each of the plurality of HDDs. In the following description, a disk number xx is denoted as “DISK #xx”, and a HDD having the disk number of “DISK #xx” is simply referred to as the “DISK #xx”.
• Further, “disk status” is registered for each HDD identified by the disk number. The “disk status” indicates an operating condition of the corresponding HDD. For example, the “disk status” is set to one of “normal”, “faulty”, and “saved in HS”. The status “normal” indicates that the corresponding HDD is normally operating, and the status “faulty” indicates that the corresponding HDD is faulty. The status “saved in HS” indicates that the rebuild process to rebuild data recorded in the corresponding HDD in a hot spare HDD has been completed, and the RLU is being operated after integrating the hot spare HDD.
• The “logical unit number” indicates an identification number of a logical storage area referred to as the logical unit (or a logical volume), which is set for the associated RLU. It is possible to set a plurality of logical units for one RLU, and the “logical unit number” is registered on a logical unit-by-logical unit basis. In the following description, the logical unit number is referred to as the “LUN” in an abbreviated form. Further, No. xx logical unit is denoted as “LUN #xx”, and a logical unit having a logical unit number of “LUN #xx” is simply referred to as the “LUN #xx”.
• Further, “logical area information” and “physical area information” are registered for each logical unit identified by the “logical unit number”. In the “logical area information”, a range of logical addresses (LBA: logical block address) given to each logical unit is registered. In the “physical area information”, a range of physical addresses in each HDD, allocated to the logical unit, is registered.
• When the LUN and the LBA of data to be accessed are designated, theCM201 identifies a HDD and a location of a block (e.g. stripe number) where data to be accessed is stored, from the RAID level of the RLU to which a logical unit indicated by the LUN belongs and the physical area information on the HDD allocated to the logical unit.
• The “HS disk number” is set only during the rebuild process, and indicates a number for identifying a hot spare HDD used for rebuilding the data.
• The “RAID status” indicates a status of the corresponding RLU. For example, the “RAID status” is set to “normal operation”, “non-redundant”, “rebuild in progress”, or “saved in HS”.
• The status “normal operation” indicates that all HDDs belonging to an RLU suffer from no abnormality and the RLU is normally operating in a state having redundancy of data. The status “non-redundant” indicates that one of HDDs belonging to an RLU is faulty, and redundancy of the data is lost. However, even in a state in which redundancy of the data is lost, when it is after an identification number of a hot spare HDD is registered in the “HS disk number” to make the rebuild process ready and before the rebuild process is completed, the “RAID status” is set to the status “rebuild in progress”. The status “saved in HS” indicates a state in which processing for rebuilding data in the hot spare HDD has been completed, and the RLU is being operated after integrating the hot spare HDD.
• FIG. 6 illustrates an example of information registered in the bad data management table.
• In the bad data management table260, location information on data in a logical unit, which has been determined to be unreadable, by therebuild controller230 or thesalvage controller240, is registered as a combination of a logical unit number (LUN) and a logical address (LBA).
• In the present embodiment, if data has not been successfully read out during the rebuild process, the LUN and LBA associated with the data which has not been successfully read out are registered in the bad data management table260 by therebuild controller230. Further, if the salvage process for data registered in the bad data management table260 is successful, location information on the data is deleted from the bad data management table260. On the other hand, if the salvage process is not successful, the location information on the data remains in the bad data management table260.
• The data structure of the bad data management table260 is not limited to the example illustrated inFIG. 6, but for example, may have a structure in which flag information indicative of whether or not a readout failure has occurred is associated with each of all LBAs of each LUN. Further, in the bad data management table260, for example, a HDD disk number and physical addresses in a HDD may be registered as the location information in place of the LUN and LBA.
• FIG. 7 illustrates an example of information registered in the non-redundant write management table.
• When the RAID status of an RLU is “non-redundant”, and host-writing of data is executed for a logical unit belonging to the RLU, information indicative of a location where data has been written is registered in the non-redundant write management table270 as a combination of a LUN and a LBA.
• The data structure of the non-redundant write management table270 is not limited to the example illustrated inFIG. 7, but for example, may have a structure in which flag information indicative of whether or not host-writing of data has been executed in the status of “non-redundant” is associated with each of all LBAs of each logical unit. Further, for example, a HDD disk number and physical addresses in the HDD may be registered in place of the LUN and LBA, as the location information, in the non-redundant write management table270.
• FIG. 8 illustrates an example of information registered in the salvage management table.
• The salvage management table280 is used for enabling thesalvage controller240 to execute the salvage process asynchronously with the rebuild process by temporarily holding information indicative of data to be salvaged. In the salvage management table280, location information indicative of data to be salvaged (i.e. data which has not been successfully read in the rebuild process) is registered as a combination of a logical unit number (LUN), a logical address (LBA), and a disk number.
• In the salvage management table280, a physical address in a HDD indicated by the disk number may be registered in place of the LUN and LBA. Further, when the data to be salvaged is data managed byRAID 1 in the normal operation, it is not necessary to register information for identifying a HDD, such as a disk number, in the salvage management table280. This is because when data is controlled according toRAID 1, thesalvage controller240 easily determines that the HDD from which data has not been successfully read out is one of the two HDDs belonging to the RLU, which is not faulty.
• FIG. 9 illustrates an example of a data recording format used in the HDD.
• Data recorded in the HDDs of theDE300 is divided into blocks each having a fixed length. Further, in each HDD, a BCC (Block Check Code) is given to a block area where each block is stored. Usually, an error detection code is recorded in the BCC, and when a block is read out, the integrity of a block to be read is checked based on the BCC associated with the block. Further, it is also possible to record information indicative of attributes of an associated block area in the BCC. For example, in a hot spare HDD where data is to be rebuilt, if data to be written into a block area has not been successfully generated during the rebuild process, information indicative of “bad data” is written into the BCC associated with the block area.
• Next, a description will be given of the salvage process. First,FIG. 10 illustrates an example of a state in which the salvage process is started.FIG. 10 illustrates a case where data is controlled according toRAID 1.
• In a “state1” inFIG. 10, theRLU #00 is formed by the storage areas inDISKs #00 and #01, and these storage areas are managed byRAID 1. In this state, if theDISK #01 fails (step S11), therebuild controller230 executes the rebuild process to store data recorded in theDISK #01 of theRLU #00 in aDISK #10 which is a hot spare (step S12). Since theRLU #00 is in a state set toRAID 1, the data of theRLU #00 recorded in theDISK #00 is directly read out, and is copied into the hotspare DISK #10 in the rebuild process.
• When the rebuild process is normally completed, the data of theRLU #00 recorded in theDISK #01 is completely restored in the hotspare DISK #10. At this time, the hotspare DISK #10 is integrated into theRLU #00 in place of thefaulty DISK #01, whereby theaccess controller220 continues host-reading and host-writing of data in a state in which redundancy of the data of theRLU #00 has been completely recovered.
• However, as illustrated in a “state2” inFIG. 10, if therebuild controller230 has failed in reading out data from one block on theDISK #00 during the rebuild process (step S13), it is impossible to copy the data which has not been successfully read out into the hotspare DISK #10, and as a result this data is lost. On the other hand, thesalvage controller240 executes processing for salvaging the data which has not been successfully read out, and stores rebuilt data in a corresponding location in the hotspare DISK #10.
• More specifically, if therebuild controller230 fails in reading out data from theDISK #00 during the rebuild process, therebuild controller230 registers location information on the data which has not been successfully read out, in the salvage management table280 (step S14). Thesalvage controller240 executes the salvage process on the data indicated by the location information registered in the salvage management table280.
• Further, therebuild controller230 registers location information of the data which has not been successfully read out also in the bad data management table260 (step S15). When theaccess controller220 receives a request from thehost apparatus400 for reading out data corresponding to the location information registered in the bad data management table260, theaccess controller220 sends an error response to thehost apparatus400. This prevents unnecessary access to data which has not been successfully read out from being performed by theaccess controller220.
• Note that it is unnecessary to register the location information in the bad data management table260 at this time point. For example, when it is impossible to salvage the data which has been unsuccessfully read out, the location information may be registered in the bad data management table260 by thesalvage controller240.
• Note that the order of the steps S14 and S15 may be reversed, or the steps S14 and S15 may be executed in parallel.
• By the way, as mentioned above, the rebuild process is intended to mean storing of data recorded in a faulty HDD in a HDD where data is to be rebuilt (hot spare HDD in the present embodiment). As in the case ofFIG. 10, in the rebuild process executed for an RLU ofRAID 1, data read out from a HDD which is not faulty is directly copied into a HDD used for rebuilding the data.
• The salvage process is originally a process storing data which has been recorded in a faulty HDD in a HDD used for rebuilding the data using some method even when a readout failure has occurred during the rebuild process. However, the salvage process in the RLU ofRAID 1 is equivalent to a process for storing the same data as the data which has not been successfully read out in a HDD used for rebuilding the data.
• On the other hand, in the rebuild process executed for the RLU ofRAID 4 orRAID 5, the data which has been recorded in a faulty HDD is rebuilt by calculation based on the data read out from the other HDD which is not faulty. Therefore, differently from the salvage process executed for the RLU of theRAID 1, the salvage process executed for the RLU ofRAID 4 orRAID 5 is not equivalent to processing for storing the same data as the data which has not been successfully read out in a HDD used for rebuilding the data.
• In the following description, the salvage process executed in the RLU ofRAID 1 will be mainly described, and a supplementary description of the salvage process executed for the RLU ofRAID 4 orRAID 5 as well will be given on an as-needed basis. Note that in the supplementary description of the salvage process executed for the RLU ofRAID 4 orRAID 5, it is assumed that the RLU is formed by storage areas in theDISKs #00 to #04.
• Thesalvage controller240 executes the salvage process using the following salvaging methods (1) to (3):
• Salvaging method (1): Data is salvaged based on data recorded in a hot spare HDD.
• Salvaging method (2): A faulty HDD is subjected to restarting, and data is salvaged based on data read from the HDD which is at least restarted.
• Salvaging method (3): A HDD from which data has not been successfully read out is subjected to restarting, and data is salvaged based on data read from the HDD which is at least restarted.
• Hereinafter, details of the above-mentioned salvaging methods (1) to (3) will be described.
• FIG. 11 illustrates preprocessing for execution of the salvaging method (1).
• A “state11” inFIG. 11 indicates a case where a failure of theDISK #01 forming theRLU #00 occurred, similarly to the “state1” inFIG. 10. Note that although therebuild controller230 selects a hot spare HDD to be used for rebuilding the data at this time, all storage areas in the hot spare HDD to be used for rebuilding the data are set to a BCC error condition in advance. The term “BCC error condition” used here is intended to mean a condition in which the BCC is set to such a value that when data in a block is read out, a readout failure is detected from the BCC associated with the block. However, in the BCC error condition, it is desirable that the BCC is set to a value other than a value corresponding to information indicative of the above-mentioned bad data. By differentiating the BCC error condition and a condition in which the BCC indicates bad data, when the BCC indicates bad data, it is possible to clearly recognize the impossibility of salvaging data in a block associated with the BCC.
• By the way, even after theDISK #01 has failed, theaccess controller220 continues host-reading and host-writing of data from and into theRLU #00 using theDISK #00 as the remaining HDD constituting theRLU #00 in a state without redundancy of the data. However, as illustrated by a “state12” inFIG. 11, when theaccess controller220 receives a write request from thehost apparatus400 during a time period after the hotspar DISK #10 in which the data is to be rebuilt is made ready for use (specifically, after theDISK #10 has been set to the “HS disk number” in therecord251 associated with theRLU #00 in the RAID management table250) and before the rebuild process is completed (step S21), theaccess controller220 executes host-writing of data not only for theDISK #00, but also for theDISK #10 where the data is to be rebuilt (steps S22 and S23).
• If theRLU #00 is managed byRAID 1, when in the “state12”, theaccess controller220 writes the data requested by thehost apparatus400 into the DISK #00 (step S22), and writes the same data also into a corresponding block on the hot spare DISK #10 (step S23). In the hotspare DISK #10, an error detection code is overwritten in the BCC corresponding to the block in which the data has been written, which makes it possible to normally read out from this block. As a result, even when the rebuild process has not been executed for a block on theDISK #10 into which data has been host-written, the latest data has been written and stored in the block.
• Now, a supplemental description of a case where theRLU #00 is at eitherRAID 4 orRAID 5 is given. When the RLU operated byRAID 4 orRAID 5 is formed by storage areas in theDISKs #00 to #04, the host-write operation during normal operation is performed as follows: Theaccess controller220 divides data requested to be written into data each having a fixed length. Theaccess controller220, for example, calculates a parity P0 based on the sequential divided data D0 to D3. In a case where theRLU #00 is managed byRAID 4, theaccess controller220 writes the divided data D0 to D3 and the parity P0 into predetermined HDDs, respectively. On the other hand, in a case where theRLU #00 is managed byRAID 5, theaccess controller220 writes the divided data D0 to D3 and the parity P0 into theDISKs #00 to #04 in a dispersed manner.
• For example, if writing of new divided data D0 to D3 is requested in a state in which the DISK #04 is faulty, preprocessing illustrated by the “state12” inFIG. 11 is executed as follows: When data to be written into the DISK #04 is divided data D3, theaccess controller220 writes divided data D0 to D2 and the parity P0 calculated based on these divided data items into the other DISKS #00 to #03 (step S22), and writes the divided data D3 into the DISK #10 (step S23). Further, for example, when data to be written into the DISK #04 is the parity P0, theaccess controller220 writes the divided data D0 to D3 into the other DISKS #00 to #03 (step S22), calculates the parity P0 based on the divided data D0 to D3, and writes the calculated parity P0 into the DISK #10 (step S23).
• However, when e.g. only the divided data D3 out of the divided data D0 to D3 is updated, data writing is not performed for the HDDs in which the divided data D0 to D2 are recorded. Therefore, in this case, data writing into the faulty DISK #04 is not necessarily performed, but only when the divided data D3 or the parity P0 were to be written into the DISK #04, the divided data D3 or the parity P0 is written into theDISK #10.
• That is, in all of the cases where theRLU #00 is at respective levels ofRAID 1,RAID 4, andRAID 5, in preprocessing illustrated by the “state12” inFIG. 11, upon receipt of a write request from thehost apparatus400, if data requested to be written includes data to be written into a faulty HDD, theaccess controller220 writes this data into a hot spare HDD.
• FIG. 12 illustrates the salvaging method (1).
• In the rebuild process for the data recorded in theDISK #01, therebuild controller230 reads out data from all of the areas within theDISK #00 from which data is to be read out, irrespective of whether or not data has been written in response to a request from thehost apparatus400 as illustrated by the “state12” inFIG. 11. Let it be assumed, as illustrated by a “state13” inFIG. 12, that during the rebuild process for the data recorded in theDISK #01 executed by therebuild controller230, data was not successfully read from a block on the DISK #00 (step S24). In this case, thesalvage controller240 determines whether or not the data has been written into a block on the hotspare DISK #10, corresponding to the block from which data reading was unsuccessful. This determination is performed by determining whether or not the data is successfully read out from the corresponding block on the DISK #10 (step S25). If the data has been successfully read out from the corresponding block on theDISK #10, this means that the latest data has been stored in the corresponding block by the host-write operation, and hence thesalvage controller240 determines that the data has been successfully salvaged. On the other hand, if a BCC error is detected in reading the data from the block on theDISK #10, thesalvage controller240 determines that the salvaging of the data is unsuccessful. In this case, thesalvage controller240 tries another salvaging method.
• The procedure of the above-described salvaging method (1) illustrated inFIG. 12 is similarly executed in both of the cases where theRLU #00 is at respective levels ofRAID 4 andRAID 5. That is, if data has been successfully read out from a corresponding block on the hot spare DISK #10 (block having the same stripe number as that of the block from which data has not been successfully read out), thesalvage controller240 determines that the data has been successfully salvaged.
• Next,FIG. 13 illustrates preprocessing for execution of the salvaging methods (2) and (3).
• A “state21” inFIG. 13 indicates a case where theDISK #01 constituting theRLU #00 has failed, similarly to the “state1” inFIG. 10. However, the “state21” indicates a state after theDISK #01 has failed and before the hotspare DISK #10 where data is to be rebuilt is made ready for use (specifically, until the “HS disk number” in therecord251 associated with theRLU #00 in the RAID management table250 is set to the DISK #10). In the “state21”, theaccess controller220 continues host-reading and host-writing of data from and into theRLU #00 using theDISK #00 alone.
• As preprocessing for execution of the salvaging methods (2) and (3), when in the “state21”, upon receipt of a write request for theRLU #00 from the host apparatus400 (step S31), theaccess controller220 writes data into a corresponding block on theDISK #00. Also, theaccess controller220 registers location information (LUN and LBA associated with a block into which data has been written) indicative of where the data has been written, in the non-redundant write management table270 (step S32).
• If a host write occurs in the “state21”, the data is written is in a non-redundant state. Therefore, in the non-redundant write management table270, location information on data having no redundancy, out of the data recorded in theRLU #00, is registered.
• Note that the procedure of the above-described preprocessing illustrated in the “state21” is similarly executed in both of the cases where theRLU #00 is at respective levels ofRAID 4 andRAID 5, and theaccess controller220 registers information on a location where host-writing is performed, in the non-redundant write management table270.
• FIG. 14 illustrates the salvaging method (2).
• A “state22” inFIG. 14 indicates a state in which the rebuild process for the hotspare DISK #10 is started from the “state21” inFIG. 13. When theRLU #00 is atRAID 1, therebuild controller230 reads out the data of theRLU #00, recorded in theDISK #00, and copies the read data to theDISK #10. Let it be assumed that therebuild controller230 has failed in reading out the data from theDISK #00 during the rebuild process mentioned above (step S33).
• Thesalvage controller240 determines whether or not host-writing of data has been executed for at least one of the block on theDISK #00 from which data reading was unsuccessful, and a block on theDISK #10 corresponding to the above-mentioned block, after theDISK #01 has failed and until now. More specifically, when thesalvage controller240 has failed in reading out the data from the corresponding block on the hotspare DISK #10 in the above-described salvaging method (1), and execution of host-writing of data into a location corresponding to the block from which data reading was unsuccessful is not registered in the non-redundant write management table270, thesalvage controller240 determines that host-writing of data has not been executed for the corresponding block on at least one of theDISK #00 and theDISK #10, after theDISK #01 has failed and until now (step S34).
• If it is determined that host-writing of data has not been executed, there is a high possibility that the data to be rebuilt in the hotspare DISK #10, corresponding to the data which has not been successfully read out from theDISK #00, exists only in thefaulty DISK #01. From this fact, thesalvage controller240 restarts the faulty DISK #01 (step S35), and checks whether or not it is possible to read out the data to be rebuilt in theDISK #10, from the restartedDISK #01.
• As indicated in a “state23” inFIG. 14, thesalvage controller240 reads out the data from a block on the restartedDISK #01, corresponding to the block on theDISK #00, from which data reading was unsuccessful. If the data has been successfully read out, thesalvage controller240 copies the read data to a block on the hotspare DISK #10, which corresponds to the same stripe number. In this case, the dada is successfully salvaged (step S36).
• Note that the process in the salvaging method (2) illustrated inFIG. 14 is similarly executed in both of the cases where theRLU #00 is at respective levels ofRAID 4 andRAID 5. That is, when the data has been successfully read out from the corresponding block on the restartedDISK #01, thesalvage controller240 copies the read data into a block of the same stripe number in the hotspare DISK #10.
• FIG. 15 illustrates the salvaging method (3).
• Let it be assumed that the rebuild process is started from the “state21” inFIG. 13, and therebuild controller230 has failed in reading out the data from theDISK #00 during the rebuild process (step S41). As illustrated in a “state31” inFIG. 15, thesalvage controller240 determines whether or not execution of host-writing of data for a location corresponding to the block from which data reading was unsuccessful is registered in the non-redundant write management table270.
• If execution of the host writing of data is registered in the non-redundant write management table270, the latest data has not been recorded in the block on theDISK #01, corresponding to the data which has not been successfully read out. Also, the data is not recorded in the block on theDISK #10, corresponding to the data which has not been successfully read out, and this block is supposed to be in the BCC error condition. From this fact, although thesalvage controller240 has failed in reading out the data, thesalvage controller240 restarts theDISK #00 in which the latest data may remain (step S43), and again attempts to read out the data from the location in the restartedDISK #00 from which data reading was unsuccessful. For example, in a case where the failure of the data reading during the rebuild process is caused by a temporary factor, there is a possibility of the data being successfully read out after theDISK #00 is restarted.
• As illustrated in a “state32” inFIG. 15, if the data has been successfully read out from the location in the restartedDISK #00 from which data reading was unsuccessful, thesalvage controller240 salvages the data to be recorded in theDISK #10 based on the read data (step S44). When theRLU #00 is atRAID 1, thesalvage controller240 copies the data read from the restartedDISK #00 into a corresponding block on the hotspare DISK #10.
• Assuming that theRLU #00 is at eitherRAID 4 orRAID 5, the storage areas of theRLU #00 are formed by theDISKs #00 to #04, the DISK #04 has failed, and the data has not been successfully read out from theDISK #00, the following processing is executed in the step S44 in the “state32”: Thesalvage controller240 identifies the HDD from which the data has not been successfully read out, based on the location information registered in the salvage management table280, and restarts the identified HDD (DISK #00 in this example). Thesalvage controller240 generates data to be stored in the hotspare DISK #10 based on the data read out from the restartedDISK #00 and the data read out from blocks, corresponding to the same stripe number, on theDISKs #01 to #03 which remain free from failure.
• For example, let it be assumed that reading of the divided data D0 from theDISK #00 was unsuccessful, and the parity P0 associated with the divided data D0 is recorded in the faulty DISK #04. In this case, thesalvage controller240 calculates the parity P0 based on the divided data D0 read out from the restartedDISK #00 and the divided data D1 to D3 read out from theDISKs #01 to #03 which remain free from failure, and stores the calculated parity P0 in the hotspare DISK #10.
• Further, for example, let it be assumed that reading of the divided data D0 from theDISK #00 was unsuccessful, and the divided data D3 is recorded in the faulty DISK #04. In this case, thesalvage controller240 calculates the divided data D3 based on the divided data D0 read out from the restartedDISK #00, and the divided data D1 and D2 and the parity P0 read out from theDISKs #01 to #03 which remain free from failure, and stores the calculated divided data D3 in the hotspare DISK #10.
• Next, a description will be given of the salvage process using the above-described salvaging methods (1) to (3) with reference to a flowchart. First,FIG. 16 is a flowchart of an example of a rebuild process.
• [Step S101] Upon detection of a failure in a HDD in theDE300, theaccess controller220 notifies therebuild controller230 of the disk number of the faulty HDD, and the RLU number of the RLU to which the faulty HDD belongs. Hereinafter, the description is continued assuming that one of the HDDs belonging to theRLU #00 has failed.
• Therebuild controller230 having received the notification from theaccess controller220 extracts therecord251 of theRLU #00 from the RAID management table250, and updates the “disk status” associated with the disk number of the faulty HDD to “faulty”, and the “RAID status” to “non-redundant” in the extractedrecord251.
• The update of the “disk status” associated with the disk number of the faulty HDD to “faulty” causes the faulty HDD to be separated from theRLU #00. Upon separation of the faulty HDD from theRLU #00, the faulty HDD may be powered off. Alternatively, for example, the faulty HDD may be powered off immediately before the faulty HDD is replaced by a new HDD.
• [Step S102] Therebuild controller230 makes a hot spare HDD where data is to be rebuilt ready for use. More specifically, therebuild controller230 selects a hot spare HDD to be used for rebuilding the data and registers the disk number of the selected HDD in the “HS disk number” in therecord251 of theRLU #00 in the RAID management table250. Further, therebuild controller230 updates the “RAID status” in therecord251 of theRLU #00 to “rebuild in progress”. As a consequence, the hot spare HDD has been made ready for use, whereby therebuild controller230 is allowed to start the rebuild process.
• [Step S103] Therebuild controller230 selects a data area to be rebuilt from the logical units belonging to theRLU #00 on a predetermined number of LBAs-by-a predetermined number of LBAs basis. Therebuild controller230 executes rebuilding data corresponding to the selected data area.
• The phrase “rebuilding data” is intended to mean generating data to be stored in the hot spare HDD, and hereinafter, the generated data is referred to as the “rebuilt data”. For example, when theRLU #00 is normally managed byRAID 1, therebuild controller230 generates rebuilt data by simply reading out data from a HDD belonging to theRLU #00, which is not faulty. On the other hand, when theRLU #00 is normally managed byRAID 4 orRAID 5, therebuild controller230 reads out data from locations of the same stripe number in respective HDDs belonging to theRLU #00, which are not faulty, and generates rebuilt data by calculation based on the read data.
• [Step S104] During generation of the rebuilt data in the step S103, if reading of the data from the HDD which is not faulty is successful (No to the step S104), therebuild controller230 executes a step S105. On the other hand, if reading of the data from the HDD which is not faulty is unsuccessful, and hence the rebuilt data has not been generated (Yes to the step S104), therebuild controller230 executes a step S106.
• [Step S105] Therebuild controller230 stores the rebuilt data generated in the step S103 in a corresponding area in the hot spare HDD.
• [Step S106] Therebuild controller230 registers the LUN and LBA indicative of the data area selected in the step S103, and the disk number of the HDD from which reading of the data in the step S104 was unsuccessful, in the salvage management table280. Further, therebuild controller230 registers the LUN and LBA indicative of the data area selected in the step S103 in the bad data management table260. Note that to which of the salvage management table280 and the bad data management table260 the information is to be first registered is not particularly limited.
• [Step S107] Therebuild controller230 determines whether or not the rebuild process has been completed for all data areas of all logical units which belong to theRLU #00. If the rebuild process has not been completed for all data areas (No to the step S107), therebuild controller230 returns to the step S103, and executes the rebuild process for the next data area. On the other hand, if the rebuild process has been completed for all data areas (Yes to the step S107), therebuild controller230 executes a step S108.
• [Step S108] Therebuild controller230 determines whether or not the salvage process for theRLU #00 by thesalvage controller240 has been completed. Here, when no location information belonging to theRLU #00 is registered in the salvage management table280, therebuild controller230 determines that the salvage process for theRLU #00 has been completed. Note that if data reading during the rebuild process for theRLU #00 has been successful (i.e. if the answer to the question of the step S104 is No), the location information which belongs to theRLU #00 is not registered in the salvage management table280.
• If it is determined that the salvage process for theRLU #00 has been completed (Yes to the step S108), therebuild controller230 updates the “disk status” associated with the faulty HDD to “saved in HS”, and updates the “RAID status” to “saved in HS” in therecord251 of theRLU #00 in the RAID management table250. As a consequence, the rebuild process is completed.
• Next,FIG. 17 is a flowchart of an example of an I/O process executed by the access controller.FIG. 17 illustrates the I/O process executed during a time period after one of the HDDs belonging to theRLU #00 to be subjected to the I/O process has failed and before the rebuild process for theRLU #00 is completed. This time period is a time period during which the “disk status” associated with the faulty HDD is “faulty” and the “RAID status” is the “non-redundant” or “rebuild in progress”, in therecord251 of theRLU #00 in the RAID management table250.
• [Step S121] theaccess controller220 receives an I/O request to theRLU #00 from thehost apparatus400.
• [Step S122] If the I/O request is a read request (Yes to the step S122), a step S123 is executed, whereas if the I/O request is a write request (No to the step S122), a step S126 is executed.
• [Step S123] If theaccess controller220 has received a read request from the host apparatus400 (Yes to the step S122), theaccess controller220 determines whether or not location information on a data area from which data is requested to be read out is registered in the bad data management table260. If the corresponding location information is not registered in the bad data management table260 (No to the step S123), theaccess controller220 executes a step S124. On the other hand, if corresponding location information is registered in the bad data management table260 (Yes to the step S123), theaccess controller220 executes a step S125.
• [Step S124] Theaccess controller220 reads out the data requested to be read out from the HDD, and sends the read data to thehost apparatus400 as a response. That is, theaccess controller220 normally responds to thehost apparatus400.
• [Step S125] Theaccess controller220 sends an error response to thehost apparatus400 since the requested data has not been normally read out.
• Note that if theaccess controller220 has received a read request from the host apparatus400 (Yes to the step S122), theaccess controller220 may execute reading of the data without referring to the bad data management table260. In this case, theaccess controller220 executes the step S124 if the data has been successfully read out, whereas theaccess controller220 executes the step S125 if the data has not been successfully read out. However, when location information of the data to be read is registered in the bad data management table260, there is a high possibility that reading of the data is not normally executed. Therefore, by referring to the bad data management table260, it is possible to prevent unnecessary data access from being executed by theaccess controller220.
• [Step S126] If theaccess controller220 has received a write request from the host apparatus400 (No to the step S122), theaccess controller220 determines whether or not the hot spare HDD is ready for being used for rebuilding the data. More specifically, when in therecord251 of theRLU #00 in the RAID management table250, the “HS disk number” has been set to the disk number of the hot spare HDD and the “RAID status” has been set to “rebuild in progress”, theaccess controller220 determines that the hot spare HDD has been made ready for use.
• If the hot spare HDD has been made ready for use (Yes to the step S126), theaccess controller220 executes a step S127. On the other hand, if the hot spare HDD has not been made ready for use (No to the step S126), theaccess controller220 executes a step S128.
• [Step S127] Theaccess controller220 performs write processing for a predetermined HDD belonging to the RLU “00, which is not faulty. Further, if there is data to be written into the faulty HDD, theaccess controller220 writes the data into the hot spare HDD.
• The details of the write processing executed in the step S127 is as described in the “state12” inFIG. 11. If the data has been written into the hot spare HDD, an error detection code based on the data written into the block is overwritten in the BCC corresponding to the block into which the data has been written. This makes it possible to normally read out the data written in the block, and enables thesalvage controller240 to recognize, during the salvage process, the fact that host-writing of data with respect to the LBA corresponding to the block was executed during the rebuild process.
• For example, in the step S127, even when writing of data into the HDD which is not faulty has not been normally performed, data which makes it possible to salvage the data which has not been written is written into the hot spare HDD. Therefore, even when the data is not successfully read out, during subsequent the rebuild process, from a location into which the data has not been successfully written, thesalvage controller240 is enabled to salvage the data which has not been successfully read out at least based on the data written into the hot spare HDD.
• [Step S128] Theaccess controller220 performs write processing for the predetermined HDD belonging to theRLU #00, which is not faulty.
• [Step S129] Theaccess controller220 registers the location information indicative of a location where the data has been written (LUN and LBA associated with the block into which the data has been written) in the non-redundant write management table270.
• The details of the steps S128 and S129 are as described in the “state21” inFIG. 13. By execution of the step S129, location information on data having no redundancy, out of the data recorded in theRLU #00, is registered in the non-redundant write management table270.
• Next,FIG. 18 illustrates an example of the salvage process. The process inFIG. 18 is executed whenever thesalvage controller240 selects one item of the location information registered in the salvage management table280. Further, thesalvage controller240 identifies, based on the RAID management table250, an RLU for which is set a LUN in the location information selected from the salvage management table280. The following process will be described assuming that theRLU #00 has been identified.
• [Step S141] Thesalvage controller240 recognizes a hot spare HDD (assumed to be theDISK #10 in the present example) from the “HS disk number” in therecord251 of theRLU #00 in the RAID management table250. Thesalvage controller240 attempts to read out data from a block on the hot spare HDD, which is identified from the location information, by the same processing as the step S25 in the “state13” inFIG. 12. In a case where theRLU #00 is atRAID 1, the location from which data is read out in this step is a block on the hot spare HDD where is stored the same data as the data stored in the block from which the data reading was unsuccessful, whereas in a case where theRLU #00 is at eitherRAID 4 orRAID 5, it is a block on the hot spare HDD, corresponding to the same stripe number as that of the block from which data reading was unsuccessful.
• [Step S142] If the data reading in the step S141 is successful (Yes to the step S142), thesalvage controller240 executes a step S151. In this case, the salvaging of data is successful. On the other hand, if the data reading in the step S141 is not successful (No to the step S142), thesalvage controller240 executes a step S143.
• If the data has been successfully salvaged, thesalvage controller240 may further perform, for example, writing of data also into a corresponding block on the HDD from which data reading was unsuccessful (i.e. block from which data reading was unsuccessful during the rebuild process). When theRLU #00 is atRAID 1, thesalvage controller240 writes the data read out from the hotspare DISK #10 into the corresponding block on the HDD from which the data reading was unsuccessful. On the other hand, when theRLU #00 is at eitherRAID 4 orRAID 5, thesalvage controller240 reads out the data in a block of the same stripe number as that of the block from which the data reading was unsuccessful from each of the other non-faulty ones of the HDDs belonging to theRLU #00 than the HDD from which the data reading was unsuccessful. Thesalvage controller240 calculates the data which has not been successfully read out based on these read data and the data read out from the hotspare DISK #10, and writes the calculated data into the block of the same stripe number on the HDD from which the data reading was unsuccessful.
• [Step S143] If a BCC error is detected in the data reading in the step S141 (Yes to the step S143), a step S144 is executed. On the other hand, if the data reading in the step S141 was unsuccessful due to a cause other than a BCC error (No to the step S143), a step S148 is executed. Note that examples of the latter case include a case where the hotspare DISK #10 is faulty.
• [Step S144] Thesalvage controller240 determines whether or not host-writing of data in the non-redundant state has been executed for theRLU #00 during a time period after the HDD in theRLU #00 has failed and before the hotspare DISK #10 is ready for use. More specifically, thesalvage controller240 determines whether or not the LUN and LBA in the location information selected from the salvage management table280 are registered in the non-redundant write management table270. This determination processing corresponds to the determination processing described in the step S34 in the “state22” inFIG. 14 and the step S42 in the “state31” inFIG. 15.
• If the same LUN and LBA are registered in the non-redundant write management table270 (Yes to the step S144), thesalvage controller240 executes the step S148. On the other hand, if the same LUN and LBA are not registered in the non-redundant write management table270 (No to the step S144), thesalvage controller240 executes a step S145.
• [Step S145] If host-writing of data has not been executed for theRLU #00 after the HDD belonging to theRLU #00 has failed and until now (No to the step S142, and Yes to a step S146), it is presumed that the latest data for generating rebuilt data is stored neither in the hotspare DISK #10 nor in the HDD from which the data read was unsuccessful. Then, as illustrated in the “states22 and23” inFIG. 14, thesalvage controller240 attempts to read out the data from the faulty HDD.
• Thesalvage controller240 recognizes the faulty HDD from therecord251 of theRLU #00 in the RAID management table250, and switches off and then on the power of the faulty HDD to thereby restart the faulty HDD. This processing corresponds to the step S35 in the “state22” inFIG. 14. Not that if the power of the faulty HDD has been already switched off, thesalvage controller240 restarts the HDD by simply switching the power on.
• [Step S146] Thesalvage controller240 reads out the data from a block identified from location information on the restarted HDD (i.e. a block on the faulty HDD, corresponding to the block from which the data reading was unsuccessful). In the case where theRLU #00 is atRAID 1, the location from which data is read out in this step is a block on the faulty HDD where is stored the same data as the data stored in the block from which the data reading was unsuccessful, whereas in the case where theRLU #00 is at eitherRAID 4 orRAID 5, it is a block on the faulty HDD, corresponding to the same stripe number of the block from which the data reading was unsuccessful.
• If the data has been successfully read out (Yes to the step S146), thesalvage controller240 executes a step S147, whereas if the data has not been successfully read out (No to the step S146), thesalvage controller240 executes the step S148.
• For example, if the power of the faulty HDD is on at the step S145, thesalvage controller240 may read out the data from the faulty HDD before restarting the faulty HDD. In this case, if the data has been successfully read out, thesalvage controller240 executes the step S147. On the other hand, if the data has not been successfully read out, thesalvage controller240 restarts the faulty HDD, and then determines whether or not the data has been successfully read out by executing the steps S145 and S146.
• [Step S147] Thesalvage controller240 writes the data read from the HDD restarted in the step S146 into a block on the hotspare DISK #10, corresponding to the block from which the data has been read out. As a consequence, the data is successfully salvaged. The above-mentioned steps S146 (in the case of Yes) and S147 correspond to the step S36 in the “state23” inFIG. 14. Thereafter, the step S151 is executed.
• In the step S147, thesalvage controller240 may further write data also into a corresponding block on the HDD from which data reading was unsuccessful (i.e. a block from which data reading was unsuccessful during the rebuild process). When theRLU #00 is atRAID 1, thesalvage controller240 writes the data read from the restarted HDD into a corresponding block on the HDD from which data reading was unsuccessful. On the other hand, when theRLU #00 is at eitherRAID 4 orRAID 5, thesalvage controller240 reads out the data in the block of the same stripe number as the block from which the data reading was unsuccessful, from the other non-faulty HDDs belonging to theRLU #00 than the HDD from which data reading was unsuccessful. Thesalvage controller240 calculates the data which has not been successfully read out based on these read data and the data read from the restarted HDD, and writes the calculated data into the block of the same stripe number on the HDD from which data reading was unsuccessful.
• Further, it is desirable that after completion of the step S147, thesalvage controller240 switches off the operation of the HDD restarted in the step S145, and separates the HDD from theRLU #00. This is because the HDD restarted in the step S145 was once determined to be faulty, and hence has a low possibility of stably operating thereafter.
• [Step S148] If execution of host-writing of data is registered in the non-redundant write management table270 (Yes to the step S144), this indicates that the latest data had been written only into the block from which the data reading was unsuccessful by the host-writing operation before then. Therefore, as illustrated in the “states31 and32” inFIG. 15, thesalvage controller240 attempts to read out data from the HDD to which belongs the block from which the data reading was unsuccessful.
• Thesalvage controller240 recognizes the HDD from which data reading was unsuccessful from the disk number in the location information selected from the salvage management table280, and switches off and then on the power of the HDD, to thereby restart the HDD. This processing corresponds to the step S43 in the “state31” inFIG. 15.
• When theRLU #00 is atRAID 1, thesalvage controller240 recognizes the HDD from which data reading was unsuccessful even without using the disk number registered in the salvage management table280. This is because when in the case ofRAID 1, it is easily determined that a HDD from which data reading was unsuccessful is the non-faulty one of the HDDs belonging to theRLU #00.
• [Step S149] Thesalvage controller240 reads out the data from a block identified from location information on the restarted HDD (i.e. a block from which the data reading was unsuccessful). If the data has been successfully read out (Yes to the step S149), thesalvage controller240 executes a step S150, whereas if the data has not been successfully read out (No to the step S149), thesalvage controller240 executes a step S152.
• [Step S150] Thesalvage controller240 generates rebuilt data based on data read out from at least the restarted HDD, and writes the generated rebuilt data into a corresponding block on the hotspare DISK #10.
• As described in the step S44 in the “state32” inFIG. 15, when theRLU #00 is atRAID 1, thesalvage controller240 writes the data read from the restarted HDD into a corresponding block on the hot spare DISK #10 (block into which the same data is to be stored). Further, when theRLU #00 is managed byRAID 4 or RAI-5, thesalvage controller240 reads out data from the restarted HDD and the respective same stripe numbers in the rest of the HDDs belonging to theRLU #00, which are not faulty, and calculates rebuilt data to be stored in the hotspare DISK #10, based on the read data. Thesalvage controller240 writes the calculated rebuilt data into a block corresponding to the same stripe number in the hotspare DISK #10. By executing the above-described processing, the salvaging of data becomes successful.
• [Step S151] Since the data has been successfully salvaged, thesalvage controller240 deletes the record in the bad data management table260, in which the same LUN and LBA as the location information selected from the salvage management table280 are registered.
• [Step S152] Thesalvage controller240 deletes the location information selected from the salvage management table280 (LUN, LBA, and the disk number) from the salvage management table280.
• Note that if it was impossible to salvage the data using any of the salvaging methods (1) to (3) (No to the step S149), the location information registered in the salvage management table280 is deleted (S152), but the location information registered in the bad data management table260 remains undeleted. When performing host-reading of data after completion of the rebuild process, if an object to be read out corresponds to a location registered in the bad data management table260, theaccess controller220 sends an error response to thehost apparatus400. This enables theaccess controller220 to respond to thehost apparatus400 without unnecessarily accessing the HDD, when a read request for lost data is received.
• If it was impossible to salvage the data (No to the step S149), instead of leaving the location information undeleted from the bad data management table260, thesalvage controller240, for example, may write information indicative of bad data for the BCC of a block on the hot spare HDD identified by the location information (i.e. a block on the hot spare HDD, corresponding to the block from which the data reading was unsuccessful). The “bad data” indicates, for example, that the data in the corresponding block is lost. In this case, when a read request for the data which has not been salvaged after the rebuild process, theaccess controller220 clearly recognizes from the BCC of the corresponding block in the hotspare DISK #10 that the data is lost.
• According to the above-described processes inFIGS. 16 to 18, even when data has not been successfully read out during the rebuild process, it is possible to prevent the data from being lost as much as possible. Therefore, it is possible to enhance the reliability of the storage system.
• In the process inFIG. 18, the salvaging method (1) out of the salvaging methods (1) to (3) is executed first (step S141). This makes it possible to minimize the influence of load of the salvage process on processing of host-writing and host-reading for theRLU #00 to be salvaged.
• Further, in the process inFIG. 18, for example, the salvaging method (2) (steps S145 and S146) and the salvaging method (3) (steps S148 and S149) may be executed in the mentioned order without executing the determination processing in the steps S143 and S144. In this case, the non-redundant write management table270 becomes unnecessary, whereby it is possible to reduce the storage capacity of theCM201.
• Further, the salvaging method (2) restarts a HDD which is not used for host-reading and host-writing data due to a failure. On the other hand, the salvaging method (3) restarts a HDD for use in host-reading and host-writing, and hence a response to thehost apparatus400 is suspended until the operation of the HDD is started again. From this fact, by executing the process using the salvaging method (2) before the process using the salvaging method (3), it is possible to reduce the influence on processing for host-writing and host-reading, and prevent the speed of response to thehost apparatus400 from being reduced as much as possible.
• In the above-described second embodiment, the salvage process is executed asynchronously with the timing in which the data reading was unsuccessful during the rebuild process. However, as another example, when the data reading was unsuccessful, the rebuild process may be interrupted to immediately execute the salvage process. For example, if it is determined in the step S104 inFIG. 16 that the data reading is unsuccessful (Yes to the step S104), the location information is registered in the bad data management table260 (S106 inFIG. 16), and the process inFIG. 18 is executed. However, in a case where the salvage process is executed when the data reading is successful, it is unnecessary to register the data in the salvage management table280 (S106 inFIG. 16), and hence the step S152 inFIG. 18 need not be executed either.
• Further, in the above-described second embodiment, when the data reading is unsuccessful during the rebuild process, the location information is immediately registered in the bad data management table260. However, as another processing example, the location information may not be registered in the bad data management table260 when the data reading is unsuccessful, but thesalvage controller240 may register the location information in the bad data management table260 when it is determined that it is impossible to salvage the data (No to the step S149 inFIG. 18). In this case, when a read request to theRLU #00 being subjected to the rebuild process is received from thehost apparatus400, even if the location from which data is requested to be read out is the location from which the data reading was unsuccessful during the rebuild process, theaccess controller220 once attempts to read the data from the HDD.
(c) Third Embodiment
• In the above-described second embodiment, after the rebuild process for an RLU has been started, when theaccess controller220 receives a read request for the RLU from thehost apparatus400, theaccess controller220 unconditionally sends an error response to thehost apparatus400 in a case where an object to be read out is registered in the bad data management table260. On the other hand, in the third embodiment described hereafter, theaccess controller220 reads out data from a HDD in response to a read request from thehost apparatus400, and if the data reading is unsuccessful, theaccess controller220 causes thesalvage controller240 to execute processing for salvaging the data. This increases the possibility of sending the data requested from thehost apparatus400 as a response even when a HDD failure has occurred.
• Note that in a storage system according to the third embodiment, the hardware configuration and the basic configuration of processing functions of the CM are the same as those of theCM201 according to the second embodiment. Therefore, hereinafter, processing executed by theCM201 according to the third embodiment will be described using the reference numerals appearing inFIG. 4 of the second embodiment.
• FIGS. 19 and 20 are flowcharts of a host-read process executed in the CM according to the third embodiment.
• First, steps S171 to S176 inFIG. 19 will be described.
• [Step S171] In this step, for example, it is assumed that one of the HDDs belonging to theRLU #00 has failed. In this state, upon receipt of a read request for reading out data from theRLU #00 from thehost apparatus400, theaccess controller220 executes the next step S172.
• [Step S172] Theaccess controller220 determines whether or not location information on a data area from which the data is requested to be read out is registered in the bad data management table260. If the corresponding location information is not registered in the bad data management table260 (No to the step S172), theaccess controller220 executes the step S173.
• On the other hand, if the corresponding location information is registered in the bad data management table260 (Yes to the step S172), theaccess controller220 notifies thesalvage controller240 of the corresponding location information, and requests thesalvage controller240 to execute the salvage process. Thesalvage controller240 executes the step S141 inFIG. 20 in response to the execution request.
• [Step S173] Theaccess controller220 reads out the data requested to be read out from the HDD.
• [Step S174] If the data has been successfully read out (Yes to the step S174), theaccess controller220 executes the step S175. On the other hand, if the data has not been successfully read out (No to the step S174), theaccess controller220 notifies thesalvage controller240 of the location information on the data which has not been successfully read out, and requests thesalvage controller240 to execute the salvage process. Thesalvage controller240 executes the step S141 inFIG. 20 in response to the execution request.
• [Step S175] Theaccess controller220 sends the data read out from the HDD in the step S173 to thehost apparatus400 as a response. That is, theaccess controller220 normally responds to thehost apparatus400.
• [Step S176] Theaccess controller220 sends an error response to thehost apparatus400 since the requested data has not been normally read out.
• Next, a process inFIG. 20 will be described. InFIG. 20, the steps in which the same processing as inFIG. 18 is executed are denoted by the same step numbers, and detailed description thereof is omitted.
• Thesalvage controller240 having received the salvage process request from theaccess controller220 attempts to read out the data from the hot spare HDD using the above-described salvaging method (1) (S141). If the data has been successfully read from the hot spare HDD (Yes to the step S142), thesalvage controller240 executes a step S142a.
• [Step S142a] Thesalvage controller240 generates readout data to be sent to thehost apparatus400 as a response based on the data read out from the hot spare HDD. When theRLU #00 is managed byRAID 1, the readout data is the same as the data read out from the hot spare HDD. On the other hand, when theRLU #00 is atRAID 4 orRAID 5, thesalvage controller240 generates readout data by calculation based on the data read out from the hot spare HDD and the data read out from non-faulty ones of the HDDs belonging to theRLU #00 except the HDD from which the data reading was unsuccessful.
• On the other hand, if the data has not been successfully read out from the hot spare HDD (No to the step S142), the cause of the unsuccessful reading is a BCC error (Yes to the step S143), and the corresponding location information is not registered in the non-redundant write management table270 (No to the step S144), thesalvage controller240 executes processing using the above-described salvaging method (2). That is, thesalvage controller240 restarts the faulty HDD (S145), and attempts to read the data from the restarted HDD. If the data has been successfully read out from the restarted HDD (Yes to the step S146), thesalvage controller240 executes a step S147a.
• [Step147a] Thesalvage controller240 generates readout data to be sent to thehost apparatus400 as a response based on the data read from the HDD restarted in the step S145. When theRLU #00 is atRAID 1, the readout data is the same as the data read out from the restarted HDD. On the other hand, when theRLU #00 is at eitherRAID 4 orRAID 5, thesalvage controller240 generates the readout data by calculation based on the data read out from the restarted HDD and the data read out from non-faulty ones of the HDDs belonging to theRLU #00, except the HDD from which the data reading was unsuccessful.
• Further, if the unsuccessful reading of the data from the hot spare HDD is due to a cause other than a BCC error (No to the step S143), or the corresponding location information is registered in the non-redundant write management table270 (Yes to the step S144), or the data reading from the HDD restarted after having failed is unsuccessful (No to the step S146), thesalvage controller240 executes processing using the above-described salvaging method (3). That is, thesalvage controller240 restarts the HDD from which the data reading was unsuccessful (S148), and attempts to read out the data from the restarted HDD.
• In this step, if the data has not been successfully read out from the restarted HDD (No to the step S149), thesalvage controller240 notifies theaccess controller220 of the failure in salvaging the data. Theaccess controller220 having received the notification of failure in the salvaging sends an error response to the host apparatus400 (S176 inFIG. 19).
• On the other hand, if the data has been successfully read out in the step S148 (Yes to the step S149), or after executing the step S142aor S147a, thesalvage controller240 deletes the location information associated with the data to be salvaged, which is registered in the bad data management table260, from the bad data management table260 (S151). Thereafter, thesalvage controller240 notifies theaccess controller220 of success in the salvaging, and passes the readout data generated in the step S142aor S147a, or the data read out from the HDD restarted in the step S148 to theaccess controller220. Theaccess controller220 sends the data received from thesalvage controller240 to thehost apparatus400 as a response (S175 inFIG. 19).
• According to the above-described third embodiment, not only during the rebuild process, but also when a HDD failure has occurred and data reading in response to a read request from thehost apparatus400 is unsuccessful, the salvage process is executed. Therefore, it is possible to increase possibility of normally executing host-reading of data when a HDD failure occurs.
(d) Fourth Embodiment
• In the salvaging method (2) described in the above-described second embodiment, it is determined whether or not host-writing of data has been executed during a time period after a HDD belonging to an RLU has failed and until now, and if host-writing of data has not been executed, the faulty HDD is restarted and data is read out from the restarted HDD. In the salvaging method (2), the above-mentioned determination is performed by performing two determinations of whether or not it is possible to read out the data from the hot spare HDD (S142 inFIG. 18), and whether or not the corresponding location information is registered in the non-redundant write management table270 (S144 inFIG. 18).
• On the other hand, in the salvage process according to the fourth embodiment described hereafter, it is determined using a write management table whether or not host-writing of data has been executed during a time period after a HDD belonging to an RLU has failed and until now. Then, if it is determined based on the write management table that host-writing of data has not been executed, the faulty HDD is restarted, and the data is read out from the restarted HDD. Hereinafter, the above-described salvaging method based on the write management table is referred to as the salvaging method (2a).
• FIG. 21 is a block diagram of an example of the configuration of processing functions of a CM according to the fourth embodiment. InFIG. 21, processing blocks corresponding to those inFIG. 4 are denoted by the same reference numerals.
• In the fourth embodiment, the storage device of theCM201 stores a write management table290 in place of the non-redundant write management table270. In the write management table290, there is registered location information indicative of a writing location where host-writing of data has been executed for the RLU during a time period after a HDD belonging to the RLU has failed and before the rebuild process is completed.
• During a time period after one of the HDDs belonging to the RLU has failed and the redundancy of the data recorded in the RLU is lost and before the rebuild process for a hot spare HDD is completed, when host-writing of data is executed for the RLU, theaccess controller220 registers location information indicative of a location where the data has been written, in the write management table290.
• Thesalvage controller240 refers to the write management table290 in the salvage process, and determines a procedure of the salvage process depending on whether or not the location information associated with the data to be salvaged is registered in the write management table290.
• FIG. 22 illustrates an example of information registered in the write management table.
• When the RAID status of the RLU is in the “non-redundant” status or the “rebuild in progress” status, if host-writing of data is executed for a logical unit which belongs to the RLU, information indicative of a location where the data has been written is registered in the write management table290, as a combination of a LUN and an LBA.
• The data structure of the write management table290 is not limited to the example illustrated inFIG. 22. For example, the write management table290 may be configured such that flag information indicative of whether or not host-writing of data in the “non-redundant” or “rebuild in progress” status has been executed is associated with each of all LBAs of each logical unit. Further, in the write management table290, there may be registered as the location information, for example, a disk number of each HDD and physical addresses in the HDD, in place of the LUN and LBA.
• Next, a description will be given of the salvage process executed in the fourth embodiment.FIG. 23 illustrates preprocessing for execution of the salvaging method (2a).
• A “state41” inFIG. 23 indicates a case where theDISK #01 constituting theRLU #00 has failed, similarly to the “state1” inFIG. 10. However, the “state41” indicates a state during a time period after theDISK #01 has failed and before the rebuild process for the hotspare DISK #10 is completed. This time period includes a time period until the hotspare DISK #10 where data is to be rebuilt is made ready for use, as indicated in the “state21” inFIG. 13.
• As preprocessing of the salvaging method (2a), upon receipt of a write request for writing data into theRLU #00 in the “state41” from the host apparatus400 (step S61), theaccess controller220 writes the data into a corresponding block on the DISK #00 (step S62). Also, theaccess controller220 registers the location information indicative of the location where the data has been written (LUN and LBA associated with the block where the data has been written) in the write management table290 (step S63). The procedure in the step S63 is similarly executed in a case where theRLU #00 is atRAID 4 orRAID 5, and theaccess controller220 registers the information on the location where host-writing of data has been executed in the non-redundant write management table270.
• If the hotspare DISK #10 has been made ready for use, theaccess controller220 may execute host-writing of data not only for theDISK #00 but also for theDISK #10 where data is to be rebuilt, similarly to the step S23 indicated in the “state12” inFIG. 11 (step S64). For example, when theRLU #00 is atRAID 1, in the step S64, theaccess controller220 writes the data requested to be written from thehost apparatus400 also into theDISK #10. Note that the step S64 executed when theRLU #00 is atRAID 4 orRAID 5 is the same as the processing described in the step S23 inFIG. 11. By executing the step S64, it is possible to use the above-described salvaging method (1) in combination.
• FIG. 24 illustrates the salvaging method (2a).
• A “state42” inFIG. 24 indicates a state in which the rebuild process is being executed for the hotspare DISK #10 in the “state41” inFIG. 23. When theRLU #00 is atRAID 1, therebuild controller230 reads out data from theRLU #00, which is recorded in theDISK #00, and copies the read data into theDISK #10. It is assumed here that during the rebuild process as mentioned above, therebuild controller230 has failed in reading the data from the DISK #00 (step S65).
• Thesalvage controller240 determines whether or not host-writing of data has been executed for theRLU #00 after theDISK #01 has failed and until now. If host-writing of data has been executed during this time period, the latest data has been recorded in at least one of a block on theDISK #00 from which data reading was unsuccessful and a block on theDISK #10, which corresponds to the block.
• Thesalvage controller240 performs the above-mentioned determination processing depending on whether or not the location information corresponding to the block from which the data reading was unsuccessful is registered in the write management table290. If the above-mentioned location information is not registered, thesalvage controller240 determines that host-writing of data has not been executed for theRLU #00 during the time period after theDISK #01 has failed and until now (step S66). In this case, there is a high possibility that the data to be rebuilt in the hotspare DISK #10, corresponding to the data which has not been successfully read out from theDISK #00, exists only in thefaulty DISK #01. Therefore, thesalvage controller240 restarts the faulty DISK #01 (step S67), and attempts to read out the data to be rebuilt in theDISK #10 from the restartedDISK #01.
• As illustrated in a “state43” inFIG. 24, thesalvage controller240 reads out the data from the block on the restartedDISK #01, corresponding to the block on theDISK #00 from which the data reading was unsuccessful. If the data has been successfully read out, thesalvage controller240 copies the read data into a corresponding block on the hotspare DISK #10. In this case, the salvaging of data is successful (step S68).
• Note that the salvaging method (2a) illustrated inFIG. 24 is similarly executed in the case where theRLU #00 is at eitherRAID 4 orRAID 5. That is, when the data has been successfully read out from the corresponding block on the restartedDISK #01, thesalvage controller240 copies the read data into a block on the hotspare DISK #10, corresponding to the same stripe number.
• Next, a description will be given of an example of the salvage process as a combination of the above-described salvaging method (2a) and the salvaging methods (1) and (3) with reference to flowcharts.
• FIG. 25 is a flowchart of an example of an I/O process executed in the fourth embodiment.FIG. 25 illustrates the I/O process executed during a time period after one of the HDDs belonging to theRLU #00 to be subjected to the I/O process has failed and before the rebuild process for theRLU #00 is completed. This time period is a time period during which the “disk status” is “faulty” and the “RAID status” is “non-redundant” or the “rebuild in progress”, in therecord251 of theRLU #00 in the RAID management table250.
• Note that inFIG. 25, the steps in which the same processing as inFIG. 17 is executed are denoted by the same step numbers, and detailed description thereof is omitted.FIG. 25 differs fromFIG. 17 in a procedure of processing executed when a write request is received from the host apparatus400 (the step S126 et seq.).
• That is, when theaccess controller220 has received a write request from the host apparatus400 (No to the step S122), theaccess controller220 determines whether or not the hot spare HDD where data is to be rebuilt is ready for use (S126). If the hot spare HDD is ready (Yes to the S126), theaccess controller220 performs processing for writing data into a predetermined HDD belonging to theRLU #00, which is not faulty (S127). Further, when there is data to be written into the faulty HDD, theaccess controller220 writes the data into the hot spare HDD. Thereafter, a step S129ais executed.
• On the other hand, if the hot spare HDD is not ready for use (No to the S126), theaccess controller220 performs processing for writing data into a predetermined HDD belonging to theRLU #00, which is not faulty (S128). Thereafter, the step S129ais executed.
• [Step S129a] Theaccess controller220 registers location information indicative of a location or locations where the data has been written (LUN and LBA corresponding to the block(s) where the data has been written) in the write management table290. As a consequence, information on the location(s) where host-writing of data into theRLU #00 is executed during a time period after one of the HDDs belonging to theRLU #00 has failed and before the rebuild process is completed is registered in the write management table290.
• FIG. 26 is a flowchart of an example of the salvage process executed in the fourth embodiment. The process inFIG. 26 is executed whenever thesalvage controller240 selects one item of the location information registered in the salvage management table280, similarly to the process inFIG. 18. Further, thesalvage controller240 identifies the RLU which is set to the LUN in the location information selected from the salvage management table280, based on the RAID management table250. The following description is given assuming that theRLU #00 is identified.
• InFIG. 26, the steps in which the same processing as inFIG. 18 is executed are denoted by the same step numbers, and detailed description thereof is omitted.
• [Step S161] Thesalvage controller240 determines whether or not host-writing of data has been executed for theRLU #00 in the non-redundant state during a time period after a HDD of theRLU #00 has failed and until now. More specifically, thesalvage controller240 determines whether or not the LUN and LBA in the location information selected from the salvage management table280 are registered in the write management table290.
• If the same LUN and LBA are registered in the write management table290 (Yes to the step S161), it is presumed that there is a high possibility that the latest data is recorded, by host-writing, in either the hotspare DISK #10 or the HDD from which the data read was unsuccessful. Then, thesalvage controller240 sequentially attempts to execute the salvage processing using the salvaging method (1) and the salvage processing using the salvaging method (3) in the mentioned order.
• First, thesalvage controller240 reads out the data from the hotspare DISK #10 using the salvaging method (1) (S141). If the data has been successfully read out from the hot spare DISK #10 (Yes to the step S142), this means that the salvaging of data is successful, and hence the step S151 is executed.
• If the data has not been successfully read out from the hot spare DISK #10 (No to the step S142), thesalvage controller240 restarts the HDD to which the block from which the data reading was unsuccessful belongs using the salvaging method (3) (S148), and attempts to read out the data from the restarted HDD (S149). If the data has been successfully read out from the restarted HDD (Yes to the step S149), thesalvage controller240 generates rebuilt data based on data read out from at least the restarted HDD, and writes the generated rebuilt data into a corresponding block in the hot spare DISK #10 (S150). Thereafter, the step S151 is executed. On the other hand, if the data has not been successfully read out from the restarted HDD (No to the step S149), this indicates that the salvaging of data is unsuccessful, and hence the step S152 is executed.
• Compared with the process inFIG. 18, in the above-described steps S161 (Yes), S141, S142 (No), S148, S149 (Yes), and S150, reading of data from the HDD from which data reading was unsuccessful is executed without executing reading of data from the faulty HDD. Therefore, in a case where the data is successfully salvaged based on the data read out from the HDD from which the data reading was unsuccessful, time taken before the data is successfully salvaged is reduced.
• Note that if it is determined in the step S161 that the corresponding location information is registered in the write management table290, the step S148 may be directly executed without executing the steps S141 and S142. In this case, inFIG. 25, the steps S126 and S127 are unnecessary, and hence when theaccess controller220 receives a write request (No to the step S122), theaccess controller220 may unconditionally execute the steps S128 and S129a.
• On the other hand, if it is determined in the step S161 that the same LUN and LBA are not registered in the write management table290 (No to the step S161), it is presumed that the latest data is recorded neither in the hotspare DISK #10 nor in the HDD from which the data reading was unsuccessful. Therefore, thesalvage controller240 restarts the faulty HDD using the salvaging method (2a) (S145), and attempts to read out the data from the restarted HDD (S146).
• If the data has been successfully read out from the restarted HDD (Yes to the step S146), thesalvage controller240 writes the read data into a block on the hotspare DISK #10, corresponding to the block from which the data has been read out (S147). As a consequence, the salvaging of data is successful, and hence the step S151 is executed. On the other hand, if the data has not been successfully read out from the restarted HDD (No to the step S146), this means that the salvaging of data is unsuccessful, and the step S152 is executed.
• In the above-described steps S161 (No), S145, S146 (Yes), and S147, reading of data from the faulty HDD is executed without attempting to read out data from the hotspare DISK #10 differently from the process inFIG. 18. Therefore, time taken before data is successfully salvaged is reduced. On the other hand, if the reading of the data from the faulty HDD is unsuccessful (No to the step S146), it is determined that the salvaging of data is unsuccessful without restarting the HDD from which the data reading was unsuccessful during the rebuild process. Therefore, it is possible to reduce probability of stoppage of the I/O process for theRLU #00 executed by theaccess controller220.
• In the above-described fourth embodiment, similarly to the second embodiment, the rebuild process and the salvage process are asynchronously executed. However, at a time point of failing in reading out data during the rebuild process, the salvage process illustrated inFIG. 26 may be executed.
• Further, the salvage process illustrated inFIG. 26 may be executed when data is not successfully read out during host-reading.
• Further, in the non-redundant write management table270 used in the above-described second embodiment, location information is registered if a host write occurs during a time period after a HDD has failed and before a hot spare HDD where data is to be rebuilt is made ready for use. On the other hand, in the write management table290 used in the fourth embodiment, location information is registered if a host write occurs during a time period after a HDD has failed and before the rebuild process is completed. Therefore, the non-redundant write management table270 is more likely to make it possible to reduce the amount of data than the write management table290. That is, the second embodiment makes it possible to reduce the capacity of a table for storing information on the occurrence of a host write, compared with the fourth embodiment.
• According to the storage system, storage control apparatus, and storage control method, described above, it is possible to reduce the probability of losing data.
• All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.

Claims (20)

1. A storage system comprising:

a plurality of storage devices;

a storage control apparatus configured to control recording of data in the plurality of storage devices such that data recorded in the plurality of storage devices is made redundant between different storage devices; and

a spare storage device configured to be used in place of any of the plurality of storage devices,

wherein the storage control apparatus:

executes, when a first storage device in the plurality of storage devices fails, a rebuild process to store the same data as data recorded in the first storage device in the spare storage device; and

reads out, when reading of data from a second storage device in the plurality of storage devices is unsuccessful in the rebuild process, data to be stored in the spare storage device from the first storage device.

2. The storage system according toclaim 1, wherein the storage control apparatus further executes access processing to access data in the plurality of storage devices in response to an access request from a host apparatus, and

wherein, when the first storage device is in failure in the access processing, the storage control apparatus continues the access processing in response to the access request from the host apparatus, using remaining ones of the plurality of storage devices except the first storage device.

3. The storage system according toclaim 2, wherein the storage control apparatus:

writes, upon receipt of a write request including data to be written into the first storage device, from the host apparatus during execution of the rebuild process, the data into the spare storage device,

performs, when reading of data from the second storage device is unsuccessful in the rebuild process, determination as to whether or not data has been written by the access processing into a location on the spare storage device, associated with the data whose reading is unsuccessful,

determines, when the data has been written, that the rebuild process with respect to the data has been completed, and

reads out, when the data has not been written, data to be stored in the spare storage device from the first storage device.

4. The storage system according toclaim 3,

wherein the storage control apparatus:

reads out, when the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful, the data to be stored in the spare storage device from the first storage device, and

restarts, when reading of the data to be stored in the spare storage device from the first storage device is unsuccessful, the second storage device, and reads out the data whose reading is unsuccessful, from the restarted second storage device again.

5. The storage system according toclaim 3,

wherein the storage control apparatus:

determines, when the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful, whether or not the data has been written by the access processing into a location on the second storage device, where the reading of the data is unsuccessful, during a time period after the first storage devices has failed and before the spare storage device becomes available for writing data by the access processing;

reads out, when the data has not been written, the data to be stored in the spare storage device from the first storage device; and

restarts, when the data has been written, the second storage device, and reads out the data whose reading is unsuccessful, from the restarted second storage device again.

6. The storage system according toclaim 3, wherein:

all storage areas in the spare storage device are set to a state in which a readout error occurs, in advance; and

the storage control apparatus:

executes, when the reading of data from the second storage device is unsuccessful in the rebuild process, reading of data from the location on the spare storage device, associated with the data whose reading is unsuccessful,

determines, when the data has been normally read out, that the rebuild process with respect to the data has been completed, and

determines, when a data readout error occurs, that the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful.

7. The storage system according toclaim 2,

wherein the storage control apparatus:

determines, when reading of data from the second storage device is unsuccessful in the rebuild process, whether or not data has been written by the access processing into a location on the second storage device, where the reading of the data is unsuccessful, during a time period after the first storage devices has failed and until now;

reads out, when the data has not been written, the data to be stored in the spare storage device from the first storage device; and

restarts, when the data has been written, the second storage device, and reads out the data whose reading is unsuccessful, from the restarted second storage device again.

8. The storage system according toclaim 2, wherein after the first storage device has failed, when reading of data from the second storage device in response to a read request from the host apparatus has failed, the storage control apparatus reads out data from a location on the first storage device, associated with data requested to be read out.

9. A storage control apparatus that controls recording of data in a plurality of storage devices such that data recorded in the plurality of storage devices is made redundant between different storage devices, the storage control apparatus comprising one or more processors configured to perform a procedure including:

executing, when a first storage device in the plurality of storage devices fails, a rebuild process to store the same data as data recorded in the first storage device in the spare storage device; and

reading out, when reading of data from a second storage device in the plurality of storage devices is unsuccessful in the rebuild process, data to be stored in the spare storage device from the first storage device.

10. The storage control apparatus according toclaim 9, wherein the procedure further includes access processing to access data in the plurality of storage devices in response to an access request from a host apparatus, and

wherein, when the first storage device is in failure in the access processing, the access processing in response to the access request from the host apparatus continues using remaining ones of the plurality of storage devices except the first storage device.

11. The storage control apparatus according toclaim 10, wherein in the access processing, upon receipt of a write request including data to be written into the first storage device, from the host apparatus during execution of the rebuild process, the data is written into the spare storage device, and

wherein the procedure further includes:

performing, when reading of data from the second storage device is unsuccessful in the rebuild process, determination as to whether or not data has been written by the access processing into a location on the spare storage device, associated with the data whose reading is unsuccessful, and

reading out, when the data has not been written, data to be stored in the spare storage device, from the first storage device.

12. The storage control apparatus according toclaim 11, wherein the procedure further includes:

reading out, when the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful, the data to be stored in the spare storage device from the first storage device, and

restarting, when reading of the data to be stored in the spare storage device from the first storage device is unsuccessful, the second storage device, and reading out the data whose reading is unsuccessful, from the restarted second storage device again.

13. A storage control method for a storage control apparatus that controls recording of data in a plurality of storage devices such that data recorded in the plurality of storage devices is made redundant between different storage devices, the storage control method comprising:

executing, when a first storage device in the plurality of storage devices fails, a rebuild process to store the same data as data recorded in the first storage device in the spare storage device; and

reading out, when reading of data from a second storage device in the plurality of storage devices is unsuccessful in the rebuild process, data to be stored in the spare storage device from the first storage device.

14. The storage control method according toclaim 13, further comprising executing access processing to access accessing data in the plurality of storage devices in response to an access request from a host apparatus,

wherein, when the first storage device is in failure in the access processing, the access processing in response to the access request from the host apparatus continues using remaining ones of the plurality of storage devices except the first storage device.

15. The storage control method according toclaim 14, wherein:

the access processing includes, upon receipt of a write request including data to be written into the first storage device, from the host apparatus during execution of the rebuild process, writing the data into the spare storage device; and

the storage control method further includes:

performing, when reading of data from the second storage device is unsuccessful in the rebuild process, determination as to whether or not data has been written by the access processing into a location on the spare storage device, associated with the data whose reading is unsuccessful,

determining, when the data has been written, that the rebuild process with respect to the data has been completed, and

reading out, when the data has not been written, data to be stored in the spare storage device from the first storage device.

16. The storage control method according toclaim 15, further comprising:

reading out, when the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful, the data to be stored in the spare storage device from the first storage device, and

restarting, when reading of the data to be stored in the spare storage device from the first storage device is unsuccessful, the second storage device, and reading out the data whose reading is unsuccessful, from the restarted second storage device again.

17. The storage control method according toclaim 15, further comprising:

determining, when the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful, whether or not the data has been written by the access processing into a location on the second storage device, where the reading of the data is unsuccessful, during a time period after the first storage devices has failed and before the spare storage device becomes available for writing data by the access processing;

reading out, when the data has not been written, the data to be stored in the spare storage device from the first storage device; and

restarting, when the data has been written, the second storage device, and reading outs the data whose reading is unsuccessful, from the restarted second storage device again.

18. The storage control method according toclaim 15, wherein:

all storage areas in the spare storage device are set to a state in which a readout error occurs, in advance; and

the storage control method further includes:

executing, when the reading of data from the second storage device is unsuccessful in the rebuild process, reading of data from the location on the spare storage device, associated with the data whose reading is unsuccessful,

determining, when the data has been normally read out, that the rebuild process with respect to the data has been completed, and

determining, when a data readout error occurs, that the data has not been written by the access processing into the location on the spare storage device, associated with the data whose reading is unsuccessful.

19. The storage control method according toclaim 14, further comprising:

determining, when reading of data from the second storage device is unsuccessful in the rebuild process, whether or not data has been written by the access processing into a location on the second storage device, where the reading of the data is unsuccessful, during a time period after the first storage devices has failed and until now;

reading out, when the data has not been written, the data to be stored in the spare storage device from the first storage device; and

restarting, when the data has been written, the second storage device, and reads out the data whose reading is unsuccessful, from the restarted second storage device again.

20. The storage control method according toclaim 14, further comprising, after the first storage device has failed, when reading of data from the second storage device in response to a read request from the host apparatus has failed, reading out data from a location on the first storage device, associated with data requested to be read out.

Images