Mass storage access method and device supporting transactionsTechnical Field
The present application relates to the field of computer and network technologies, and in particular, to a method and an apparatus for mass storage access supporting transactions.
Background
With the rapid development of internet technology, internet of things technology and cloud computing technology, a large amount of data information is generated based on a network. When storing these massive data information, on one hand, recording and storing these data information requires a great deal of space and throughput capacity, which often exceeds the range of the capacity of ordinary storage. On the other hand, business application and data acquisition terminals are often five-door, not only the system implementation is complex, but also the data interfaces are various and complementary and compatible, and a unified and convenient data storage interface is lacked.
The existing application program and operating system are closely coupled with the storage system in access, and are not suitable for storage of mass data. This is because, when the conventional storage is used, the application program is implemented by accessing the local operating system driver through the API, which makes the application program and the local system coupled too tightly, and greatly reduces the universality and the lateral expansion capability of the application program. While some web applications may access local storage resources through controls such as ActiveX, these controls also add complexity to local development and maintenance management, as well as introduce network security issues. Therefore, the traditional storage method not only increases the complexity of the application program and the operating system, but also faces a series of problems of complex structure, limited capacity, limited access distance, safety control and the like, and causes difficulty in implementation and use of mass data storage.
The storage problem of mass data is solved, and the distributed network is an economical and effective method. At present, schemes for realizing mass data storage by using a distributed network include a Storage Area Network (SAN) technology, a Network Attached Storage (NAS) technology, an iSCSI (Internet Small Computer System Interface) storage technology, and a Hadoop HDFS/Google GFS/KFS file System service. Among these storage schemes:
storage Area Network (SAN) technology is the connection of SANs (storage area networks) to computers through fibre channel, providing multi-host connectivity in the network, but not through a standard network topology. SANs address the unique issues of enterprise-level storage, namely architectural limitations arising from the tight integration of data with application systems, and limitations of the current Small Computer System Interface (SCSI) standard. SANs are easy to integrate, improve data availability and network performance, and ease management tasks, but are costly and limited by storage capacity and network distribution distance.
Network Attached Storage (NAS) technology provides cross-platform file sharing functionality using a special dedicated data storage server, including storage devices and embedded system software. The NAS can support a variety of protocols (e.g., NFS, CIFS, FTP, HTTP, etc.), and can support a variety of operating systems including Windows, Unix, Linux, etc. However, the disadvantage is that the fixed network access mode based on file sharing imposes constraints on the access performance and interface requirements of the application, and the cost is also high.
iSCSI (Internet Small Computer System Interface) storage technology is a protocol for running SCSI command sets over TCP/IP to establish and manage interconnections between IP storage devices, hosts, clients, and the like, and to create Storage Area Networks (SANs). The iSCSI is widely used in virtualization environments, but the iSCSI storage base of the current main product is still provided on the basis of traditional storage or on the basis of SAN storage, and the cost is still high.
The Hadoop HDFS/GFS/KFS file system service is a high-fault-tolerance, high-throughput and ultra-large-scale data storage service provided by a distributed network formed by using low-cost hardware. The application of the high-fault-tolerance and high-throughput IO service mainly depends on a special API and is tightly coupled with the application.
In addition, the storage devices in these storage schemes all belong to basic hardware devices, and often directly or indirectly require a bus driver, or are more embodied as an intermediate service platform, which is expensive and not beneficial to miniaturization.
Whether in a traditional storage scheme or in a distributed storage scheme, such storage systems are typically only available to file systems and/or applications through an API (application programming interface), and are difficult to access directly by external file systems in a distributed network. At the same time, distributed future storage operations, especially network-based operations, lack transaction control capabilities. The instability of storage access and the difficulty of solution are increased due to the instability of the network system.
How to realize direct and quick storage access of mass data at low cost in such a situation becomes a technical problem which needs to be urgently solved by a person skilled in the art.
Disclosure of Invention
The technical problem to be solved by the application is to provide a method and a device for mass storage access supporting transactions, which can realize direct and rapid storage access of mass data at a low cost.
In order to solve the above problem, the present application discloses a mass storage access method supporting transactions, including: receiving a WEB storage access instruction in a webpage command form; determining a storage space to be used by the WEB storage access instruction, wherein the storage space is located in one or more storage devices in a distributed network system; and performing data storage access on the determined storage space according to the WEB storage access instruction.
Preferably, before the step of receiving a WEB storage access instruction in the form of a WEB page command, the method further includes: setting a storage pool and a Storage Pool Manager (SPM) in at least one server or storage device in the distributed network system; wherein the storage pool includes storage resources in one or more storage devices in the distributed network system, and the SPM is configured to perform pooling and pooling processing on the storage resources in the storage pool.
Preferably, after the step of setting the storage pool in at least one server or storage device in the distributed network system, the method further comprises: dividing the storage resources in the storage pool into at least one storage resource block (SRT) according to a set size; and setting an internal index code IIC and an external service index code EIC for each SRT, wherein the IIC is used for identifying the storage device to which each SRT belongs and the position of each SRT in the storage device, and the EIC is used for identifying the position of each SRT in the storage pool.
Preferably, after the step of setting the inner index coding IIC and the outer service index coding EIC for the SRT, the method further comprises: storing the mapping relation between the IIC and the EIC of all the SRTs in the storage pool; the step of determining the storage space to be used by the WEB storage access instruction comprises the following steps: determining the EIC to be accessed by the WEB storage access instruction; determining the IIC corresponding to the EIC to be accessed according to the stored mapping relation; according to the corresponding IIC, determining the storage equipment to which the SRT identified by the EIC to be accessed belongs and the position of the SRT in the storage equipment; and determining the storage space used by the WEB storage access instruction according to the determined position of the SRT in the storage equipment to which the SRT belongs.
Preferably, the mapping relationship comprises a transaction mapping relationship and a direct mapping relationship; the transaction mapping relation is a Hash mapping relation and is used for mapping IIC and EIC when the target SRT relates to a transaction; the direct mapping relationship is a linear mapping relationship and is used for mapping the IIC and the EIC when the transaction is not involved.
Preferably, after the step of setting the storage pool and the storage pool manager SPM in at least one server or storage device in the distributed network system, the method further includes: and setting a transaction rollback area in the storage pool, wherein the transaction rollback area comprises a transaction index area and a transaction data area, the transaction index area is used for storing information of transactions, and the transaction data area is used for storing the prior backup of target SRTs influenced by the transactions.
Preferably, according to the WEB storage access instruction, the step of performing data storage access on the determined storage space includes at least one of: when the WEB storage access instruction is a write instruction, inserting the transaction information indicated by the write instruction into the transaction index area; determining IIC and EIC of the target SRT of the write instruction, and determining the data of the position of the target SRT in the storage equipment to which the target SRT belongs according to the transaction mapping relation; establishing backup for the data in the transaction data area, and modifying the transaction mapping relation to ensure that the IIC of the target SRT and the EIC of the backup of the transaction data area establish the mapping relation; executing the write instruction on data in the storage device; when the WEB storage access instruction is a read instruction, the read instruction is executed after the write instruction is executed and the write instruction is not submitted, and the transaction information indicated by the read instruction is inserted into the transaction index area; determining IIC and EIC of the target SRT of the read instruction, and determining backup of the target SRT in the transaction data area according to the transaction mapping relation; executing the read instruction on the backup; when the WEB storage access instruction is a commit instruction, setting commit information in the information of the transaction indicated by the commit instruction in the transaction index area; deleting the transaction mapping relation between the IIC of the target SRT of the submitted instruction and the backup EIC; judging whether the transaction data area has a backup of the target SRT, if so, releasing the backup; when the WEB storage access instruction is a rollback instruction, rollback information is set in the information of the transaction indicated by the rollback instruction in the transaction index area; copying the backup of the target SRT in the transaction data area back to the position of the backup in the storage device to which the backup belongs; and deleting the transaction mapping relation between the IIC of the target SRT of the rollback instruction and the backup EIC, and deleting the backup of the target SRT in the transaction data area.
Preferably, the step of performing data storage access on the determined storage space according to the WEB storage access instruction includes: converting the WEB storage access instruction into a storage access instruction suitable for the determined storage space; and performing data storage access on the determined storage space by using the converted storage access instruction.
Preferably, after the step of setting the storage pool and the storage pool manager SPM in at least one server or storage device in the distributed network system, the method further includes: setting a Storage Abstraction Layer (SAL) in at least one server or storage device in the distributed network system, wherein the SAL is used for dividing and managing storage resources in the storage pool; and receiving the WEB storage access instruction, determining a storage space to be used by the WEB storage access instruction, and converting the WEB storage access instruction into a storage access instruction matched with the determined storage space.
Preferably, the WEB storage access instruction comprises a storage management request instruction and a storage data operation request instruction; wherein the storage management request instruction comprises at least one of: the method comprises the steps of storing an initialization instruction, a storage parameter query instruction, a storage transaction configuration instruction, a storage access connection instruction, a storage access closing instruction and a storage transaction processing instruction; the storage data operation request instruction comprises at least one of the following: read instructions, write instructions, copy instructions.
Preferably, the WEB storage access instruction is an extended HTTP protocol instruction.
In order to solve the above problem, the present application further discloses a mass storage access device supporting transactions, which includes a storage abstraction layer SAL; the SAL comprises: the receiving module is used for receiving a WEB storage access instruction in a webpage command form; the determining module is used for determining a storage space to be used by the WEB storage access instruction, wherein the storage space is located in one or more storage devices in a distributed network system; and the execution module is used for performing data storage access on the determined storage space according to the WEB storage access instruction.
Preferably, the transaction-enabled mass storage access device further comprises: and the storage pool manager SPM is used for performing pool entering and pool exiting processing on the storage resources in the storage pool of the distributed network system, wherein the storage pool comprises the storage resources in one or more storage devices in the distributed network system.
Compared with the prior art, the method has the following advantages:
according to the method and the device, the traditional storage idea is converted into the WEB storage idea, the WEB storage access instruction is used for directly accessing the distributed storage resource, the WEB storage access can be directly sent to the storage end through a network protocol, such as an HTTP (hyper text transfer protocol), and the instruction content can be directly analyzed and obtained as long as the storage end supports the network protocol, so that the storage access is carried out. The storage resources do not need to be integrated through the application program like the existing storage access mode, and when the storage access is performed, the API of the application program needs to be called first, and then the storage access instruction is sent to the storage block device through the API and the disk drive, and then the storage access can be performed. By the mass storage access scheme supporting the transaction, an application program is not needed as a storage medium, the storage access process is simplified, and the storage access cost is reduced. Moreover, the expansion of WEB storage is not limited by a bus and a switch, and the storage capacity of mass data can be more effectively realized; and the WEB storage access instruction based on the network protocol is used for storage access, so that the requirements of adapting to various network environments can be reduced, the network configuration is simplified, and meanwhile, the WEB storage access instruction can be conveniently used by various applications and operating systems. Therefore, the technical scheme of the application effectively realizes direct and quick storage and access of mass data at lower cost.
Drawings
FIG. 1 is a flowchart illustrating steps of a method for supporting transactional mass storage access according to a first embodiment of the present application;
FIG. 2 is a flowchart illustrating steps of a method for supporting transactional mass storage access according to a second embodiment of the present application;
FIG. 3 is a flowchart illustrating steps of a method for supporting transactional mass storage access according to a third embodiment of the present application;
FIG. 4 is a schematic structural diagram of the memory side in the embodiment shown in FIG. 3;
fig. 5 is a block diagram of a structure of a mass storage access device supporting transactions according to a fourth embodiment of the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.
Example one
Referring to fig. 1, a flowchart illustrating steps of a method for supporting transactional mass storage access according to an embodiment of the present application is shown.
The mass storage access method supporting the transaction of the embodiment comprises the following steps:
step S102: and receiving a WEB storage access instruction in the form of a webpage command.
When the storage access is performed, a WEB storage access instruction in the form of a WEB page command is used, and the WEB storage access instruction follows a certain network protocol, such as an HTTP protocol, a TCP protocol (transmission control protocol), or a UDP protocol (user datagram protocol). In specific implementation, the WEB storage access instruction may be implemented by a special definition, such as extending a command of a network protocol or making a new command of the network protocol; the present storage access instruction can also be converted into a WEB storage access instruction by a proper way. Of course, those skilled in the art may implement the WEB storage access instruction in any other suitable manner in practical applications.
Step S104: and determining the storage space to be used by the WEB storage access instruction.
At least one storage device exists in the distributed network system, and each storage device has a certain storage resource, namely storage space (internal storage space). In this step, the storage space is located in one or more storage devices in the distributed network system.
Step S106: and performing data storage access on the determined storage space according to the WEB storage access instruction.
The storage end supporting the network protocol can directly analyze the WEB storage access instruction to obtain the content of the storage access instruction, and after the storage space is determined, data storage access can be performed on the determined storage space according to the WEB storage access instruction.
Through the embodiment, the traditional storage idea is converted into the WEB storage idea, the WEB storage access instruction is used for directly accessing the distributed storage resources, the WEB storage access can be directly sent to the storage end through a network protocol, such as an HTTP (hyper text transfer protocol), and the instruction content can be directly analyzed and obtained as long as the storage end supports the network protocol, so that the storage access is performed. The storage resources do not need to be integrated through the application program like the existing storage mode, and when storage access is performed, the API of the application program needs to be called first, and then a storage access instruction is sent to the storage terminal through the API, and then storage access can be performed. Therefore, by the mass storage access scheme supporting the transaction, an application program is not needed as a storage medium, the storage access flow is saved, the storage access speed is increased, and the storage access cost is reduced. Moreover, the expansion of WEB storage is not limited by a bus and a switch, and the mass storage of data can be more effectively realized; and the WEB storage access instruction based on the network protocol is used for storage access, so that the requirements of adapting to various network environments can be reduced, the network configuration is simplified, and meanwhile, the WEB storage access instruction can be conveniently used by various applications and operating systems. The embodiment effectively realizes direct and quick storage and access of mass data at lower cost.
Example two
Referring to fig. 2, a flowchart illustrating steps of a method for supporting transactional mass storage access according to a second embodiment of the present application is shown.
The mass storage access method supporting the transaction of the embodiment comprises the following steps:
step S202: a storage pool, a storage pool manager SPM and a storage abstraction layer SAL are provided in at least one server or storage device in a distributed network system.
Wherein the storage pool comprises storage resources in one or more storage devices in the distributed network system; the SPM is used for performing pool entering and pool exiting treatment on storage resources in the storage pool; the SAL is used for dividing and managing storage resources in the storage pool, receiving a WEB storage access instruction, determining a storage space to be used by the WEB storage access instruction, and preferably, converting the WEB storage access instruction into a storage access instruction matched with the determined storage space.
It should be noted that, in practical applications, the functions of the SPM and the SAL may also be implemented by those skilled in the art in other suitable manners, such as being implemented by a system, or dividing the functions of the SPM and the SAL, and implementing the functions by a plurality of functional modules.
In this embodiment, a storage pool and an SPM are set in one server in the distributed network system, and a plurality of storage devices are set in the distributed network system, where the storage pool includes storage resources in all the storage devices in the distributed network system.
By setting the storage pool, the SPM and the SAL, the access and the management of the storage resources in the distributed network system are facilitated, and the access and the management efficiency of the storage resources are improved.
Step S204: and dividing the storage resources in the storage pool into at least one storage resource block SRT according to a set size.
The size of each SRT is set by the person skilled in the art as a function of the actual situation, and is typically 4K or 8K. In this embodiment, the storage resources in the storage pool are divided into a plurality of SRTs with a size of 4K. The storage resources are divided into a plurality of SRTs with equal sizes, so that unified deployment and management of the storage resources are facilitated.
Step S206: an inner index code IIC and an outer service index code EIC are set for each SRT.
The IIC is used to identify the storage device to which each SRT belongs and its location in the storage device, and the EIC is used to identify the location of each SRT in the storage pool.
Through the IIC, the actual physical address of the SRT in the distributed network system can be quickly determined, and through the EIC, the address of the SRT in the storage pool, the position relation with other SRTs and the like can be quickly determined.
Step S208: the mapping relationship of IIC and EIC of all SRT in the storage pool is saved.
Step S210: and receiving a WEB storage access instruction in the form of a webpage command.
In this embodiment, the SAL may receive a WEB storage access instruction. The WEB storage access instruction carries information of the EIC to be accessed.
Step S212: and acquiring the EIC information from the WEB storage access instruction, and determining the EIC to be accessed.
The information of the EIC to be accessed is carried in the WEB storage access instruction, so that the storage end can determine the actual address of the IIC to be accessed through the stored mapping relation between the IIC and the EIC. On one hand, the EIC format is consistent, the information quantity is small, and the carrying is convenient; on the other hand, when some storage accesses are performed, for example, when data is written, the client does not need to concern about the actually written address, and only one or more EICs are specified according to the information of the EICs, then the storage end determines the IICs corresponding to the EICs, and the data is written after the storage addresses to be written are determined, so that the burden of the client and the information interaction amount between the client and the storage end are greatly reduced.
Step S214: and determining the IIC corresponding to the EIC to be accessed according to the stored mapping relation.
Step S216: and determining the storage device to which the SRT identified by the EIC to be accessed belongs and the position of the SRT in the storage device according to the corresponding IIC.
Step S218: and determining the storage space to be used by the WEB storage access instruction according to the determined position of the SRT in the storage equipment to which the SRT belongs.
Step S220: and converting the WEB storage access instruction into a storage access instruction suitable for the determined storage space.
This step is an optional step. If the format of the storage access instruction used by the server receiving the WEB storage access instruction is consistent with that of the storage access instruction used by the actual storage equipment to be accessed, the received WEB storage access instruction can be directly used for carrying out storage access on the storage space; if the format of the storage access instruction used by the server receiving the WEB storage access instruction is inconsistent with the format of the storage access instruction used by the storage device to be actually accessed, the received WEB storage access instruction needs to be converted into the format same as the format of the storage access instruction used by the storage device to be actually accessed, that is, the received WEB storage access instruction needs to be converted into the storage access instruction suitable for the storage device where the determined storage space is located.
Step S222: and performing data storage access on the determined storage space by using the converted storage access instruction.
The WEB storage access instruction is converted by the storage terminal, so that the client can access different types of storage equipment by using a uniform storage access instruction without paying attention to a specific instruction format for realizing storage, and the efficiency of storage access and the compatibility of a system are effectively improved.
On the basis of effectively realizing direct and rapid storage and access of mass data at lower cost, the embodiment realizes efficient management of storage resources of the distributed network system by setting the storage pool, the SPM and the SAL; by dividing the SRT and setting and storing the mapping relation between the IIC and the EIC, flexible and quick storage access is realized; by converting WEB storage access instructions, the efficiency of storage access and the compatibility of the system are improved.
EXAMPLE III
Referring to fig. 3, a flowchart illustrating steps of a method for supporting transactional mass storage access according to a third embodiment of the present application is shown.
In this embodiment, a storage side is provided with devices such as a storage pool, an SPM, an SAL, a Block Device Driver Interface (BDDI), and a Cache Manager (CM), and accesses and manages a storage resource (in this embodiment, a storage block device SBD), and the structure of the device is shown in fig. 4.
The Storage Block Device (SBD) may be a storage block device such as a disk, an array, a Solid State Disk (SSD), a U-disk, etc. for storing data, the storage block devices are distributed in the distributed network system, and a storage resource group in the storage block device constitutes a storage pool of the distributed network system; the BDDI is a driver program used for providing storage device access in the existing operating system, and the driver program interface can access the disk/whole column device of various interfaces (SCSI, ATA, SATA, PATA), and the BDDI can also access the storage device such as U disk and movable hard disk through PCI, PCI-E, USB, FireWire, Thunderbolt and other buses; a Storage Abstraction Layer (SAL) divides and manages storage units (namely SRTs) of one or more storage block devices, and abstracts the access of a client to the storage block devices to serve the abstract query, access, control and transaction services of the SAL upper layer, that is, the client does not need to consider the specific instruction details of storage access, and can access the storage devices supporting different storage access instruction formats by using uniform storage access instructions; a resource pool manager (SPM) manages metadata for storage pool configuration and storage service control, is used for the pool entering and pool exiting processing of resources (SBD or network storage) in the pool, and is also used for data operation inquiry, modification, backup recovery, transaction management and other matters; the Cache Manager (CM) is used to cache recently accessed data within the memory pool to provide the best IO performance. Wherein, SBD, BDDI and CM can all be realized by adopting the prior art.
In this embodiment, the method for accessing mass storage supporting transactions by using the storage structure includes the following steps:
step S302: and defining a WEB storage access instruction.
The WEB Storage access instruction in this embodiment relates to a Storage management instruction and a Storage data operation instruction, and for this reason, this embodiment defines a WEB Storage access communication protocol websa (WEB Storage access) for the instruction. WebSA is expanded based on an HTTP1.1 protocol, and a series of storage access methods are added on the basis of standard methods such as GET/POST/HEAD and the like in an HTTP request.
Wherein:
the storage management request command is composed of a management method name and management information, and is transmitted by an HTTP request, the management method name corresponds to an HTTP extension method, and the management information corresponds to HTTP entity content and is represented as an entity information body of an XML (or JSON) structure. The storage management instructions include: 1) storing an initialization instruction; 2) storing a parameter query instruction; 3) storing a transaction configuration instruction; 4) storing the access connection instruction; 5) a store access close instruction; 6) the transaction instructions are stored. The storage management response is also implemented based on an HTTP response. The response comprises source request entity information, response state and result data. One storage management instruction based on the HTTP1.1 protocol is shown in table 1.
TABLE 1
| Management instructions | HTTP method | HTTP entity content |
| Initializing storage | InitStorage | <InitByte>0x00</InitByte> |
| Storage parameter query | GetStorageConfig | |
| Storage transaction configuration | SetTranConfig | <RollbackSize>128M</RollbackSize> |
| Storage access connection | OpenSession | <ClientName></ClientName> |
| Memory access closure | CloseSession | <StorageSessionId>Storage001</StorageSessionId> |
| Starting a storage transaction | BeginTrans | <LockRecord>True</LockRecord> |
| Ending a store transaction | EndTrans | <TransId>TransId</TransId> |
| Committing store transactions | CommitTrans | <TransId>TransId</TransId> |
| Rollback store transactions | RollbackTrans | <TransId>TransId</TransId> |
The storage data operation request command consists of an operation command name and operation information, the operation command name corresponds to an HTTP extension method, and the operation information consists of a session identifier, a transaction identifier, a data offset and range, a data update count value and operation data. The session identifier, the transaction identifier, the data offset, and the range, together with the data update count value, correspond to the HTTP entity content, and are represented as an XML (or JSON) entity body. The session identification, transaction identification and data offset and range are implemented as HTTP Tag values and the data information is implemented as HTTP body. The store data operation instructions include: 1) reading an instruction; 2) writing an instruction; 3) the instructions are replicated. Similar to the storage management response, the storage data operation response is also implemented using an HTTP response. The response comprises source request entity information, response state and result data. A storage data operation instruction based on HTTP1.1 protocol is shown in table 2.
TABLE 2
The WEB storage access instruction based on the HTTP1.1 protocol is simple to implement, changes to the existing HTTP1.1 protocol and WEB access are small, implementation is easy, storage access speed is improved, and storage access implementation cost is saved.
It should be noted that the above-mentioned WEB storage access instruction based on the HTTP1.1 protocol is only an exemplary illustration, and a person skilled in the art may define other WEB storage access instructions according to the above-mentioned example, or make appropriate modifications to the above-mentioned WEB storage access instruction, and the like. Of course, as mentioned above, the WEB storage access instruction may also be implemented in other forms, such as instruction conversion, and those skilled in the art may implement the instructions in any appropriate manner according to actual needs when implementing the instructions specifically.
Step S304: storage pools, SPMs and SALs are set.
Specifically, the method comprises the following steps:
step S3042: the SPM manages all SBDs in the pool and assigns them natural numbers, all of which are consecutive and not repeated.
Step S3044: the storage resources of the SBD are divided into standard size storage resource blocks (SRTs). The SRT partition size is in units of 4K, and each SRT has a 64-bit Internal Index Code (IIC) within the SBD device.
Step S3046: the SPM manages all SBDs in the storage pool by using the collection class, and records the metadata of the SBDs. The metadata includes SRT cell number, SRT in-use status, physical device parameters (hard disk head/cylinder/sector, block number, etc.). The SPM aggregates all SRTs, assigns a 64-bit external service index code (EIC) to each SRT, and maintains a mapping relationship between IICs and EICs.
Preferably, the mapping relationship between IIC and EIC is divided into two parts: (1) the direct mapping relation of the IIC and the EIC is used for mapping the IIC and the EIC without transaction. The implementation method is mainly based on the piecewise linear mapping relation formed by the capacity of the SBD. (2) The transaction mapping relation of the IIC and the EIC is used for mapping the IIC and the EIC when the target SRT relates to the transaction, and is used for transaction isolation and dirty reading. The realization method mainly utilizes Hash index to establish mapping relation. In the mapping relation, the direct mapping adopts a linear mapping method, the mapping relation is flexible, and the transaction mapping adopts a Hash mode, so that the mapping and searching speed is high. Different mapping relations are set for whether the transaction is related or not, so that the efficiency of performing storage access by using the mapping relations is improved on one hand, and the flexibility of the storage access is also improved on the other hand.
Step S3048: the SAL layer enables HTTP protocol service for the storage client to initiate storage request, and SSL-based HTTPS service can be adopted when the storage client needs security authentication.
Step S30410: and setting an SAL layer to receive a WEB storage access instruction sent by a client through an HTTP protocol.
That is, the client may send storage management instructions and storage data operation instructions to the SAL layer via the HTTP protocol.
Step S30412: after receiving a storage management instruction and a storage data operation instruction (based on EIC) aiming at the SRT by a client, the SAL layer is set to inquire metadata recorded by the SPM, and the IIC of the SRT influenced by the instruction is calculated according to the mapping relation between the IIC and the EIC. And calling the BDDI interface to execute data operation on the SRT, returning the instruction result to the client, and updating the result into the cache manager CM when the result relates to the SRT data.
Through the arrangement, the management of the storage resources and the control of the storage access are realized.
Preferably, a transaction rollback area may be further set in the storage pool, the transaction rollback area including: (1) the transaction index area is realized by adopting a Hash index and is used for storing transaction information, including transaction marks, transaction states, SRT lists influenced by the transactions and the like; (2) and the transaction data area is used for storing the prior backup of the SRT influenced by the transaction. By setting the transaction index area, the management of storage access transactions is effectively realized, and by setting the transaction data area, the stability and the safety of data storage are effectively ensured.
In this embodiment, the implementation of the storage transaction mainly depends on the transaction rollback area and the mapping relationship between the IIC and the EIC.
Step S306: and realizing WEB storage transaction.
In the step, the storage access instruction is sent by the client, and the storage transaction is processed by the storage terminal. It should be noted that, in the following process of this step, according to the difference of the WEB storage access instruction sent by the client, that is, according to the different storage access purposes realized by the client, one client is described as a management client, a data client, a transaction client, and the like, which are actually the same physical entity.
Specifically, the implementation of the WEB storage transaction includes:
step S3062: the management client accesses SAL of the storage server through a storage transaction configuration instruction (SetTranConfig), the SAL gives the instruction to the SPM, and the SPM reserves SRT with corresponding size for the client by combining the size of a transaction rollback area.
Step S3064: the data client is connected with the HTTP service of the storage server through HTTP or HTTPS authentication.
Step S3066: the data client sends an OpenSession instruction to the storage server, the SAL of the storage server receives the instruction and transmits the instruction to the SPM, and the SPM distributes a Session Id to the storage server through the OpenSession instruction to start a storage session; the SAL returns the results to the client.
Step S3068: the data client sends a BeginTrans instruction to the storage server, the SAL of the storage server receives the instruction and gives the instruction to the SPM, the SPM distributes a TransactionId to the instruction through the BeginTrans instruction, a record with an active state is inserted into the transaction index area, and the SAL returns the result to the client.
Step S30610: a data client initiates a RangeWrite instruction (write instruction) to a storage server, SAL copies a source SRT to a transaction data area, then modifies a transaction index record with a state of 'start', and then updates a transaction mapping relation table of IIC and EIC to enable the IIC and EIC of the SRT copied to the transaction data area to establish a transaction mapping relation; finally, SAL will start to modify the target linear mapped SRT (i.e. source SRT), and return the operation result.
In specific implementation, the RangeWrite instruction carries the EIC information of the target SRT, and the storage server determines the physical location of the target SRT and obtains the data of the location according to the stored mapping relationship between the EIC and the IIC of the target SRT, and then copies the data to the transaction data area to establish backup for the target SRT, and provides the backup for other sessions for reading.
Step S30612: then, other data clients initiate a RangeRead instruction (read instruction) to read the backup SRT of the transaction data area only; only the present session client can read the modified SRT.
The RangeRead instruction also carries the EIC information of the target SRT, and since the transaction mapping relationship between the IIC and the EIC of the target SRT is modified when data is written, other clients can only read the backup SRT of the transaction data area according to the modified transaction mapping relationship. When the written data is not submitted, the non-data writing client reads the backup data information, so that the effectiveness of data reading is ensured, and the stability of storage transactions is also ensured.
Step S30614: if the transaction client side initiates a CommitTrans instruction (commit instruction), the SAL modifies the transaction state in the transaction index area record to be 'commit', then deletes the transaction mapping relation record of the IIC and the EIC, then deletes the data record in the transaction index area, and releases the backup SRT of the transaction data area.
Step S30616: if the transaction client side initiates a rollback instruction, the SAL modifies the transaction state in the transaction index area record to be "rollback", copies the backup SRT of the transaction data area to the original linear mapping SRT, deletes the transaction mapping relation record of the IIC and the EIC, then deletes the data record in the transaction index area, and releases the backup SRT of the transaction data area.
Step S30618: if the transaction client initiates an endrecords instruction (end store transaction instruction), if the transaction status in the transaction index area record is "start", step S30614 is performed, and then the transaction record in the transaction index area is deleted.
Step S30620: if the SPM checks that the client session has timed out and the transaction status in the transaction index area record is "start," step S30616 is performed and the transaction record in the transaction index area is deleted.
In practical applications, for example, independent instructions such as read, commit, rollback, end, and processing of session timeout may be performed by a person skilled in the art by referring to the storage transaction implementation process, and combining the mapping relationship between the EIC and the IIC and the transaction rollback area, to perform corresponding storage access processing.
Through this embodiment, realized: (1) the mass capacity is not limited by a bus and a switch, and the theoretical maximum capacity is expanded to 64 Zbytes; (2) based on the HTTP protocol, the requirements of adapting to various network environments can be reduced, and the network configuration is simplified; meanwhile, the system can be conveniently used by various application and operation systems, and is particularly convenient for various virtual machine systems; (3) the transaction and rollback mechanism is supported, the pressure of the stored transaction on the application when a network problem occurs is avoided, and IO communication and complexity are reduced; (4) the separation of Web service and storage becomes easy, and the separation of an operating system and application based on the Internet and data storage is facilitated; (5) compared with HDFS/GFS and the like: the embodiment is based on a small data block (4K), and HDFS and the like adopt 64M (or higher) data block design, so that the random read-write performance is better; the embodiment is based on the index management of the hash and linear metadata, so that the management cost is reduced; the embodiment directly provides a transaction recovery mechanism based on block recording, reduces data management recovery, and can more effectively adapt to severe network environment; the embodiment is more beneficial to realizing the iSCSI protocol so as to provide support for the existing operating system; and the embodiment is more convenient for the use of HTTP pages, Javascript and the like.
In addition, based on the storage structure of the storage end shown in fig. 4, the application realizes the decoupling of the application program and the storage. Specifically, the implementation process of decoupling the application program from the storage is as follows:
step A: the user establishes a storage pool and issues a storage service.
In the decoupling process, the user is a storage end user. The storage pool device may be a server cluster, or a PC, or a mobile storage device, etc.
And B: a user accesses or downloads an application (e.g., word processing software) or operating system software or virtual machine with a browser.
The browser may be a PC, an intelligent mobile terminal, or the like.
And C: the user manages service configuration and access constraints for the storage pool while modifying configuration parameters for the application, operating system, and virtual machines.
Step D: applications or operating system software use the storage pool services directly, based on HTTP, without services like file system API, ActiveX plug-ins, etc.
By decoupling the application program from the storage, the storage access of the data does not depend on the application program any more, and the direct and quick data storage access based on the network is effectively realized.
Example four
Referring to fig. 5, a block diagram of a mass storage access device supporting transactions according to a fourth embodiment of the present application is shown.
The mass storage access device supporting transactions of the embodiment comprises: the storage abstraction layer SAL502, SAL502 comprising: a receiving module 5022, configured to receive a WEB storage access instruction in the form of a WEB page command; a determining module 5024, configured to determine a storage space to be used by the WEB storage access instruction, where the storage space is located in one or more storage devices in the distributed network system; and the execution module 5026 is configured to perform data storage access on the determined storage space according to the WEB storage access instruction. The SAL502 is disposed in at least one server or storage device in a distributed network system.
SAL502 is also preferably used to partition and manage storage resources in the storage pool.
Preferably, the mass storage access device supporting transactions of the present embodiment further includes: a storage pool manager SPM504 for pooling and pooling storage resources in a storage pool of the distributed network system, wherein the storage pool comprises storage resources in one or more storage devices in the distributed network system. The storage pool and SPMs 504 are located in at least one server or storage device in the distributed network system.
The SAL502, the storage pool, and the SPM504 are preferably located in the same server or storage device, but may also be located in different servers or storage devices, and interaction is achieved through communication between the different servers or storage devices.
Preferably, the SPM504 is further configured to divide the storage resources in the storage pool into at least one storage resource block SRT according to a set size; and setting an internal index code IIC and an external service index code EIC for each SRT, wherein the IIC is used for identifying the storage device to which each SRT belongs and the position of the storage device in the storage device, and the EIC is used for identifying the position of each SRT in the storage pool.
Preferably, the SPM504 is also used to store the IIC and EIC mapping relationships for all SRTs in the storage pool; the determining module 5024 is used for determining the EIC to be accessed by the WEB storage access instruction; determining an IIC corresponding to the EIC to be accessed according to the stored mapping relation; according to the corresponding IIC, determining the storage equipment to which the SRT identified by the EIC to be accessed belongs and the position of the storage equipment; and determining the storage space to be used by the WEB storage access instruction according to the determined position of the SRT in the storage equipment to which the SRT belongs.
Preferably, the mapping relationship comprises a transaction mapping relationship and a direct mapping relationship; the transaction mapping relation is a Hash mapping relation and is used for mapping IIC and EIC when the target SRT relates to a transaction; the direct mapping relationship is a linear mapping relationship used for mapping of IIC and EIC when the transaction is not involved.
Preferably, a transaction rollback area is set in the storage pool, and the transaction rollback area includes a transaction index area and a transaction data area, where the transaction index area is used to store information of the transaction, and the transaction data area is used to store a prior backup of the target SRT affected by the transaction.
Preferably, the execution module 5026 is configured to perform, according to the WEB storage access instruction, data storage access on the determined storage space, where the data storage access includes at least one of: when the WEB storage access instruction is a write instruction, inserting transaction information indicated by the write instruction into a transaction index area; determining IIC and EIC of the target SRT of the write instruction, and determining data of the position of the target SRT in the storage equipment to which the target SRT belongs according to the transaction mapping relation; establishing backup for the data in the transaction data area, and modifying the transaction mapping relation to ensure that the IIC of the target SRT and the EIC of the backup in the transaction data area establish the mapping relation; executing a write instruction on data in a storage device; when the WEB storage access instruction is a read instruction, the read instruction is executed after the write instruction is executed and the write instruction is not submitted, and the transaction information indicated by the read instruction is inserted into a transaction index area; determining IIC and EIC of the target SRT of the reading instruction, and determining backup of the target SRT in a transaction data area according to a transaction mapping relation; executing a read instruction on the backup; when the WEB storage access instruction is a commit instruction, setting commit information in the information of the transaction indicated by the commit instruction in the transaction index area; deleting the transaction mapping relation between the IIC of the target SRT of the submitted instruction and the backup EIC; judging whether a backup of the target SRT exists in the transaction data area, and if so, releasing the backup; when the WEB storage access instruction is a rollback instruction, rollback information is set in the information of the transaction indicated by the rollback instruction in the transaction index area; copying the backup of the target SRT in the transaction data area back to the position of the target SRT in the storage device to which the target SRT belongs; and deleting the transaction mapping relation between the IIC of the target SRT of the rollback instruction and the backup EIC, and deleting the backup of the target SRT in the transaction data area.
Preferably, the execution module 5026 is configured to convert the WEB storage access instruction into a storage access instruction suitable for the determined storage space; and performing data storage access on the determined storage space by using the converted storage access instruction.
Preferably, the WEB storage access instruction comprises a storage management request instruction and a storage data operation request instruction; wherein the storage management request instruction comprises at least one of: the method comprises the steps of storing an initialization instruction, a storage parameter query instruction, a storage transaction configuration instruction, a storage access connection instruction, a storage access closing instruction and a storage transaction processing instruction; the store data operation request instruction includes at least one of: read instructions, write instructions, copy instructions.
Preferably, the WEB storage access command is an extended HTTP protocol command.
Preferably, the mass storage access device supporting transactions of the present embodiment further includes: and the cache manager is used for caching data (which can be data in a preset period of time or a certain amount of data) accessed recently in the storage pool so as to provide the best IO performance.
The overall structural diagram of the storage side of the transaction-supporting mass storage access device provided with the embodiment can refer to fig. 4 and the related description thereof.
The mass storage access device supporting the transaction in this embodiment is used to implement the corresponding mass storage access method supporting the transaction in the foregoing multiple method embodiments, and has the beneficial effect of the corresponding mass storage access method supporting the transaction, which is not described herein again.
The application provides a direct, quick and massive storage system based on a network, traditional file concepts and constraints are skipped, and storage is directly abstracted into storage services. Meanwhile, based on the network characteristics, the transaction support is increased, and the IO efficiency and complexity of transaction processing and recovery are reduced.
After the distributed storage system is realized, the distributed storage system can be used for realizing separation of various operating systems, application programs and data storage besides common virtual machine application. With the storage access technology of the application, an application system or an operating system can be realized through a browser page and Script scripts, and the application does not need to be concerned about local or remote services. And the user can configure and manage the storage pool manager to control the access of the software service provider to the user storage. Such as the user's office software, cameras, will have the ability to directly access the user's storage pool rather than requiring a large number of file transfers or system interfaces as is common.
By the method, the storage is directly converted into Web storage service by using the HTTP, and transaction support is provided, so that the method is suitable for mass storage systems constructed in LAN and Internet environments, and the direct and rapid storage access of mass data is effectively realized at lower cost.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The above detailed description is given to a method and an apparatus for accessing mass storage supporting transactions provided by the present application, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.