Disclosure of Invention
Accordingly, embodiments of the present application provide a storage system, a data storage method, apparatus, a product, and a medium to overcome or at least partially solve the above-described problems.
A first aspect of an embodiment of the present application provides a storage system, including:
a first storage cluster and a second storage cluster;
A metadata unified namespace is deployed in the first storage cluster;
The first storage clusters comprise a plurality of first storage nodes, the second storage clusters comprise a plurality of second storage nodes, the area distance between the first storage clusters and the service system is smaller than a preset distance, the area distance between the second storage clusters and the service system is not smaller than the preset distance, the areas of the plurality of second storage nodes in the second storage clusters are different, and the plurality of second storage nodes are respectively connected into the first storage clusters through a target protocol;
The metadata unified naming space is used for storing metadata information of data stored in the storage system.
Optionally, the method further comprises:
The first storage cluster is further used for establishing a first corresponding relation between data stored in a plurality of first storage nodes contained in the first storage cluster and the first storage nodes in the metadata unified name space;
The first storage cluster is configured to update the first correspondence to a second correspondence between the data migrated to the second storage cluster and the second storage node after migrating the data already stored in the plurality of first storage nodes to the plurality of second storage nodes of the second storage cluster.
Optionally, the first storage cluster is provided with a hot data storage space, and the hot data storage space is used for storing hot data through a plurality of first storage nodes, and dividing the data stored by the plurality of first storage nodes into cold data and hot data according to the data heat of the data stored by the plurality of first storage nodes;
the hot data storage space is further used for initiating a data migration task for the cold data, and migrating the cold data in the plurality of first storage nodes to a plurality of second storage nodes in the second storage cluster.
Optionally, the first storage cluster is configured to modify metadata information stored in the metadata unified namespace after the hot data storage space migrates the cold data to the plurality of second storage nodes of the second storage cluster, the metadata information being related to cold data migrated to the plurality of second storage nodes of the second storage cluster.
Optionally, a thermal data storage space is deployed in the first storage cluster, where the thermal data storage space is used to store the thermal data through a plurality of first storage nodes;
The first storage cluster is used for determining the storage position of the first target data from the metadata unified naming space when receiving a read request of the first target data;
The first storage cluster is further configured to read the first target data from the hot data storage space when it is determined that the first target data is stored in the hot data storage space;
The first storage cluster is further configured to read the first target data through a target protocol when it is determined that the first target data is stored in any one of the plurality of second storage nodes.
Optionally, the first storage cluster is configured to obtain modified second target data when receiving a modification command for the second target data sent by the service system;
The first storage cluster is used for determining the storage position of the second target data through the metadata unified naming space;
The first storage cluster is configured to store the modified second target data into a storage location of the second target data.
Optionally, the plurality of first storage nodes of the first storage cluster are used for executing data migration tasks of a plurality of cold data, the first storage cluster shares a plurality of data migration tasks, the first storage cluster is used for transmitting a plurality of cold data in a multi-thread concurrency mode, the plurality of cold data comprises first data and second data, the data size of the first data is larger than a preset data size, and the data size of the second data is not larger than the preset data size;
When a faulty first storage node exists in the first storage cluster, a plurality of data migration tasks on the faulty first storage node are distributed to a plurality of other first storage nodes in the first storage cluster in an average mode.
Optionally, the plurality of first storage nodes included in the first storage cluster are configured to split corresponding first data in the data migration task into a plurality of objects through a target protocol according to the data migration task executed by each first storage node, and divide each object into a plurality of fragments;
And the plurality of first storage nodes contained in the first storage cluster transmit the plurality of fragments in parallel in a multithreading concurrent mode.
Optionally, the plurality of first storage nodes included in the first storage cluster are used for splitting corresponding second data in the data migration task into a plurality of fragments through a target protocol according to the data migration task executed by each first storage node;
And the plurality of first storage nodes contained in the first storage cluster transmit the plurality of fragments in parallel in a multithreading concurrent mode.
Optionally, the multiple first storage nodes included in the first storage cluster transmit the multiple slices in parallel through a multithreading concurrency manner, including:
The first storage node is used for asynchronously reading the offset of each fragment obtained after the data division and the data content included by the fragment;
the first storage node is configured to divide the data migration task into a plurality of subtasks corresponding to a plurality of objects when the data includes the plurality of objects, and place the plurality of subtasks in a queue corresponding to the data in sequence, where the subtasks corresponding to the plurality of data are placed in the plurality of queues;
And the first storage node is used for transmitting the fragments corresponding to the subtasks in the queues to the second storage nodes in the second storage cluster in a concurrent mode of multithreading according to the offset corresponding to the fragments.
Optionally, the multiple first storage nodes included in the first storage cluster transmit the multiple fragments in parallel through a multithreading concurrent mode, and the method further includes:
The first storage node is configured to determine a total number of the plurality of fragments contained by the data;
The first storage node is configured to reduce, for each piece of data, a total number of pieces contained in the data by one after one piece of the data is transmitted in a process of concurrently transmitting the plurality of pieces;
the first storage node is configured to determine that the data transmission is completed when a count of a total number of the plurality of fragments contained in the data is 0;
And the first storage node is used for restarting transmission from the fragments with transmission failure when the fragments with transmission failure exist in the process of transmitting the fragments simultaneously.
The first storage cluster is configured to modify metadata information stored in the metadata unified namespace and related to cold data migrated to a plurality of second storage nodes in the second storage cluster after the hot data storage space migrates the cold data to the plurality of second storage nodes in the second storage cluster.
The second aspect of the embodiment of the present application provides a data storage method, applied to a storage system, where the storage system includes a first storage cluster and a second storage cluster, where a metadata unified namespace is deployed in the first storage cluster, the first storage cluster includes a plurality of first storage nodes, the second storage cluster includes a plurality of second storage nodes, an area where the first storage cluster is located is less than a preset distance from a service system, an area where the second storage cluster is located is not less than the preset distance from the service system, areas where the plurality of second storage nodes included in the second storage cluster are located are different from each other, the plurality of second storage nodes are respectively connected to the first storage cluster through a target protocol, and the metadata unified namespace is used for storing metadata information of data stored in the storage system, and the method includes:
Establishing a first corresponding relation between the data stored in the plurality of first storage nodes contained in the first storage cluster and the first storage nodes in the metadata unified name space;
After migrating the data already stored in the plurality of first storage nodes to a plurality of second storage nodes of the second storage cluster, updating the first correspondence to a second correspondence between the data migrated to the second storage cluster and the second storage nodes.
In a third aspect of the embodiment of the present application, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the data storage method according to the second aspect.
In a fourth aspect of embodiments of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the data storage method according to the second aspect.
In a fifth aspect of embodiments of the present application, there is provided a computer readable medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the data storage method according to the second aspect.
The application has the beneficial effects that:
The embodiment of the application provides a storage system, a data storage method, equipment, a product and a medium, wherein the storage system comprises: a first storage cluster and a second storage cluster; a metadata unified namespace is deployed in the first storage cluster; the first storage clusters comprise a plurality of first storage nodes, the second storage clusters comprise a plurality of second storage nodes, the area distance between the first storage clusters and the service system is smaller than a preset distance, the area distance between the second storage clusters and the service system is not smaller than the preset distance, the areas of the plurality of second storage nodes in the second storage clusters are different, and the plurality of second storage nodes are respectively connected into the first storage clusters through a target protocol; the metadata unified naming space is used for storing metadata information of data stored in the storage system.
The first storage cluster can be set to be smaller than the area with the preset distance from the service system, so that high-performance and low-delay data access is realized, and the requirement of the service system on real-time data is met. The second storage cluster is arranged in an area which is not smaller than the preset distance from the service system, and is accessed to the first storage cluster through the target protocol, so that large-capacity storage resources can be provided, the requirement of the service system on large-scale data storage is met, management and access of data of the storage system in different areas are realized through the unified metadata naming space and the target protocol, and the requirement of the service system on data storage is met.
Detailed Description
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Fig. 1 is a schematic diagram of a storage system according to an embodiment of the present application, and fig. 2 is a hierarchical schematic diagram of a storage system according to an embodiment of the present application, as shown in fig. 1, according to an embodiment of the present application, the storage system includes:
a first storage cluster and a second storage cluster;
A metadata unified namespace is deployed in the first storage cluster;
The first storage clusters comprise a plurality of first storage nodes, the second storage clusters comprise a plurality of second storage nodes, the area distance between the first storage clusters and the service system is smaller than a preset distance, the area distance between the second storage clusters and the service system is not smaller than the preset distance, the areas of the plurality of second storage nodes in the second storage clusters are different, and the plurality of second storage nodes are respectively connected into the first storage clusters through a target protocol;
The metadata unified naming space is used for storing metadata information of data stored in the storage system.
In one embodiment, as shown in FIG. 2, the storage system includes two layers, a unified namespace of storage metadata for storage clusters that are closer to the application layer, and multiple vendor stores (e.g., vendor 1 in Shanghai, vendor 2 in Beijing, vendor 3 in Chengdu, etc.) that access different regions via a target protocol.
Specifically, as shown in fig. 1, the storage system includes a first storage cluster and a second storage cluster, where the first storage cluster and the second storage cluster are both used to store data generated by the service system, and different data may be stored in different positions according to a lifecycle of the data.
The business system is responsible for processing business logic and business data and writing the data into the storage system, which in turn provides storage and access capabilities for the data and stores the data of the business system into the appropriate storage nodes.
The first storage cluster comprises a plurality of first storage nodes, in order to realize full utilization of storage space of the storage system, data can be stored in different storage nodes according to different data heat degrees, and specifically, the first storage nodes are connected with storage pools of different types according to service requirements and performance requirements of the stored data to store and manage the data, such as a full flash pool, a mixed flash pool and an HDD (hard disk drive) pool.
Preferably, for a full flash pool, frequently accessed data may be stored, such as: thermal data, cache data, real-time analysis data, transaction records and the like of the database; for a shuffle pool, data with a certain access frequency may be stored, such as: temperature data (backup of hot data) of a database, historical data, intermediate results of big data analysis and the like; for the HDD pool, data requiring mass storage, such as: archival data, backup data, cold data in large-scale data warehouse, long-term stored log data
Preferably, for different business systems, different storage pools may also be selected, such as:
for an e-commerce platform, sales data of popular merchandise and real-time transaction records of users are suitably stored in a full flash pool to provide high performance and low latency data access.
For a financial institution, historical transaction data and some intermediate analysis results are suitable for storage in a flash pool to achieve a balance of performance and capacity.
For a medical facility, the patient's electronic medical record and large-scale medical image data are suitably stored in a pool of HDDs to provide a large amount of storage resources.
The first storage clusters can be used for caching most frequently accessed data, and when a client of the service system needs to read or write the data, the first storage clusters can be directly accessed without accessing the remote storage through a network, so that the performance is improved.
The second storage cluster comprises a plurality of second storage nodes, the different second storage nodes are located in different areas and are all remote storage with the distance between the second storage nodes and the service system not smaller than the preset distance, the plurality of second storage nodes are accessed into the first storage cluster through a target protocol, such AS AS13000, third party storage, cloud storage, blue light, a tape library and the like, wherein the target protocol is an Object storage protocol, namely Simple Storage Service (S3 for short), which is a public cloud storage service, the S3 provides a RESTful API to realize interaction with the service in a programming mode, the data storage structure of the S3 is divided into two layers, the first layer is a Bucket, and the second layer is an Object.
In the first storage cluster, a metadata unified namespace is deployed for storing metadata information of data stored in a plurality of first storage nodes in the first storage cluster and a plurality of second storage nodes in the second storage cluster in the storage system, thereby centrally managing and accessing metadata of data objects. Metadata refers to information describing attributes of a data object, such as creation time, modification time, owner, rights, size, type, etc. Unified namespaces mean that the metadata share a global namespace so that the metadata information can be accessed and referenced by the entire storage system or any portion of the storage cluster. Through the metadata unified namespaces, whether the data is migrated to a third party cloud or storage node of other storage systems, unified views can be provided for management through metadata management.
Through the embodiment, the first storage cluster is arranged at the near end of the service system (the distance from the first storage cluster to the service system is smaller than the preset distance), so that high-performance and low-delay data access can be realized, and the requirement of the service system on real-time data is met. The second storage cluster is positioned at the far end of the service system (the distance from the second storage cluster to the service system is not smaller than the preset distance), and the second storage cluster is accessed into the first storage cluster through the target protocol, so that large-capacity storage resources can be provided, and the requirement of the service system on large-scale data storage is met. And the management and access of the data of the storage system in different areas are realized through the unified metadata naming space and the target protocol, so that the requirement of the service system on the data storage is met.
Optionally, the method further comprises:
The first storage cluster is further used for establishing a first corresponding relation between data stored in a plurality of first storage nodes contained in the first storage cluster and the first storage nodes in the metadata unified name space;
The first storage cluster is configured to update the first correspondence to a second correspondence between the data migrated to the second storage cluster and the second storage node after migrating the data already stored in the plurality of first storage nodes to the plurality of second storage nodes of the second storage cluster.
Specifically, in one embodiment, the generated data of the service system may be directly stored in the second storage cluster, or may be migrated to the second storage cluster through the first storage cluster, and after the data is stored in the first storage cluster, a corresponding first correspondence between the data and the first storage node where the data is located is newly created in the metadata unified namespace, so as to record the storage location mapping of the data in each first storage node in the first storage cluster.
Further, when the data is migrated from the first storage cluster to the plurality of storage nodes of the second storage cluster, corresponding modifications need to be made in the metadata unified namespace to update metadata information associated with the migrated data.
Further, the mapping relation between the data originally recorded in the first corresponding relation in the metadata unified naming space and the first storage node is modified into a second corresponding relation, and the second corresponding relation is mapped to the actual storage position of the migrated data in the second storage cluster.
In this way, the unified naming space of the metadata can track and record the latest storage position of the data in real time, and metadata support is provided for the application layer to correctly access the data.
Optionally, the first storage cluster is provided with a hot data storage space, and the hot data storage space is used for storing hot data through a plurality of first storage nodes, and dividing the data stored by the plurality of first storage nodes into cold data and hot data according to the data heat of the data stored by the plurality of first storage nodes;
the hot data storage space is further used for initiating a data migration task for the cold data, and migrating the cold data in the plurality of first storage nodes to a plurality of second storage nodes in the second storage cluster.
Specifically, in one embodiment, a plurality of first storage nodes are used as hot data storage spaces in a first storage cluster, hot data is stored through the hot data storage spaces, the hot data is frequently accessed and updated data, the data stored in the plurality of first storage nodes are divided into cold data and hot data according to the data heat of the data stored in the plurality of first storage nodes (when the data heat is greater than a heat threshold value, the data is determined to be hot data and vice versa), when the data is determined to be cold data (for example, the data heat of the hot data is reduced and becomes cold data), a data migration task for the cold data is correspondingly initiated, the cold data in the plurality of first storage nodes is migrated to a plurality of second storage nodes in a second storage cluster, and in the dynamic storage process of the data, the data stored in the first storage cluster is the hot data and the data stored in the second storage cluster is the cold data.
According to the embodiment, the hot data can be stored in the hot data storage space and stored through the plurality of first storage nodes, when the hot data is required to be accessed, the hot data can be quickly obtained from the hot data storage space, so that the access speed and the availability of the hot data are improved, cold data are stored through the second storage clusters, the utilization of storage resources can be optimized, the capacity and the performance of the whole storage system are improved, the storage process of the data is dynamically changed along with the data heat of the data, and when certain hot data is changed into cold data, the data can be migrated from the hot data storage space to the second storage nodes of the second storage clusters, so that the flexibility of the storage system is realized.
Optionally, the first storage cluster is configured to modify metadata information stored in the metadata unified namespace after the hot data storage space migrates the cold data to the plurality of second storage nodes of the second storage cluster, the metadata information being related to cold data migrated to the plurality of second storage nodes of the second storage cluster.
Specifically, after the data is migrated to the second storage cluster far away from the service system, the metadata information corresponding to the migrated data is centrally managed and tracked through the unified metadata naming space. The purpose of the metadata information is to reflect that the actual storage location of the migrated data has changed.
The metadata information of the data includes: name, path, size, creation/modification/access time, owner/rights information, checksum, storage node where the data is located, mapping information of the data (e.g. block storage), storage media type (SSD/HDD, etc.), identification of hot/cold data of the data, last access time, migration history, storage cluster topology, capacity utilization, failure/backup information, etc., after migrating the data, the metadata information to be modified at least includes: the storage node where the data is located, mapping information (such as block storage) of the data, storage medium type (SSD/HDD, etc.), hot data/cold data identification of the data, and migration history.
Specifically, the embodiment of the application manages metadata through the target protocol (the object storage protocol S3), and the second storage cluster is managed into the first storage cluster. First, the second storage cluster creates a bucket and generates the public and private keys of the object storage protocol as authentication credentials. Then, a migration relationship is established by commanding the addition of the public key and the private key to the first storage cluster. Thus, the first storage cluster may communicate with the second storage cluster via the object storage protocol. Next, a migration relationship is established between the local directory of the first storage cluster and the local directory of the second storage cluster by a command. In this way, the first storage cluster is able to determine which data to migrate to the second storage cluster. And when the data is determined to be migrated according to the rule of the data heat, the first storage cluster initiates a data migration task. The first storage cluster automatically transfers data from the first storage cluster to the second storage cluster via an object storage protocol. This means that the actual storage location of the data becomes the second storage cluster, but the metadata information of the data is still kept in the first storage cluster. After the data migration is completed, the first storage cluster modifies metadata information for the data, marking that the data has been migrated to the second storage cluster by the migration relationship. When a user accesses the data of the first storage cluster, according to the metadata information, if the data is still in the first storage cluster, directly accessing the first storage cluster; if the data has migrated to the second storage cluster, the data is read from the second storage cluster via the object storage protocol. If the user modifies the content of the data, the modified content is also sent to the second storage cluster via the object storage protocol. The method ensures that the read-write operation of the user on the data can still be managed through the metadata of the first storage cluster, and provides a unified view for the user.
Optionally, the degree of the migrated second storage node may be determined according to the degree of cold or hot of the data, where the lower the data heat of the cold data is, the farther the stored second storage node is, the higher the data heat of the cold data (but not higher than the heat critical value of the cold data and the hot data) is, and the closer the stored second storage node is.
According to the embodiment, the hot data and the cold data in the storage system can be separately stored, and only the hot data can be stored in the first storage cluster through dynamic storage, so that the hot data access performance is improved, the cold data is migrated to the second storage cluster far away from the service system, the cold data storage cost can be reduced, metadata information is modified after migration, and efficient utilization of resources is realized through metadata centralized management.
As one example, when data is changed from hot data in the hot data storage space of a first storage cluster to cold data, the process of migrating the data into a second storage cluster includes the steps of:
1. And (3) thermal data identification:
the storage system continuously monitors the access frequency of the data and judges whether the data is changed from hot data to cold data according to a certain strategy and a threshold value. Common ways of identification are Least Recently Used (LRU), most recently used (LFU) and other algorithms.
The storage system generally determines whether the data is changed from hot data to cold data according to the data access frequency and a certain policy and threshold value, including the following determination modes:
① Recent access time (LAST ACCESS TIME)
When the data has not been accessed within a first target period of time (e.g., 30 days), it may be considered cold data.
② Access Frequency (Access Frequency)
The number of accesses to each data in the second target period (for example, 90 days) is counted periodically, and when the number of accesses to the data is lower than a certain threshold (for example, 10 times), the data is regarded as cold data.
③ Modification time (Modification Time)
When the data has not been modified within the third target period, it is considered cold data.
④ Size (Size)
For data having a data size larger than a preset size (several GB or more), even if the access frequency is high, it is regarded as cold data.
⑤ File Type (File Type)
For data of some file types, such as journals, temporary files, etc., the storage system may treat it directly as cold data.
2. And (3) data migration:
for files identified as cold data, the storage system triggers a data migration task to transfer those data from the near-end storage (first storage cluster) to the far-end storage (second storage cluster).
3. Metadata update:
After the data migration is completed, metadata information related to the data in the metadata unified namespace, including storage locations, access paths and the like after the data migration, needs to be updated to ensure that the application program can correctly access the migrated data.
4. Proximal data cleaning:
To free up storage space on the near-end storage (the first storage cluster) for new hot data use, the storage system pre-configures a scrubbing policy to scrub or delete cold data on the near-end storage that has migrated to the far-end storage.
Optionally, a thermal data storage space is deployed in the first storage cluster, where the thermal data storage space is used to store the thermal data through a plurality of first storage nodes;
The first storage cluster is used for determining the storage position of the first target data from the metadata unified naming space when receiving a read request of the first target data;
The first storage cluster is further configured to read the first target data from the hot data storage space when it is determined that the first target data is stored in the hot data storage space;
The first storage cluster is further configured to read the first target data through a target protocol when it is determined that the first target data is stored in any one of the plurality of second storage nodes.
Specifically, in one embodiment, a hot data storage space is disposed in the first storage cluster, and is used for storing frequently accessed hot data, where the hot data is stored in the hot data storage space formed by a plurality of first storage nodes. When a read request for a certain data (first target data) is received, the first storage cluster firstly queries metadata information corresponding to the first target data in the metadata unified naming space, and obtains a storage position of the first target data from the metadata information. When it is determined that the first target data is stored in one or more first storage nodes of the hot data storage space, the first target data is directly read from one or more first storage nodes of the hot data storage space, and when the metadata information indicates that the target data has migrated to one of the second storage nodes of the second storage cluster (the cluster storing cold data), the first storage cluster remotely reads the first target data from the corresponding second storage node of the second storage cluster through a specific target protocol (e.g., S3 protocol, shift protocol, NFS protocol, CIFS protocol, HADOOP protocol, POSIX protocol).
In the above embodiment, by querying the storage location of the first target data in the metadata unified namespace, and then accessing the data in different storage clusters according to the hot/cold state of the first target data, efficient data reading service can be provided for an application layer (such as a service system) more transparently.
Optionally, the first storage cluster is configured to obtain modified second target data when receiving a modification command for the second target data sent by the service system;
The first storage cluster is used for determining the storage position of the second target data through the metadata unified naming space;
The first storage cluster is configured to store the modified second target data into a storage location of the second target data.
Specifically, in one embodiment, the stored data may be modified, first, when the service system issues a modification command (including data that needs to be operated, modified content, etc.) to the second target data, the first storage cluster receives the modification command and acquires the modified data. And finally, the first storage cluster persists the modified second target data to the determined storage position of the two target data to finish the data modification of the two target data.
For example, suppose that order data for a certain e-commerce system is stored in this distributed storage system. After the user orders, the business system sends out a modification command and writes new order data into the storage system. After receiving the modification command, the storage system acquires the storage position of the order data by inquiring the metadata, and then persists the new order data to the position, thereby ensuring the consistency and reliability of the data.
By the method, the access path of the data is simple, the application layer only needs to pay attention to metadata information, and the specific position of the data in the storage system is not required to be paid attention to. Through the unified metadata naming space, data can be seamlessly migrated to storage nodes of different storage clusters, and the corresponding application layer is transparent.
Optionally, the plurality of first storage nodes of the first storage cluster are used for executing data migration tasks of a plurality of cold data, the first storage cluster shares a plurality of data migration tasks, the first storage cluster is used for transmitting a plurality of cold data in a multi-thread concurrency mode, the plurality of cold data comprises first data and second data, the data size of the first data is larger than a preset data size, and the data size of the second data is not larger than the preset data size;
When a faulty first storage node exists in the first storage cluster, a plurality of data migration tasks on the faulty first storage node are distributed to a plurality of other first storage nodes in the first storage cluster in an average mode.
Specifically, in one embodiment, a plurality of first storage nodes included in a first storage cluster are responsible for performing data migration tasks for a plurality of cold data. In view of the failover requirements, the plurality of data migration tasks of each first storage node are shared throughout the first storage cluster. That is, these data migration tasks are not bound to a particular first storage node, but rather are distributed and executed within the scope of the first storage cluster. The data migration task is distributed, managed and coordinated by a centralized scheduling component preset in the first storage cluster, and task distribution can be completed cooperatively among nodes based on a distributed consistency protocol.
Moreover, it should be noted that, the allocation of the data migration task is automatically completed by the internal mechanism of the first storage cluster, rather than relying on an external component, so that the efficiency of task scheduling and the autonomy of the storage system can be improved.
Further, the first storage cluster uses a multi-threaded concurrent manner to transfer the plurality of cold data. This means that the data migration tasks of a plurality of cold data can be performed simultaneously, improving the transmission efficiency.
The cold data are divided into first data and second data according to different data sizes, wherein the first data is larger than the preset data size, and the second data is not larger than the preset data size. I.e. the first data is larger data and the second data is smaller data.
When a first storage node in the first storage cluster fails, data migration tasks on the failed first storage node are distributed to other normal first storage nodes in the first storage cluster. Thus, the continuous execution of the data migration task can be ensured, and the usability and fault tolerance of the system are improved.
By the method, resource sharing and task allocation optimization of the first storage cluster are utilized, the efficiency is improved by using multi-thread concurrent transmission, different data with different sizes are treated differently, and meanwhile, the method has the fault transfer capability, so that the overall performance and reliability of the storage system in the cold data migration process are improved.
Optionally, the plurality of first storage nodes included in the first storage cluster are configured to split corresponding first data in the data migration task into a plurality of objects through a target protocol according to the data migration task executed by each first storage node, and divide each object into a plurality of fragments;
And the plurality of first storage nodes contained in the first storage cluster transmit the plurality of fragments in parallel in a multithreading concurrent mode.
Specifically, in one embodiment, according to a data migration task executed by each first storage node, first data with larger data size to be migrated is split into a plurality of objects according to a target protocol, each object obtained through splitting is further divided into a plurality of smaller fragments, and the split fragments are simultaneously and parallelly transmitted in a multithreading concurrency mode of the first storage node, so that fine-granularity processing is achieved, and parallelism of a data transmission process is further improved.
By means of the combination of splitting objects, dividing fragments and multithreading, resources such as a CPU, a memory and a network of the first storage node can be utilized to the maximum extent, and data migration efficiency and throughput of larger data (first data) are improved remarkably.
Optionally, the plurality of first storage nodes included in the first storage cluster are used for splitting corresponding second data in the data migration task into a plurality of fragments through a target protocol according to the data migration task executed by each first storage node;
And the plurality of first storage nodes contained in the first storage cluster transmit the plurality of fragments in parallel in a multithreading concurrent mode.
Specifically, in one embodiment, since the second data is data smaller than the preset data size, each second data may be directly used as an object, and each first storage node uses the target protocol to split the second data into multiple smaller fragments according to the second data (i.e. smaller data) in the data migration task performed by each first storage node. The purpose of data slicing is to improve transmission efficiency.
Further, the split fragments are simultaneously and parallelly transmitted by the first storage node in a multithreading concurrent mode. The computing capacity and network resources of each first storage node can be utilized to the maximum extent through multi-thread concurrent transmission, and the condition that the resource utilization rate is low during single-thread transmission is avoided. Further, data splitting and multi-thread concurrent transmission are combined to significantly improve migration efficiency and throughput of the second data.
In general, the segmentation and parallel transmission optimization strategy aims at the characteristics of smaller data, and by fully utilizing calculation and network resources, the data migration efficiency is remarkably improved, and meanwhile, higher flexibility and expandability are also embodied. This is an important performance optimization approach for storage systems that require frequent migration of large amounts of data.
Optionally, the multiple first storage nodes included in the first storage cluster transmit the multiple slices in parallel through a multithreading concurrency manner, including:
The first storage node is used for asynchronously reading the offset of each fragment obtained after the data division and the data content included by the fragment;
the first storage node is configured to divide the data migration task into a plurality of subtasks corresponding to a plurality of objects when the data includes the plurality of objects, and place the plurality of subtasks in a queue corresponding to the data in sequence, where the subtasks corresponding to the plurality of data are placed in the plurality of queues;
And the first storage node is used for transmitting the fragments corresponding to the subtasks in the queues to the second storage nodes in the second storage cluster in a concurrent mode of multithreading according to the offset corresponding to the fragments.
Specifically, in one embodiment, first, according to respective data migration tasks, a plurality of first storage nodes divide data to be migrated indicated by the data migration tasks into a plurality of slices, where each slice corresponds to an offset and data content. A plurality of first storage nodes in the first storage cluster are balanced to a plurality of physical disks through a striping technology in an asynchronous reading mode, so that the data can be effectively stored in a scattered manner. Each tile may acquire its offset in the data and its data content via an object storage protocol. When the data to be migrated contains a plurality of objects, the whole data migration task is divided into a plurality of subtasks the same as the number of objects. For one data, triggering a callback function every time asynchronous reading of one object is completed, sequentially placing subtasks of a plurality of objects corresponding to the data into a queue related to the data, when one large data (first data) is divided into a plurality of objects, respectively placing the subtasks of the plurality of objects corresponding to the data into different queues, each subtask corresponds to a plurality of fragments, and the first storage node concurrently acquires the data content of the fragments corresponding to the subtasks from the queues by creating a plurality of threads according to the offset corresponding to the fragments of one subtask, and concurrently transmits the acquired data content of the fragments to a corresponding second storage node in a second storage cluster by using the threads. When a smaller data (second data) is taken as an object, the data is divided into a plurality of fragments, the data corresponds to a data migration task, and the data transmission of the plurality of fragments is realized through a queue.
Alternatively, in order to avoid occupation of excessive resources, the transmission bandwidth may be controlled by controlling the read-write speed ratio. Thus, the speed of data migration can be balanced, and overload on the system is avoided.
By the mode, the network and IO resources of the storage nodes can be utilized in parallel in a parallel and multithreading mode, and the transmission efficiency and throughput of the data migration task are obviously improved. Meanwhile, through the subtask queue, big data migration tasks containing a plurality of objects are processed orderly, and processing reliability is improved.
Optionally, the multiple first storage nodes included in the first storage cluster transmit the multiple fragments in parallel through a multithreading concurrent mode, and the method further includes:
The first storage node is configured to determine a total number of the plurality of fragments contained by the data;
The first storage node is configured to reduce, for each piece of data, a total number of pieces contained in the data by one after one piece of the data is transmitted in a process of concurrently transmitting the plurality of pieces;
the first storage node is configured to determine that the data transmission is completed when a count of a total number of the plurality of fragments contained in the data is 0;
And the first storage node is used for restarting transmission from the fragments with transmission failure when the fragments with transmission failure exist in the process of transmitting the fragments simultaneously.
Specifically, in one embodiment, during the parallel transmission of the slices, the integrity and correctness of data transmission needs to be guaranteed by multiple first storage nodes in the first storage cluster, so that the method can be implemented in an atomic counting mode, and specifically, it needs to be determined, by the first storage nodes, how many slices the data to be transmitted are divided into in total, that is, the total number of the multiple slices contained in the data. Further, in the process of concurrently transmitting each slice in multithreading, each time one slice transmission is completed, the first storage node reduces the count of the total number of the plurality of slices contained in the data by 1 until the count of the total number of the plurality of slices contained in the data is 0, it is determined that all slices of the data are transmitted, and when partial slices are transmitted in the process of concurrently transmitting, the first storage node determines the offset of the slices with failed transmission and initiates data transmission again from the slices with failed transmission.
By the method, the total number of the fragments is firstly confirmed and the fragments are counted, so that each fragment is ensured to be successfully transmitted once, meanwhile, retransmission can be realized when the failure of the fragment transmission is found, and data loss is avoided. And each fragment is transmitted in parallel by adopting a multithreading concurrence mode, so that the transmission efficiency is improved.
Optionally, after all data transmission of the data is completed, the method further includes that on a second storage node storing the fragments of the data transmission in a second storage cluster, reorganizing the fragments of which the data transmission is completed according to the sequence label of each fragment to obtain original data before transmission.
Fig. 3 is a flow chart of a data storage method provided by an embodiment of the present application, as shown in fig. 3, where the data storage method is applied to a storage system, the storage system includes a first storage cluster and a second storage cluster, a metadata unified naming space is deployed in the first storage cluster, the first storage cluster includes a plurality of first storage nodes, the second storage cluster includes a plurality of second storage nodes, an area where the first storage cluster is located is less than a preset distance from a service system, an area where the second storage cluster is located is not less than the preset distance from the service system, areas where the plurality of second storage nodes included in the second storage cluster are located are different from each other, the plurality of second storage nodes are respectively connected to the first storage cluster through a target protocol, and the metadata unified naming space is used for storing metadata information of data stored in the storage system, and the method includes:
in step S101, a first correspondence relationship between data already stored in a plurality of first storage nodes included in the first storage cluster and the first storage nodes is established in the metadata unified namespace;
In step S12, after migrating the data already stored in the plurality of first storage nodes to the plurality of second storage nodes of the second storage cluster, the first correspondence is updated to be a second correspondence between the data migrated to the second storage cluster and the second storage nodes.
Optionally, the first storage cluster is deployed with a hot data storage space, the method further comprising:
storing hot data through a plurality of first storage nodes, and dividing the data stored in the plurality of first storage nodes into cold data and hot data according to the data heat of the data stored in the plurality of first storage nodes;
and initiating a data migration task for the cold data, and migrating the cold data in the plurality of first storage nodes to a plurality of second storage nodes in the second storage cluster.
Optionally, after the hot data storage space migrates the cold data to the plurality of second storage nodes of the second storage cluster, modifying metadata information stored in the metadata unified namespace that is related to cold data migrated to the plurality of second storage nodes of the second storage cluster.
Optionally, a hot data storage space is deployed in the first storage cluster, and the method further includes:
Storing the thermal data by a plurality of first storage nodes;
When a read request of first target data is received, determining a storage position of the first target data from the metadata unified naming space;
reading the first target data from the thermal data storage space when the first target data is determined to be stored in the thermal data storage space;
And reading the first target data through the target protocol when the first target data is determined to be stored in any one of the plurality of second storage nodes.
Optionally, the method further comprises:
when a modification command aiming at second target data sent by the service system is received, modified second target data are obtained;
determining a storage position of the second target data through the metadata unified naming space;
and storing the modified second target data in a storage position of the second target data.
Optionally, the method further comprises:
Splitting corresponding first data in the data migration task into a plurality of objects through a target protocol, and dividing each object into a plurality of fragments;
Splitting corresponding second data in the data migration task into a plurality of fragments through a target protocol;
Transmitting the plurality of fragments in parallel by a multithreading concurrent manner, including:
Asynchronously reading the offset of each fragment obtained after the data division and the data content included by the fragment;
When the data comprises a plurality of objects, dividing the data migration task into a plurality of subtasks corresponding to the plurality of objects, and sequentially placing the plurality of subtasks into a queue corresponding to the data, wherein the subtasks corresponding to the plurality of data are placed into a plurality of queues;
And transmitting the fragments corresponding to the subtasks in the queues to a plurality of second storage nodes in the second storage cluster in a concurrent manner through multithreading according to the offset of the fragments.
Fig. 4 is a schematic diagram of a frame of a data storage device according to an embodiment of the present application, where as shown in fig. 4, an embodiment of the present application provides a data storage device, which is applied to a storage system, where the storage system includes a first storage cluster and a second storage cluster, where a metadata unified namespace is deployed in the first storage cluster, where the first storage cluster includes a plurality of first storage nodes, where the second storage cluster includes a plurality of second storage nodes, where a region of the first storage cluster is located at a distance from a service system that is smaller than a preset distance, where a region of the second storage cluster is located at a distance from the service system that is not smaller than the preset distance, where regions of the plurality of second storage nodes included in the second storage cluster are different from each other, where the plurality of second storage nodes access the first storage cluster through a target protocol, and where the metadata unified namespace is used to store metadata information of data stored in the storage system, where the device includes:
A building module 11, configured to build, in the metadata unified namespace, a first correspondence relationship between data already stored in a plurality of first storage nodes included in the first storage cluster and a first storage node;
and the migration module 12 is configured to update the first correspondence to a second correspondence between the data migrated to the second storage cluster and the second storage node after migrating the data already stored in the plurality of first storage nodes to the plurality of second storage nodes of the second storage cluster.
Optionally, the first storage cluster is deployed with a hot data storage space, and the apparatus further comprises:
The data dividing module is used for storing hot data through the plurality of first storage nodes and dividing the data stored by the plurality of first storage nodes into cold data and hot data according to the data heat of the data stored by the plurality of first storage nodes;
And the data migration task initiating module is used for initiating a data migration task for the cold data and migrating the cold data in the plurality of first storage nodes to a plurality of second storage nodes in the second storage cluster.
Optionally, the apparatus further comprises: and the modifying module is used for modifying metadata information stored in the metadata unified naming space and related to the cold data migrated to the plurality of second storage nodes in the second storage cluster after the hot data storage space migrates the cold data to the plurality of second storage nodes in the second storage cluster.
Optionally, a hot data storage space is deployed in the first storage cluster, and the apparatus further includes:
A thermal data storage module for storing the thermal data through a plurality of first storage nodes;
The storage position determining module is used for determining the storage position of the first target data from the metadata unified naming space when a read request of the first target data is received;
a first reading module for reading the first target data from the thermal data storage space when it is determined that the first target data is stored in the thermal data storage space;
And the second reading module is used for reading the first target data through the target protocol when the first target data is determined to be stored in any one of the plurality of second storage nodes.
Optionally, the apparatus further comprises:
The acquisition module is used for acquiring the modified second target data when receiving a modification command aiming at the second target data sent by the service system;
a storage position determining module of second target data, configured to determine a storage position of the second target data through the metadata unified namespace;
And the modified storage module is used for storing the modified second target data into a storage position of the second target data.
Optionally, the apparatus further comprises:
the first splitting module is used for splitting corresponding first data in the data migration task into a plurality of objects through a target protocol and dividing each object into a plurality of fragments;
the second splitting module is used for splitting the corresponding second data in the data migration task into a plurality of fragments through a target protocol;
The parallel transmission module is used for transmitting the plurality of fragments in parallel in a multithreading concurrency mode, and comprises the following steps:
the asynchronous reading module is used for asynchronously reading the offset of each fragment obtained after the data division and the data content included by the fragment;
The enqueuing module is used for dividing the data migration task into a plurality of subtasks corresponding to a plurality of objects when the data comprises the plurality of objects, and sequentially placing the plurality of subtasks into a queue corresponding to the data, wherein the subtasks corresponding to the plurality of data are placed into the plurality of queues;
And the concurrent transmission unit is used for transmitting the fragments corresponding to the subtasks in the queues to a plurality of second storage nodes in the second storage cluster in a concurrent manner in a multithreading manner according to the offset of the fragments.
Based on the same inventive concept, another embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the data storage method according to any one of the above embodiments.
Based on the same inventive concept, a further embodiment of the present application also provides a computer program product comprising a computer program for executing the data storage method according to any of the above embodiments by a processor.
Based on the same inventive concept, another embodiment of the present application further provides a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the data storage method according to any of the above embodiments.
Since it is substantially similar to the system embodiment, the description is relatively simple, and reference is made to the section of the system embodiment for relevant points.