技术领域Technical Field
本申请实施例涉及通信领域，尤其涉及一种数据存储的方法、设备和计算机可读存储介质。The embodiments of the present application relate to the field of communications, and more particularly to a data storage method, device, and computer-readable storage medium.
背景技术Background Art
Spark作为一种大规模数据处理的通用计算引擎，其作业的执行可以分为映射(map)阶段和规约(reduce)阶段。Map阶段执行完成后，会产生洗牌(shuffle)数据作为map阶段和reduce阶段的中间数据。原生的spark计算引擎的shuffle数据在map阶段，由映射任务(maptask)生成并写入本地文件中，在reduce阶段，由规约任务(reducetask)从maptask所在的节点提取shuffle数据并进行处理。当spark中进行大数据量的数据处理时，由于shuffle数据存储于map所在节点的本地文件，因此shuffle数据对计算资源的占用就会成为性能瓶颈，并且大幅浪费节点的计算资源。Spark is a general computing engine for large-scale data processing. The execution of its jobs can be divided into the map phase and the reduce phase. After the map phase is completed, shuffle data will be generated as the intermediate data for the map phase and the reduce phase. In the map phase, the shuffle data of the native spark computing engine is generated by the map task and written to the local file. In the reduce phase, the reduce task extracts the shuffle data from the node where the maptask is located and processes it. When processing large amounts of data in spark, since the shuffle data is stored in the local file of the node where the map is located, the occupation of computing resources by the shuffle data will become a performance bottleneck and greatly waste the computing resources of the node.
为避免shuffle数据对map所在节点计算资源的占用，技术人员提出利用远程数据洗牌服务(remoteshuffleservice，RSS)，将maptask生成的shuffle数据存储于专门的计算能力较弱但是具有强大存储能力的集群中。Maptask和reducetask通过网络读写shuffle数据，实现spark计算引擎的存算分离。To avoid shuffle data occupying the computing resources of the map node, the technicians proposed to use the remote shuffle service (RSS) to store the shuffle data generated by the maptask in a dedicated cluster with weaker computing power but strong storage capacity. Maptask and reducetask read and write shuffle data through the network, realizing the storage and computing separation of the spark computing engine.
在进行spark计算引擎存算分离的过程中，为充分利用不同设备的存储资源，避免存储资源利用不均衡的情况，通常会采用负载均衡的方式将应用生成的shuffle数据进行分区后，平均地存储于存储系统的所有服务器中。因此，当存储系统中的任一服务器发生故障，都会发生进行存算分离的应用大面积被迫停用的情况。In the process of storage and computing separation of the spark computing engine, in order to make full use of the storage resources of different devices and avoid uneven storage resource utilization, load balancing is usually used to partition the shuffle data generated by the application and evenly store it in all servers of the storage system. Therefore, when any server in the storage system fails, a large area of applications that are separated from storage and computing will be forced to stop working.
发明内容Summary of the invention
本申请实施例提供了一种数据存储的方法、设备和计算机可读存储介质。用于解决RSS技术中，远端用于存储的一个服务器故障会引起多个应用无法正常运行的情况。The embodiments of the present application provide a data storage method, device and computer-readable storage medium for solving the problem in RSS technology that a failure of a remote storage server may cause multiple applications to fail to operate normally.
本申请实施例第一方面提供了一种数据存储的方法，包括：获取第一应用创建后第一节点发送的创建存储空间请求，所述第一节点为与存储系统以可通信方式连接的服务器；生成所述第一应用与第一存储空间之间的对应关系，所述第一存储空间包括至少N个服务器的存储资源，所述存储系统中包括至少N+1个服务器，N为大于1的正整数；获取所述第一节点发送的第一数据，所述第一数据为所述第一应用运行时生成的中间数据；根据所述第一应用与所述第一存储空间之间的对应关系将所述第一数据写入所述第一存储空间。A first aspect of an embodiment of the present application provides a method for data storage, including: obtaining a storage space creation request sent by a first node after a first application is created, the first node being a server connected to a storage system in a communicative manner; generating a correspondence between the first application and a first storage space, the first storage space including storage resources of at least N servers, the storage system including at least N+1 servers, and N being a positive integer greater than 1; obtaining first data sent by the first node, the first data being intermediate data generated when the first application is running; and writing the first data into the first storage space according to the correspondence between the first application and the first storage space.
本申请实施例中，获取第一应用创建后发送的创建存储空间请求，生成第一应用与第一存储空间之间的对应关系，并根据第一应用与第一存储空间之间的对应关系将获取到的第一数据写入第一存储空间，其中，第一数据为第一应用生成的中间数据，第一存储空间包括N个服务器的存储资源，N个服务器包含于存储系统，且该存储系统包含至少N+1个服务器的存储资源。通过为应用划分存储空间，使得中间数据仅受存储系统中N个服务器的影响，缩小了第一应用的中间数据会受到影响的范围。In an embodiment of the present application, a storage space creation request sent after the first application is created is obtained, a correspondence between the first application and the first storage space is generated, and the first data obtained is written into the first storage space according to the correspondence between the first application and the first storage space, wherein the first data is intermediate data generated by the first application, and the first storage space includes storage resources of N servers, the N servers are included in a storage system, and the storage system includes storage resources of at least N+1 servers. By dividing the storage space for the application, the intermediate data is only affected by the N servers in the storage system, which reduces the scope of the intermediate data of the first application that will be affected.
在第一方面一种可能的实施方式中，该方法还包括：获取所述第一应用结束运行时，所述第一节点发送的第一应用结束运行信号；删除所述第一应用与所述第一存储空间之间的对应关系和所述第一存储空间中存储的所述第一数据。In a possible implementation of the first aspect, the method further includes: obtaining a first application termination signal sent by the first node when the first application terminates execution; and deleting the correspondence between the first application and the first storage space and the first data stored in the first storage space.
本申请实施例中，在第一应用结束运行时，获取第一节点发送的第一应用结束运行信号，并删除第一应用与第一存储空间之间的对应关系和第一存储空间中存储的第一数据，在第一应用结束运行后，及时释放为第一应用的中间数据分配的存储资源。提升了存储系统中，存储资源的利用效率。In the embodiment of the present application, when the first application ends running, a signal indicating that the first application has ended running is obtained from the first node, and the corresponding relationship between the first application and the first storage space and the first data stored in the first storage space are deleted. After the first application ends running, the storage resources allocated for the intermediate data of the first application are released in time, thereby improving the utilization efficiency of storage resources in the storage system.
在第一方面一种可能的实施方式中，所述生成所述第一应用与第一存储空间之间的对应关系前，所述方法还包括：获取所述存储系统的可分配存储资源清单，所述可分配存储资源清单包括所述存储系统中可分配的存储资源；选取第一服务器的存储资源作为所述第一存储空间，所述第一服务器包含于所述存储系统，所述第一服务器的存储资源为所述可分配存储资源清单中的存储资源。In a possible implementation of the first aspect, before generating the correspondence between the first application and the first storage space, the method also includes: obtaining a list of allocatable storage resources of the storage system, the list of allocatable storage resources including allocatable storage resources in the storage system; selecting the storage resources of a first server as the first storage space, the first server being included in the storage system, and the storage resources of the first server being the storage resources in the list of allocatable storage resources.
本申请实施例中，通过获取存储系统的可分配存储资源清单，并根据可分配资源清单选取第一服务器的存储资源作为第一存储空间，可分配存储资源清单中包含该存储系统中所有可分配的存储资源，第一服务器包含于存储系统，第一服务器的存储资源为可分配存储资源清单中的存储资源。通过获取存储系统的可分配存储资源清单，根据清单中所示的存储资源分布情况，按照一定规律对存储资源进行分配，或随机为第一应用生成的中间数据分配存储资源，可以为存储资源的合理利用提供更多的可能性。In the embodiment of the present application, by obtaining a list of allocatable storage resources of the storage system, and selecting the storage resources of the first server as the first storage space according to the list of allocatable resources, the list of allocatable storage resources includes all the allocatable storage resources in the storage system, the first server is included in the storage system, and the storage resources of the first server are the storage resources in the list of allocatable storage resources. By obtaining the list of allocatable storage resources of the storage system, according to the distribution of storage resources shown in the list, the storage resources are allocated according to a certain rule, or storage resources are randomly allocated to the intermediate data generated by the first application, more possibilities can be provided for the reasonable use of storage resources.
在第一方面一种可能的实施方式中，所述第一服务器为所述可分配存储资源清单中可分配存储资源最多的服务器。本申请实施例中，第一服务器得可分配存储资源清单中可分配存储资源最多的服务器，通过将可分配存储资源清单中，存储资源最多的服务器分配给第一存储空间，以便使用尽量少的服务器为第一应用的中间数据提供存储服务，降低了第一应用的中间数据占用存储设备的数量，避免当一个服务器发生故障时，同时影响多个应用的正常运行的情况发生。In a possible implementation of the first aspect, the first server is the server with the most allocatable storage resources in the list of allocatable storage resources. In the embodiment of the present application, the first server is the server with the most allocatable storage resources in the list of allocatable storage resources, and the server with the most storage resources in the list of allocatable storage resources is allocated to the first storage space, so as to use as few servers as possible to provide storage services for the intermediate data of the first application, thereby reducing the number of storage devices occupied by the intermediate data of the first application, and avoiding the situation where when a server fails, the normal operation of multiple applications is affected at the same time.
在第一方面一种可能的实施方式中，所述创建存储空间请求包括第一信息，所述第一信息指示所述第一应用所需的存储空间容量为第一容量，所述第一存储空间的容量大于或等于所述第一容量。本申请实施例中，创建存储空间请求中包括第一信息，第一信息用于指示第一应用所需的存储空间容量为第一容量，其中，第一存储空间的容量大于或等同于第一容量。通过限制第一存储空间的容量大小，避免为第一应用分配存储资源时造成不必要的存储资源浪费，使得技术人员在进行配置时，可以将第一存储空间的容量限制在合理的范围内。In a possible implementation of the first aspect, the request to create a storage space includes first information, the first information indicates that the storage space capacity required by the first application is a first capacity, and the capacity of the first storage space is greater than or equal to the first capacity. In an embodiment of the present application, the request to create a storage space includes first information, the first information is used to indicate that the storage space capacity required by the first application is a first capacity, wherein the capacity of the first storage space is greater than or equal to the first capacity. By limiting the capacity of the first storage space, unnecessary waste of storage resources is avoided when allocating storage resources to the first application, so that the technician can limit the capacity of the first storage space within a reasonable range when performing configuration.
在第一方面一种可能的实施方式中，所述第一信息包括中间数据的分类规则，所述中间数据的分类规则包括所述第一应用的第一类型的中间数据所需的存储空间容量为第一子容量，所述第一应用的第二类型的中间数据所需的存储空间容量为第二子容量，所述第一容量为所述第一子容量和所述第二子容量的和，所述第一存储空间包括第一子存储空间和第二子存储空间；所述选取第一服务器的存储资源作为所述第一存储空间，包括：选取第一服务器的存储资源作为所述第一子存储空间，所述第一子存储空间的容量大于或等于所述第一子容量；选取第二服务器的存储资源作为所述第二子存储空间，所述第二子存储空间的容量大于或等于所述第二子容量。In a possible implementation of the first aspect, the first information includes a classification rule for intermediate data, the classification rule for intermediate data includes that the storage space capacity required for the first type of intermediate data of the first application is a first sub-capacity, the storage space capacity required for the second type of intermediate data of the first application is a second sub-capacity, the first capacity is the sum of the first sub-capacity and the second sub-capacity, and the first storage space includes a first sub-storage space and a second sub-storage space; the selecting the storage resources of the first server as the first storage space includes: selecting the storage resources of the first server as the first sub-storage space, the capacity of the first sub-storage space is greater than or equal to the first sub-capacity; selecting the storage resources of the second server as the second sub-storage space, the capacity of the second sub-storage space is greater than or equal to the second sub-capacity.
本申请实施例中，第一信息中可以包括中间数据的分类规则，中间数据的分类规则包括第一应用的第一类型的中间数据所需的存储空间为第一子容量，第一应用的第二类型的中间数据所需的存储空间为第二子容量，第一容量为第一子容量和第二子容量的和，第一存储空间包括第一子存储空间和第二子存储空间。选取第一服务器的存储资源作为第一存储空间具体为：选取第一服务器的存储资源作为第一子存储空间，第二服务器的存储资源作为第二子存储空间。其中，第一子存储空间的容量大于或等于第一子容量，第二子存储空间的容量大于或等于第二子容量。创建存储空间请求中，通过携带中间数据的分类规则，将不同类型的中间数据进行分类存储，提升了计算设备的存储效率。In an embodiment of the present application, the first information may include a classification rule for intermediate data, and the classification rule for intermediate data includes that the storage space required for the first type of intermediate data of the first application is the first sub-capacity, the storage space required for the second type of intermediate data of the first application is the second sub-capacity, the first capacity is the sum of the first sub-capacity and the second sub-capacity, and the first storage space includes the first sub-storage space and the second sub-storage space. Selecting the storage resources of the first server as the first storage space is specifically: selecting the storage resources of the first server as the first sub-storage space, and the storage resources of the second server as the second sub-storage space. Among them, the capacity of the first sub-storage space is greater than or equal to the first sub-capacity, and the capacity of the second sub-storage space is greater than or equal to the second sub-capacity. In the creation of a storage space request, by carrying the classification rule of intermediate data, different types of intermediate data are classified and stored, thereby improving the storage efficiency of the computing device.
在第一方面一种可能的实施方式中，所述第一应用与第一存储空间之间的对应关系包括所述第一类型的中间数据与所述第一子存储的对应关系和所述第二类型的中间数据与所述第二子存储的对应关系。In a possible implementation of the first aspect, the correspondence between the first application and the first storage space includes a correspondence between the first type of intermediate data and the first sub-storage and a correspondence between the second type of intermediate data and the second sub-storage.
在第一方面一种可能的实施方式中，所述第一存储空间与第二存储空间之间存在映射关系；所述获取所述第一节点发送的第一数据后，所述方法还包括：根据所述第一应用与所述第一存储空间之间的对应关系和所述第一存储空间与所述第二存储空间之间的映射关系，将所述第一数据写入所述第二存储空间。本申请实施例中，由于第一存储空间与第二存储空间之间存在映射关系，在获取第一节点发送的第一数据后，根据第一应用与第一存储空间之间的对应关系和第一存储空间与第二存储空间之间的映射关系，将第一数据写入第二存储空间。第一存储空间与第二存储空间之间存在映射关系时，在获取到第一数据后，根据第一数据与第一存储空间之间的对应关系和第一存储空间与第二存储空间之间的映射关系将第一数据写入第二存储空间，作为第一存储空间中第一数据的备份，提升了存储系统中，数据存储的安全性。In a possible implementation of the first aspect, there is a mapping relationship between the first storage space and the second storage space; after obtaining the first data sent by the first node, the method further includes: writing the first data into the second storage space according to the corresponding relationship between the first application and the first storage space and the mapping relationship between the first storage space and the second storage space. In the embodiment of the present application, since there is a mapping relationship between the first storage space and the second storage space, after obtaining the first data sent by the first node, the first data is written into the second storage space according to the corresponding relationship between the first application and the first storage space and the mapping relationship between the first storage space and the second storage space. When there is a mapping relationship between the first storage space and the second storage space, after obtaining the first data, the first data is written into the second storage space according to the corresponding relationship between the first data and the first storage space and the mapping relationship between the first storage space and the second storage space as a backup of the first data in the first storage space, thereby improving the security of data storage in the storage system.
本申请实施例第二方面提供了一种数据存储的设备，包括：A second aspect of an embodiment of the present application provides a data storage device, including:
获取单元，用于获取第一应用创建后第一节点发送的创建存储空间请求，所述第一节点为与存储系统以可通信方式连接的服务器；an acquisition unit, configured to acquire a storage space creation request sent by a first node after the first application is created, wherein the first node is a server connected to the storage system in a communicable manner;
生成单元，用于生成所述第一应用与第一存储空间之间的对应关系，所述第一存储空间包括至少N个服务器的存储资源，所述存储系统中包括至少N+1个服务器，N为大于1的正整数；A generating unit, configured to generate a correspondence between the first application and a first storage space, wherein the first storage space includes storage resources of at least N servers, and the storage system includes at least N+1 servers, where N is a positive integer greater than 1;
所述获取单元，还用于获取所述第一节点发送的第一数据，所述第一数据为所述第一应用运行时生成的中间数据；The acquiring unit is further configured to acquire first data sent by the first node, where the first data is intermediate data generated when the first application is running;
读写单元，用于根据所述第一应用与所述第一存储空间之间的对应关系将所述第一数据写入所述第一存储空间。A reading and writing unit is used to write the first data into the first storage space according to the corresponding relationship between the first application and the first storage space.
在第二方面一种可能的实施方式中，所述获取单元，还用于获取所述第一应用结束运行时，所述第一节点发送的第一应用结束运行信号；In a possible implementation manner of the second aspect, the acquisition unit is further configured to acquire a first application execution end signal sent by the first node when the first application finishes execution;
所述设备还包括删除单元，用于删除所述第一应用与所述第一存储空间之间的对应关系和所述第一存储空间中存储的所述第一数据。The device further includes a deleting unit configured to delete the correspondence between the first application and the first storage space and the first data stored in the first storage space.
在第二方面一种可能的实施方式中，所述获取单元，还用于获取所述存储系统中可分配存储资源清单，所述可分配存储资源清单包括所述存储系统中可分配的存储资源；In a possible implementation manner of the second aspect, the acquisition unit is further configured to acquire a list of allocatable storage resources in the storage system, where the list of allocatable storage resources includes allocatable storage resources in the storage system;
所述设备还包括选取单元，用于选取第一服务器的存储资源作为所述第一存储空间，所述第一服务器包含于所述存储系统，所述第一服务器的存储资源为所述可分配存储资源清单中的存储资源。The device further includes a selection unit configured to select storage resources of a first server as the first storage space, the first server being included in the storage system, and the storage resources of the first server being storage resources in the list of allocatable storage resources.
在第二方面一种可能的实施方式中，所述第一服务器为所述可分配存储资源清单中可分配存储资源最多的服务器。In a possible implementation of the second aspect, the first server is a server with the most allocatable storage resources in the list of allocatable storage resources.
在第二方面一种可能的实施方式中，所述创建存储空间请求包括第一信息，所述第一信息指示所述第一应用所需的存储空间容量为第一容量，所述第一存储空间的容量大于或等于所述第一容量。In a possible implementation of the second aspect, the storage space creation request includes first information, where the first information indicates that the storage space capacity required by the first application is a first capacity, and the capacity of the first storage space is greater than or equal to the first capacity.
在第二方面一种可能的实施方式中，所述第一信息包括中间数据的分类规则，所述中间数据的分类规则包括所述第一应用的第一类型的中间数据所需的存储空间容量为第一子容量，所述第一应用的第二类型的中间数据所需的存储空间容量为第二子容量，所述第一容量为所述第一子容量和所述第二子容量的和，所述第一存储空间包括第一子存储空间和第二子存储空间；In a possible implementation manner of the second aspect, the first information includes a classification rule for intermediate data, the classification rule for intermediate data includes that a storage space capacity required for intermediate data of a first type of the first application is a first sub-capacity, a storage space capacity required for intermediate data of a second type of the first application is a second sub-capacity, the first capacity is a sum of the first sub-capacity and the second sub-capacity, and the first storage space includes a first sub-storage space and a second sub-storage space;
所述选取单元，具体用于：The selection unit is specifically used for:
选取第一服务器的存储资源作为所述第一子存储空间，所述第一子存储空间的容量大于或等于所述第一子容量；Selecting a storage resource of a first server as the first sub-storage space, wherein the capacity of the first sub-storage space is greater than or equal to the first sub-capacity;
选取第二服务器的存储资源作为所述第二子存储空间，所述第二子存储空间的容量大于或等于所述第二子容量。A storage resource of a second server is selected as the second sub-storage space, and a capacity of the second sub-storage space is greater than or equal to the second sub-capacity.
在第二方面一种可能的实施方式中，所述第一应用与第一存储空间之间的对应关系包括所述第一类型的中间数据与所述第一子存储的对应关系和所述第二类型的中间数据与所述第二子存储的对应关系。In a possible implementation of the second aspect, the correspondence between the first application and the first storage space includes a correspondence between the first type of intermediate data and the first sub-storage and a correspondence between the second type of intermediate data and the second sub-storage.
在第二方面一种可能的实施方式中，所述第一存储空间与第二存储空间之间存在映射关系；In a possible implementation of the second aspect, there is a mapping relationship between the first storage space and the second storage space;
所述读写单元，还用于根据所述第一应用与所述第一存储空间之间的对应关系和所述第一存储空间与所述第二存储空间之间的映射关系，将所述第一数据写入所述第二存储空间。The reading and writing unit is further used to write the first data into the second storage space according to the corresponding relationship between the first application and the first storage space and the mapping relationship between the first storage space and the second storage space.
本申请实施例第三方面提供了一种数据存储的设备，包括处理器和存储器，所述存储器用于存储指令，所述处理器用于执行所述指令，当所述处理器执行所述指令时，执行前述第一方面或第一方面任一种可能的实施方式中所述的方法。A third aspect of an embodiment of the present application provides a data storage device, including a processor and a memory, wherein the memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions, the method described in the first aspect or any possible implementation of the first aspect is executed.
本申请实施例第四方面提供了一种计算机可读存储介质，包括程序，当其在计算机上运行时，使得计算机执行前述第一方面或第一方面任一种可能的实施方式中所述的方法。A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, including a program, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation of the first aspect.
本申请实施例第五方面提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行前述第一方面或第一方面任一种可能的实施方式中所述的方法。A fifth aspect of the embodiments of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation manner of the first aspect.
本申请实施例第六方面提供了一种计算设备集群，其中，包括至少一个计算设备，每个计算设备包括处理器和存储器，至少一个计算设备的处理器用于执行至少一个计算设备的存储器中存储的指令，以使得计算设备集群执行如第一方面的任意一种可能的方法。A sixth aspect of an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device includes a processor and a memory, and the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes any possible method such as the first aspect.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为本申请实施例提供的MapReduce服务集群的一种结构示意图；FIG1 is a schematic diagram of a structure of a MapReduce service cluster provided in an embodiment of the present application;
图2为本申请实施例提供的数据存储的方法的一种流程示意图；FIG2 is a schematic diagram of a flow chart of a method for storing data provided in an embodiment of the present application;
图3为本申请实施例提供的diskpool的一种分布情况示意图；FIG3 is a schematic diagram of a distribution of a diskpool provided in an embodiment of the present application;
图4为本申请实施例提供的第一存储空间的一种分布情况示意图；FIG4 is a schematic diagram of a distribution of a first storage space provided in an embodiment of the present application;
图5为本申请实施例提供的数据存储的方法的另一种流程示意图；FIG5 is another schematic diagram of a flow chart of a method for storing data provided in an embodiment of the present application;
图6为本申请实施例提供的第一存储空间的一种分布示意图；FIG6 is a schematic diagram of a distribution of a first storage space provided in an embodiment of the present application;
图7为本申请实施例提供的数据存储的设备的一种结构示意图；FIG7 is a schematic diagram of a structure of a data storage device provided in an embodiment of the present application;
图8为本申请实施例提供的数据存储的设备的另一种结构示意图；FIG8 is another schematic diagram of the structure of a data storage device provided in an embodiment of the present application;
图9为本申请实施例提供的计算设备集群的一种结构示意图；FIG9 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;
图10为本申请实施例提供的计算设备集群的另一种结构示意图。FIG. 10 is another schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application.
具体实施方式DETAILED DESCRIPTION
下面结合本申请实施例中的附图对本申请实施例进行描述。本申请的实施方式部分使用的术语仅用于对本申请的具体实施例进行解释，而非旨在限定本申请。本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，其目的在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
Spark是一种处理大规模数据的通用计算框架，其作业过程可以分为map阶段和reduce阶段。在原生spark中，一个sparkapp在执行的过程中，在map阶段，由maptask生成shuffle数据并将shuffle数据写入本地文件中以供reduce阶段的取用。在reduce阶段，reducetask从maptask所在的节点获取shuffle数据，并在reducetask所在节点进行数据处理。在sparkapp的执行过程中，存在大量的数据读写和存储需求，将shuffle数据存储于maptask所在节点的本地文件中这一过程，为一个重输入/输出(input/output，IO)轻计算的过程，在大数据量的计算处理中，将shuffle数据存储于maptask所在节点，大幅浪费该节点的计算资源。Spark is a general computing framework for processing large-scale data. Its operation process can be divided into the map phase and the reduce phase. In native Spark, during the execution of a SparkApp, in the map phase, the maptask generates shuffle data and writes the shuffle data to local files for use in the reduce phase. In the reduce phase, the reducetask obtains the shuffle data from the node where the maptask is located and processes the data on the node where the reducetask is located. During the execution of SparkApp, there are a large number of data reading, writing and storage requirements. The process of storing shuffle data in the local file of the node where the maptask is located is an input/output (IO)-light computing process. In the computing and processing of large amounts of data, storing shuffle data on the node where the maptask is located greatly wastes the computing resources of the node.
为降低shuffle数据对maptask所在节点的计算资源的占用，技术人员提出将shuffle服务单独放在一个专门的集群中，该集群具有丰富的存储资源和较弱的计算能力，maptask和reducetask通过网络向该集群读写shuffle数据，以实现sparkapp的存算分离。具体的，可以通过将shuffle数据存储于RSS服务端所在设备的内存中，或者，将shuffle数据存储在RSS服务端的本地盘上，或者，将shuffle数据写入DFS。In order to reduce the computing resource occupation of the maptask node by shuffle data, the technicians proposed to place the shuffle service in a dedicated cluster with abundant storage resources and weak computing power. The maptask and reducetask read and write shuffle data to the cluster through the network to achieve storage and computing separation of sparkapp. Specifically, the shuffle data can be stored in the memory of the device where the RSS server is located, or the shuffle data can be stored on the local disk of the RSS server, or the shuffle data can be written to DFS.
在进行spark计算引擎存算分离的过程中，为充分利用不同设备的存储资源，避免存储资源利用不均衡的情况，通常会采用负载均衡的方式将应用生成的shuffle数据进行分区后，平均地存储于存储系统的所有服务器中。因此，当存储系统中的任一服务器发生故障，都会发生进行存算分离的应用大面积被迫停用的情况。In the process of storage and computing separation of the spark computing engine, in order to make full use of the storage resources of different devices and avoid uneven storage resource utilization, load balancing is usually used to partition the shuffle data generated by the application and evenly store it in all servers of the storage system. Therefore, when any server in the storage system fails, a large area of applications that are separated from storage and computing will be forced to stop working.
为解决这一问题，本申请提出，在sparkapp创建时，向存储集群发送创建存储空间请求，触发存储集群为该sparkapp分配对应的存储空间，当该sparkapp的maptask生成中间数据后，根据存储集群中存储的该sparkapp与对应的存储空间之间的对应关系，向对应的存储空间写入maptask生成的shuffle数据，以供reducetask的取用。To solve this problem, the present application proposes that, when a sparkapp is created, a storage space creation request is sent to the storage cluster, triggering the storage cluster to allocate corresponding storage space for the sparkapp; when the maptask of the sparkapp generates intermediate data, according to the correspondence between the sparkapp stored in the storage cluster and the corresponding storage space, the shuffle data generated by the maptask is written to the corresponding storage space for use by the reducetask.
为便于对方案的理解，首先对本申请实施例具体的应用场景进行介绍，MapReduce部署于多个服务器组成的服务器集群，其中，多个服务器中的每个服务器中包括计算单元(executor)和服务单元(worker)，其中，worker跨服务器组成了读写缓存单元(memartsstore)，其中，多个worker中任一worker被选取为管理单元(master)，在sparkapp运行前触发master创建多个磁盘池(diskpool)。每个diskpool中包括多个容量为128mb的子存储单元(plog)。请参阅图1，图1为本申请实施例提供的MapReduce服务集群的一种结构示意图。To facilitate the understanding of the solution, the specific application scenario of the embodiment of the present application is first introduced. MapReduce is deployed in a server cluster composed of multiple servers, wherein each of the multiple servers includes a computing unit (executor) and a service unit (worker), wherein the workers form a read-write cache unit (memartsstore) across servers, wherein any worker among the multiple workers is selected as a management unit (master), and triggers the master to create multiple disk pools (diskpool) before sparkapp runs. Each diskpool includes multiple sub-storage units (plog) with a capacity of 128mb. Please refer to Figure 1, which is a structural schematic diagram of a MapReduce service cluster provided in an embodiment of the present application.
MapReduce服务集群中包括至少两个服务器，示例性的，至少两个服务器中包括服务器1和服务器2，服务器1包括executor1和worker1；服务器2包括executor2和worker2。1其中，worker1和worker2组成memartsstore，并在worker1和worker2中随机的选取worker1作为memartsstore的master。memartspool将多个服务器的存储资源整合管理，为memartsstore提供存储资源的支持。memartspool跨服务器生成diskpool，其中，每个diskpool中包含至少两个plog。The MapReduce service cluster includes at least two servers. Exemplarily, the at least two servers include server 1 and server 2. Server 1 includes executor 1 and worker 1; server 2 includes executor 2 and worker 2. 1 Among them, worker 1 and worker 2 form memartsstore, and worker 1 is randomly selected from worker 1 and worker 2 as the master of memartsstore. memartspool integrates and manages the storage resources of multiple servers to provide storage resource support for memartsstore. memartspool generates diskpool across servers, and each diskpool contains at least two plogs.
Sparkapp1在服务器1上创建后，通过executor1向master发送创建存储空间请求，创建存储空间请求指示创建容量为A的容器(container)，master将创建存储空间请求转发给memartspool，memartspool创建一个用于存储sparkapp1的shuffle数据的container1。其中，container1中包括特定的plog，也就是sparkapp1的shuffle数据只可以存储于该特定的plog。创建完成后，sparkapp1通过executor1和master去读写特定的plog中的shuffle数据。Sparkapp1结束时，再通过executor向master发送删除存储空间请求，删除container1中特定的plog与sparkapp1之间的对应关系，以及特定的plog中存储的shuffle数据，将plog释放出来以供其他sparkapp使用。After Sparkapp1 is created on server 1, it sends a storage space creation request to the master through executor1. The storage space creation request indicates the creation of a container with a capacity of A. The master forwards the storage space creation request to memartspool, and memartspool creates a container1 for storing the shuffle data of sparkapp1. Among them, container1 includes a specific plog, that is, the shuffle data of sparkapp1 can only be stored in this specific plog. After the creation is completed, sparkapp1 reads and writes the shuffle data in the specific plog through executor1 and the master. When Sparkapp1 ends, it sends a storage space deletion request to the master through the executor to delete the corresponding relationship between the specific plog in container1 and sparkapp1, as well as the shuffle data stored in the specific plog, and release the plog for use by other sparkapps.
基于以上结构和思路，下面将以RSS技术的具体实现为memartsstore为例，结合附图对本申请实施例提供的数据存储的方法进行介绍，请参阅图2，图2为本申请实施例提供的数据存储的方法的一种流程示意图。Based on the above structure and ideas, the following will take the specific implementation of RSS technology as memartsstore as an example, and introduce the data storage method provided by the embodiment of the present application in combination with the accompanying drawings. Please refer to Figure 2, which is a flow chart of the data storage method provided by the embodiment of the present application.
201、获取第一应用创建后发送的创建存储空间请求；201. Obtain a storage space creation request sent by a first application after the application is created;
当第一应用在第一节点上创建后，向memartsstore的master发送创建存储空间请求，该请求中携带了第一信息，第一信息指示第一应用所需的第一存储空间的容量为第一容量。其中，第一节点可以是与存储系统以可通信方式连接的任一服务器。具体的，第一节点可以为存储系统中任一服务器，还可以为存储系统外，任一与存储系统以可通信方式连接的服务器，此处不做限制。After the first application is created on the first node, a request to create a storage space is sent to the master of memartsstore. The request carries the first information, and the first information indicates that the capacity of the first storage space required by the first application is the first capacity. The first node can be any server connected to the storage system in a communicative manner. Specifically, the first node can be any server in the storage system, and can also be any server outside the storage system that is connected to the storage system in a communicative manner, which is not limited here.
本申请实施例中，创建存储空间请求中包括第一信息，第一信息用于指示第一应用所需的存储空间容量为第一容量，其中，第一存储空间的容量大于或等同于第一容量。通过限制第一存储空间的容量大小，避免为第一应用分配存储资源时造成不必要的存储资源浪费，使得技术人员在进行配置时，可以将第一存储空间的容量限制在合理的范围内。In the embodiment of the present application, the request to create a storage space includes first information, and the first information is used to indicate that the storage space capacity required by the first application is the first capacity, wherein the capacity of the first storage space is greater than or equal to the first capacity. By limiting the capacity of the first storage space, unnecessary waste of storage resources when allocating storage resources to the first application is avoided, so that the technician can limit the capacity of the first storage space within a reasonable range when performing configuration.
示例性的，存储系统坐在的服务器集群中，部署了MapReduce系统，在第一节点上创建第一应用后，向该MapReduce系统中的memartsstore发送创建存储空间请求，具体的，向memartsstore中的master发送创建存储空间请求，该创建存储空间请求中携带第一应用的身份标识和第一信息。Exemplarily, the storage system is located in a server cluster and a MapReduce system is deployed. After the first application is created on the first node, a storage space creation request is sent to the memartsstore in the MapReduce system. Specifically, a storage space creation request is sent to the master in the memartsstore, and the storage space creation request carries the identity of the first application and the first information.
示例性的，第一信息可以为第一容量，第一容量为第一应用创建阶段用户通过交互界面进行配置的第一应用的shuffle数据存储容量。Exemplarily, the first information may be a first capacity, where the first capacity is the shuffle data storage capacity of the first application configured by the user through an interactive interface during a first application creation phase.
可以理解的是，此处对步骤201中的应用场景描述仅为示例，在实际应用中，应结合具体的应用场景进行设置，此处不做限制。It is understandable that the description of the application scenario in step 201 here is only an example. In actual application, it should be set in combination with the specific application scenario, and no limitation is made here.
202、生成第一应用与第一存储空间之间的对应关系；202. Generate a correspondence between a first application and a first storage space;
Memartsstore在获取到创建存储空间请求后，根据创建存储空间请求，由memartspool生成第一应用与第一存储空间之间的对应关系，第一存储空间的容量大于或等于第一容量，第一存储空间包括第一服务器的存储资源。并且，第一存储空间包含N个服务器的存储资源，存储系统包括至少N+1个服务器的存储资源，其中，N为大于或等于1的整数。After Memartsstore obtains the request to create a storage space, memartspool generates a corresponding relationship between the first application and the first storage space according to the request to create a storage space. The capacity of the first storage space is greater than or equal to the first capacity, and the first storage space includes the storage resources of the first server. In addition, the first storage space includes the storage resources of N servers, and the storage system includes the storage resources of at least N+1 servers, where N is an integer greater than or equal to 1.
具体的，memartsstore在获取到创建存储空间请求后，根据创建存储空间请求携带的第一容量为第一应用选取对应的存储空间，直至第一存储空间的容量大于或等于第一容量。例如，memartsstore中的master在收到创建存储空间请求后，将该创建存储空间请求转发至memartspool，由memartspool在存储系统创建一个用于存储第一应用对应的shuffle数据的第一存储空间，并生成第一存储空间与第一应用的对应关系，第一存储空间中包括第一服务器的存储资源。Specifically, after obtaining the request to create a storage space, memartsstore selects a corresponding storage space for the first application according to the first capacity carried in the request to create a storage space, until the capacity of the first storage space is greater than or equal to the first capacity. For example, after receiving the request to create a storage space, the master in memartsstore forwards the request to create a storage space to memartspool, which creates a first storage space in the storage system for storing shuffle data corresponding to the first application, and generates a corresponding relationship between the first storage space and the first application, and the first storage space includes the storage resources of the first server.
例如，在创建第一存储空间的过程中，先获取存储系统的可分配存储资源清单，该可分配存储资源清单中包含该存储系统中所有可分配的存储资源，可分配存储资源清单中包括第一服务器的存储资源，第一容量小于第一服务器的存储资源容量，memartspool可以选取第一服务器的存储资源作为第一存储空间，并生成第一存储空间与第一应用的对应关系。若第一容量大于第一服务器的存储资源容量，memartspool在选取第一服务器的存储资源作为第一存储空间的同时，选取第二服务器的存储资源为第一存储空间的存储资源。使第一存储空间的容量大于或等于第一容量，也就是第一服务器的存储资源的容量和第二服务器的存储资源的容量之和大于或等于第一容量。For example, in the process of creating the first storage space, first obtain the list of allocatable storage resources of the storage system, the list of allocatable storage resources includes all the allocatable storage resources in the storage system, the list of allocatable storage resources includes the storage resources of the first server, and the first capacity is less than the storage resource capacity of the first server. Memartspool can select the storage resources of the first server as the first storage space, and generate a corresponding relationship between the first storage space and the first application. If the first capacity is greater than the storage resource capacity of the first server, memartspool selects the storage resources of the second server as the storage resources of the first storage space while selecting the storage resources of the first server as the first storage space. Make the capacity of the first storage space greater than or equal to the first capacity, that is, the sum of the capacity of the storage resources of the first server and the capacity of the storage resources of the second server is greater than or equal to the first capacity.
可以理解的是，此处对创建第一存储空间的过程中，选取第一服务器的存储资源作为第一存储空间的方式仅为示例，在实际应用中，可以根据预先设定好的规则选取可分配存储资源清单中的存储资源作为第一存储空间，此处不做限制。It can be understood that the method of selecting the storage resources of the first server as the first storage space in the process of creating the first storage space is only an example. In actual applications, storage resources in the list of allocatable storage resources can be selected as the first storage space according to pre-set rules, and there is no limitation here.
本申请实施例中，通过获取存储系统的可分配存储资源清单，并根据可分配资源清单选取第一服务器的存储资源作为第一存储空间，可分配存储资源清单中包含该存储系统中所有可分配的存储资源，第一服务器包含于存储系统，第一服务器的存储资源为可分配存储资源清单中的存储资源。通过获取存储系统的可分配存储资源清单，根据清单中所示的存储资源分布情况，按照一定规律对存储资源进行分配，或随机为第一应用生成的中间数据分配存储资源，可以为存储资源的合理利用提供更多的可能性。In the embodiment of the present application, by obtaining a list of allocatable storage resources of the storage system, and selecting the storage resources of the first server as the first storage space according to the list of allocatable resources, the list of allocatable storage resources includes all the allocatable storage resources in the storage system, the first server is included in the storage system, and the storage resources of the first server are the storage resources in the list of allocatable storage resources. By obtaining the list of allocatable storage resources of the storage system, according to the distribution of storage resources shown in the list, the storage resources are allocated according to a certain rule, or storage resources are randomly allocated to the intermediate data generated by the first application, more possibilities can be provided for the reasonable use of storage resources.
更进一步的，为了更加高效的管理存储系统中的存储资源，还可以由memartspool在存储系统中的多个服务器的基础上，创建虚拟存储，利用虚拟存储对存储系统中的服务器的存储资源进行管理。Furthermore, in order to more efficiently manage storage resources in the storage system, memartspool may create virtual storage based on multiple servers in the storage system, and use the virtual storage to manage storage resources of the servers in the storage system.
其中，多个服务器的本地磁盘通过memartspool生成多个diskpool，也即虚拟存储，若干个服务器组成一个或多个diskpool，对外提供plog语义服务，使得外界可以对其中任一个plog进行创建、追加写、读取、封闭和销毁的操作。除此以外，每个plog的容量是固定的，并且具有至少一个副本，该至少一个副本与plog位于不同服务器。由系统内包含的所有服务器的本地磁盘组成的plog资源池和diskpool，组成了存储系统。Among them, the local disks of multiple servers generate multiple diskpools through memartspool, that is, virtual storage. Several servers form one or more diskpools, providing plog semantic services to the outside world, so that the outside world can create, append, read, close and destroy any of the plogs. In addition, the capacity of each plog is fixed and has at least one copy, which is located on a different server from the plog. The plog resource pool and diskpool composed of the local disks of all servers included in the system constitute the storage system.
由于每个plog都会具有至少一个副本，因此由多个plog组成的第一存储空间也存在一个副本第二存储空间。Since each plog has at least one replica, a first storage space composed of multiple plogs also has a replica second storage space.
示例性的，请参阅图3，图3为本申请实施例提供的diskpool的一种分布情况示意图。For example, please refer to FIG. 3 , which is a schematic diagram of a distribution of a diskpool provided in an embodiment of the present application.
由图3可知，由memartspool划分的一个diskpool可以包括服务器1和服务器2的本地磁盘能够提供的存储空间。As shown in FIG. 3 , a diskpool divided by memartspool may include storage spaces that can be provided by local disks of server 1 and server 2 .
由于第一容量可能大于存储系统中任一diskpool的可分配plog资源的容量，因此，可以选取位于多个diskpool的多个子存储空间组成第一存储空间。具体的，可以利用贪心算法，进行第一子存储空间的选取时，memartspool查询DFS系统的可分配存储资源清单，该可分配存储资源清单包括各个diskpool可分配的plog资源，也就是存储系统中可用的存储容量分布情况，将多个diskpool按照剩余可分配存储容量大小，进行从大到小的排序。并从排序中选取可分配资源最多的diskpool中的可分配plog资源组成的第一子存储空间分配给第一应用，即第一子存储空间包含于第一存储空间。若第一存储空间的容量小于第一容量，则继续从排序中选取剩余diskpool中，可分配资源最多的diskpool中的可分配plog资源组成的第二子存储空间分配给第一应用，即第一子存储空间包含于第一存储空间且第二子存储空间包含于第一存储空间。直至第一存储空间的容量大于或等于第一容量，完成第一存储空间的创建。Since the first capacity may be greater than the capacity of the allocatable plog resources of any diskpool in the storage system, multiple sub-storage spaces located in multiple diskpools may be selected to form the first storage space. Specifically, a greedy algorithm may be used to select the first sub-storage space. Memartspool queries the list of allocatable storage resources of the DFS system, which includes the allocatable plog resources of each diskpool, that is, the distribution of the available storage capacity in the storage system, and sorts the multiple diskpools from large to small according to the remaining allocatable storage capacity. The first sub-storage space composed of the allocatable plog resources in the diskpool with the most allocatable resources is selected from the sorting to be allocated to the first application, that is, the first sub-storage space is included in the first storage space. If the capacity of the first storage space is less than the first capacity, the second sub-storage space composed of the allocatable plog resources in the diskpool with the most allocatable resources in the sorting is selected to be allocated to the first application, that is, the first sub-storage space is included in the first storage space and the second sub-storage space is included in the first storage space. Until the capacity of the first storage space is greater than or equal to the first capacity, the creation of the first storage space is completed.
本申请实施例中，通过将可分配存储资源清单中，存储资源最多的服务器分配给第一存储空间，以便使用尽量少的服务器为第一应用的中间数据提供存储服务，降低了第一应用的中间数据占用存储设备的数量，避免当一个服务器发生故障时，同时影响多个应用的正常运行的情况发生。In an embodiment of the present application, the server with the most storage resources in the list of allocatable storage resources is allocated to the first storage space so that as few servers as possible can be used to provide storage services for the intermediate data of the first application. This reduces the number of storage devices occupied by the intermediate data of the first application and avoids the situation where the normal operation of multiple applications is affected simultaneously when a server fails.
示例性的，请参阅图4，图4为本申请实施例提供的第一存储空间的一种分布情况示意图。For example, please refer to FIG. 4 , which is a schematic diagram of a distribution of a first storage space provided in an embodiment of the present application.
由图4可以看出，当存储系统包括diskpool1、diskpool2和diskpool3时，第一存储空间可以包括diskpool1和diskpool2中的部分或者全部plog资源。在这种存储资源的分配规则下，当第一存储空间占用的任一个服务器发生超冗余故障时，只会影响diskpool1和diskpool2中所存储中间数据对应的应用。As can be seen from Figure 4, when the storage system includes diskpool1, diskpool2, and diskpool3, the first storage space may include part or all of the plog resources in diskpool1 and diskpool2. Under this storage resource allocation rule, when a super-redundancy failure occurs in any server occupied by the first storage space, only the applications corresponding to the intermediate data stored in diskpool1 and diskpool2 will be affected.
可选的，memartspool在生成分配完成的第一存储空间与第一应用之间的对应关系后，将第一存储空间对应的plog资源标记为已使用的plog资源。Optionally, after generating the correspondence between the allocated first storage space and the first application, memartspool marks the plog resources corresponding to the first storage space as used plog resources.
可以理解的是，此处对生成第一应用与第一存储空间之间的对应关系时，创建第一存储空间的方法的描述仅为示例，在实际应用中，还可以随机为第一存储空间选取可分配的plog资源，此处不做限制。It is understandable that the description of the method of creating the first storage space when generating the correspondence between the first application and the first storage space is only an example. In actual applications, allocatable plog resources can also be randomly selected for the first storage space, and there is no limitation here.
203、获取第一应用生成的第一数据；203. Acquire first data generated by the first application;
Memartpool生成第一应用与第一存储空间之间的对应关系后，获取第一应用生成的第一数据，第一数据为第一应用生成的中间数据。After Memartpool generates a corresponding relationship between the first application and the first storage space, it obtains first data generated by the first application, where the first data is intermediate data generated by the first application.
具体的，第一应用生成携带第一应用的身份标识(identifydocument，ID)的第一数据，第一数据为第一应用在进行RSS服务时，生成的中间数据，也就是shuffle数据。Memartsstore获取到第一应用生成的第一数据，并转发至memartsstore的master。Specifically, the first application generates first data carrying an identification document (ID) of the first application, and the first data is intermediate data generated by the first application when performing RSS service, that is, shuffle data. Memartsstore obtains the first data generated by the first application and forwards it to the master of memartsstore.
可以理解的是，此处对获取第一应用生成的第一数据的具体应用场景仅为示例，在实际应用中，可以结合实际情况调节具体的应用场景，此处不做限制。It can be understood that the specific application scenario for obtaining the first data generated by the first application is only an example. In actual applications, the specific application scenario can be adjusted according to the actual situation, and no limitation is made here.
204、根据第一应用与第一存储空间之间的对应关系将第一数据写入第一存储空间；204. Write the first data into the first storage space according to the correspondence between the first application and the first storage space;
Memartsstore在获取到第一数据后，根据第一应用与第一存储空间之间的对应关系将第一数据写入第一存储空间。After acquiring the first data, Memartsstore writes the first data into the first storage space according to the corresponding relationship between the first application and the first storage space.
具体的，memartsstore中的memartspool在获取到第一应用生成的第一数据后，根据第一数据携带的第一应用的ID，确定第一数据与第一应用的关联性，再根据memartpool生成的第一应用与第一存储空间之间的对应关系，将第一数据写入第一存储空间。Specifically, after obtaining the first data generated by the first application, memartspool in memartsstore determines the association between the first data and the first application based on the ID of the first application carried by the first data, and then writes the first data into the first storage space based on the correspondence between the first application and the first storage space generated by memartpool.
示例性的，第一数据可以随机的写入第一存储空间的组成成分：第一子存储空间和第二子存储空间中；或者，利用负载均衡的方法，平均地将第一数据写入第一子存储空间和第二子存储空间；除此以外，还可以依次向第一子存储空间写入第一数据，当第一子存储空间被完全占用后，将剩余的第一数据写入第二子存储空间。Exemplarily, the first data can be randomly written into the components of the first storage space: the first sub-storage space and the second sub-storage space; or, using a load balancing method, the first data can be evenly written into the first sub-storage space and the second sub-storage space; in addition, the first data can be written to the first sub-storage space in sequence, and when the first sub-storage space is fully occupied, the remaining first data can be written into the second sub-storage space.
可选的，memartspool根据第一存储空间与第二存储空间的映射关系，将写入第一存储空间的第一数据写入第二存储空间。Optionally, memartspool writes the first data written into the first storage space into the second storage space according to a mapping relationship between the first storage space and the second storage space.
需要说明的是，此处对第一数据写入第一存储空间的方式仅为示例，在实际应用中，应当结合具体的应用场景对应的需求进行设置，此处不做限制。It should be noted that the method of writing the first data into the first storage space here is only an example. In actual application, it should be set in combination with the requirements corresponding to the specific application scenario, and is not limited here.
205、获取第一应用结束运行时发送的第一应用结束运行信号；205. Obtain a first application execution termination signal sent when the first application terminates execution;
Memartsstore在第一应用结束运行时，获取第一应用发送的第一应用结束运行信号。When the first application finishes running, Memartsstore obtains a first application running end signal sent by the first application.
具体的，当第一应用运行结束时，第一应用会发送第一应用结束运行信号给memarts store，由memartstore向memartsstore中的memartspool转发该第一应用结束运行信号。Specifically, when the first application ends running, the first application will send a first application running end signal to the memarts store, and the memartstore will forward the first application running end signal to the memartspool in the memartsstore.
206、删除第一应用与第一存储空间之间的对应关系和第一存储空间中存储的第一数据。206. Delete the correspondence between the first application and the first storage space and the first data stored in the first storage space.
Memartsstore在获取到第一应用结束运行信号后，向memartsstore中的memartspool转发该第一应用结束运行信号。Memartspool在接收到第一应用结束运行信号后，删除第一应用与第一存储空间之间的对应关系，并且销毁第一存储空间中存储的第一数据。After obtaining the first application end-of-run signal, Memartsstore forwards the first application end-of-run signal to memartspool in memartsstore. After receiving the first application end-of-run signal, Memartspool deletes the corresponding relationship between the first application and the first storage space, and destroys the first data stored in the first storage space.
可选的，memartspool根据第一存储空间与第二存储空间之间的映射关系删除第二存储空间的第一数据。Optionally, memartspool deletes the first data in the second storage space according to a mapping relationship between the first storage space and the second storage space.
进一步的，memartspool将第一存储空间中包含的plog标记为可分配资源。Furthermore, memartspool marks the plog contained in the first storage space as an allocatable resource.
可以理解的是，此处对memartsstore删除第一应用与第一存储空间之间的对应关系和第一存储空间中存储的第一数据的具体操作仅为示例，在实际应用中，可以结合其他RSS技术实现，此处不做限制。It can be understood that the specific operation of memartsstore deleting the correspondence between the first application and the first storage space and the first data stored in the first storage space is only an example. In actual applications, it can be implemented in combination with other RSS technologies and is not limited here.
本申请实施例中，在第一应用结束运行时，获取第一节点发送的第一应用结束运行信号，并删除第一应用与第一存储空间之间的对应关系和第一存储空间中存储的第一数据，在第一应用结束运行后，及时释放为第一应用的中间数据分配的存储资源。提升了存储系统中，存储资源的利用效率。In the embodiment of the present application, when the first application ends running, a signal indicating that the first application has ended running is obtained from the first node, and the corresponding relationship between the first application and the first storage space and the first data stored in the first storage space are deleted. After the first application ends running, the storage resources allocated for the intermediate data of the first application are released in time, thereby improving the utilization efficiency of storage resources in the storage system.
本申请实施例中，通过获取第一应用创建后发送的创建存储空间请求，生成第一应用与第一存储空间之间的对应关系后，根据第一应用与第一存储空间之间的对应关系将获取到的第一数据写入第一存储空间，其中，第一数据为第一应用生成的中间数据，第一存储空间包括N个服务器的存储资源，N个服务器包含于存储系统，且该存储系统包含至少N+1个服务器的存储资源。在第一应用结束运行时，获取第一应用发送的第一应用结束运行信号后，删除第一应用与第一存储空间之间的对应关系和第一存储空间中存储的第一数据，以释放第一应用占用的第一存储空间。通过为应用划分存储空间，确保第一应用出现超冗余故障的时候，中间数据存储于其他虚拟存储空间的应用仍可以正常运行，降低了第一服务器发生故障时，对中间数据存储于存储系统的应用的影响范围。In an embodiment of the present application, by obtaining a storage space creation request sent after the first application is created, after generating a correspondence between the first application and the first storage space, the first data obtained is written into the first storage space according to the correspondence between the first application and the first storage space, wherein the first data is intermediate data generated by the first application, the first storage space includes storage resources of N servers, the N servers are included in the storage system, and the storage system includes storage resources of at least N+1 servers. When the first application ends its operation, after obtaining a first application end operation signal sent by the first application, the correspondence between the first application and the first storage space and the first data stored in the first storage space are deleted to release the first storage space occupied by the first application. By dividing the storage space for the application, it is ensured that when the first application has a super-redundant failure, the application whose intermediate data is stored in other virtual storage spaces can still operate normally, which reduces the scope of influence on the application whose intermediate data is stored in the storage system when the first server fails.
基于前述图2中的方法，为了提升存储效率，还可以在为第一应用分配第一存储空间时，将第一存储空间中的子存储空间按照存储数据的功能进行分类，将第一数据中用于特定功能的的数据存储于特定的子存储空间。具体实现方式请参阅图5，图5为本申请实施例提供的数据存储的方法的另一种流程示意图。Based on the method in FIG. 2 , in order to improve storage efficiency, when allocating the first storage space for the first application, the sub-storage spaces in the first storage space can be classified according to the functions of the stored data, and the data used for specific functions in the first data can be stored in specific sub-storage spaces. For a specific implementation method, please refer to FIG. 5 , which is another flow chart of the data storage method provided in an embodiment of the present application.
501、获取第一应用创建后发送的创建存储空间请求；501. Obtain a storage space creation request sent by a first application after the application is created;
当第一应用在第一节点上创建后，向memartsstore的master发送创建存储空间请求，该请求中携带了第一信息，第一信息指示第一应用所需的第一存储空间的容量为第一容量。其中，第一节点可以与存储系统以可通信方式连接的任一服务器。具体的，第一节点可以为存储系统中任一服务器，还可以为存储系统外，任一与存储系统以可通信方式连接的服务器，此处不做限制。After the first application is created on the first node, a request to create a storage space is sent to the master of memartsstore. The request carries the first information, and the first information indicates that the capacity of the first storage space required by the first application is the first capacity. Among them, the first node can be any server connected to the storage system in a communicative manner. Specifically, the first node can be any server in the storage system, and can also be any server outside the storage system that is connected to the storage system in a communicative manner, which is not limited here.
示例性的，第一信息包括中间数据的分类规则，中间数据的分类规则可以为第一应用生成的第一类型的中间数据需要第一子容量的存储空间，第一应用生成的第二类型的中间数据需要第二子容量的存储空间。Exemplarily, the first information includes a classification rule for the intermediate data, and the classification rule for the intermediate data may be that the first type of intermediate data generated by the first application requires a first sub-capacity of storage space, and the second type of intermediate data generated by the first application requires a second sub-capacity of storage space.
可以理解的是，此处对中间数据的分类规则的说明仅为示例，在实际应用中可以有多种不同的解释，此处不做限制。It is understandable that the description of the classification rules for intermediate data here is only an example, and there may be many different interpretations in actual applications, which are not limited here.
由于中间数据的每个分类的容量太大的话，每个分类的中间数据读写任务会出现执行时间过长或并发过大、消耗资源过多等情况，因此，技术人员在配置中间数据的分类规则时，会充分考虑分类的容量大小，将分类的容量控制在一定范围内，例如，将每个分类的容量设置为128MB。If the capacity of each category of intermediate data is too large, the execution time of the intermediate data reading and writing tasks of each category will be too long or the concurrency will be too large, and too many resources will be consumed. Therefore, when configuring the classification rules of intermediate data, the technicians will fully consider the capacity of the category and control the capacity of the category within a certain range. For example, the capacity of each category is set to 128MB.
具体的，可以根据中间数据的分类规则和每个分类的容量为128MB，按照如下第一公式计算获得第一容量：Specifically, according to the classification rule of the intermediate data and the capacity of each classification being 128 MB, the first capacity can be calculated using the following first formula:
第一容量＝分类数量*128MB。The first capacity = number of categories * 128MB.
示例性的，MapReduce系统中，第一应用在第一节点创建后，第一节点向该MapReduce集群中的memartsstore的memartspool发送创建存储空间请求，该创建存储空间请求中携带第一应用的身份标识和中间数据的分类规则，根据中间数据的分类规则按照第一公式计算获得第一容量，第一容量为第一应用创建阶段用户通过交互界面进行配置的第一应用的shuffle数据存储容量，中间数据的分类规则为根据shuffle数据的类型不同对其进行存储位置的划分规则。Exemplarily, in a MapReduce system, after a first application is created on a first node, the first node sends a storage space creation request to the memartspool of the memartsstore in the MapReduce cluster. The storage space creation request carries the identity of the first application and the classification rule of intermediate data. A first capacity is calculated according to the classification rule of the intermediate data using a first formula. The first capacity is the shuffle data storage capacity of the first application configured by a user through an interactive interface during the first application creation phase. The classification rule of intermediate data is a rule for dividing the storage location of shuffle data according to different types of shuffle data.
可以理解的是，此处对步骤501的应用场景描述和每个分类的容量说明仅为示例，在实际应用中，应结合具体的应用场景进行设置，此处不做限制。It is understandable that the description of the application scenario of step 501 and the capacity description of each category are only examples. In actual application, they should be set in combination with specific application scenarios and are not limited here.
502、生成第一类型的中间数据与第一子存储空间的对应关系和第二类型的中间数据与第一存储空间之间的对应关系；502. Generate a correspondence between the first type of intermediate data and the first sub-storage space and a correspondence between the second type of intermediate data and the first storage space;
Memartsstore在获取到创建存储空间请求后，根据创建存储空间请求，由memartspool选取第一服务器的存储资源作为第一子存储空间，选取第二服务器的存储资源作为第二子存储空间，并生成第一类型的中间数据与第一子存储空间之间的对应关系和第二类型的中间数据与第二子存储空间之间的对应关系。其中，第一子存储空间为第一类型的中间数据的存储空间，第二子存储空降为第二类型的中间数据的存储空间，第一子存储空间的容量大于或等于第一子容量，第二子存储空间的容量大于或等于第二子容量，第一子存储空间和第二子存储空间包含于第一存储空间。After Memartsstore receives the request to create a storage space, memartspool selects the storage resources of the first server as the first sub-storage space and the storage resources of the second server as the second sub-storage space according to the request to create a storage space, and generates a correspondence between the first type of intermediate data and the first sub-storage space and a correspondence between the second type of intermediate data and the second sub-storage space. The first sub-storage space is the storage space for the first type of intermediate data, the second sub-storage space is the storage space for the second type of intermediate data, the capacity of the first sub-storage space is greater than or equal to the first sub-capacity, the capacity of the second sub-storage space is greater than or equal to the second sub-capacity, and the first sub-storage space and the second sub-storage space are included in the first storage space.
为了更加灵活的管理存储系统中的存储资源，还可以由memartspool在存储系统中的多个服务器的基础上，创建虚拟存储，利用虚拟存储对存储系统中的服务器的存储资源进行管理。In order to manage the storage resources in the storage system more flexibly, memartspool can also create virtual storage based on multiple servers in the storage system, and use the virtual storage to manage the storage resources of the servers in the storage system.
其中，多个服务器的本地磁盘通过memartspool生成多个diskpool，也即虚拟存储，若干个服务器组成一个或多个diskpool，对外提供plog语义服务，使得外界可以对其中任一个plog进行创建、追加写、读取、封闭和销毁的操作。除此以外，每个plog的容量是固定的，并且具有至少一个副本，该至少一个副本与plog位于不同服务器。由系统内包含的所有服务器的本地磁盘组成的plog资源池和diskpool，组成了存储系统。Among them, the local disks of multiple servers generate multiple diskpools through memartspool, that is, virtual storage. Several servers form one or more diskpools, providing plog semantic services to the outside world, so that the outside world can create, append, read, close and destroy any of the plogs. In addition, the capacity of each plog is fixed and has at least one copy, which is located on a different server from the plog. The plog resource pool and diskpool composed of the local disks of all servers included in the system constitute the storage system.
具体的，第一子存储空间包含于第一虚拟存储，第一虚拟存储包括第一服务器中的存储资源。第二子存储空间包含于第二虚拟存储，第二虚拟存储包括第二服务器中的存储资源。请参阅图6，图6为本申请实施例提供的第一存储空间的一种分布示意图。Specifically, the first sub-storage space is included in the first virtual storage, and the first virtual storage includes storage resources in the first server. The second sub-storage space is included in the second virtual storage, and the second virtual storage includes storage resources in the second server. Please refer to Figure 6, which is a distribution diagram of the first storage space provided in an embodiment of the present application.
由图6可知，第一服务器中的存储资源和第三服务器中的存储资源组成了第一虚拟存储，memartspool可以访问设置于第一虚拟存储的第一子存储空间进行数据读写，进而通过第一虚拟存储访问第一服务器和/或第三服务器的存储资源。第二服务器中的存储资源组成了第二虚拟存储，使得memartspool可以访问设置于第二虚拟存储的第二子存储空间进行数据读写，进而通过第二虚拟存储访问第二服务器中的存储资源。As shown in FIG6 , the storage resources in the first server and the storage resources in the third server constitute the first virtual storage, and memartspool can access the first sub-storage space set in the first virtual storage to read and write data, and then access the storage resources of the first server and/or the third server through the first virtual storage. The storage resources in the second server constitute the second virtual storage, so that memartspool can access the second sub-storage space set in the second virtual storage to read and write data, and then access the storage resources in the second server through the second virtual storage.
可以理解的是，此处仅以memartspool通过访问设置于第一虚拟存储的第一子存储空间，并且第一子存储空间只占用了第一服务器中的存储资源；memartspool通过访问设置于第二虚拟存储的第二子存储空间，第二自存储空间占用第二服务器中的存储资源为例进行说明，在实际应用中，根据实际的需求，第二子存储空间还可以设置于第一虚拟存储，此处不做限制。It can be understood that here only the example of memartspool accessing the first sub-storage space set in the first virtual storage, and the first sub-storage space only occupies the storage resources in the first server; memartspool accesses the second sub-storage space set in the second virtual storage, and the second self-storage space occupies the storage resources in the second server is used for explanation. In actual applications, according to actual needs, the second sub-storage space can also be set in the first virtual storage, and there is no limitation here.
示例性的，memartsstore中的master在收到创建存储空间请求后，将该创建存储空间请求传达至memartspool，由memartspool在存储系统创建用于存储第一应用对应的shuffle数据中第一类型shuffle数据的第一子存储空间和用于存储第二类型shuffle数据的第二子存储空间，并生成第一子存储空间与第一类型的中间数据的对应关系和第二子存储空间与第二类型的中间数据之间的对应关系，第一存储空间由第一子存储空间和第二子存储空间组成。Exemplarily, after receiving a request to create a storage space, the master in memartsstore communicates the request to create a storage space to memartspool, and memartspool creates a first sub-storage space for storing the first type of shuffle data in the shuffle data corresponding to the first application and a second sub-storage space for storing the second type of shuffle data in the storage system, and generates a correspondence between the first sub-storage space and the first type of intermediate data and a correspondence between the second sub-storage space and the second type of intermediate data. The first storage space consists of the first sub-storage space and the second sub-storage space.
可选的，memartspool在生成第一子存储空间与第一类型的中间数据的对应关系和第二子存储空间与第二类型的中间数据之间的对应关系后，将第一存储空间对应的plog资源标记为已使用的plog资源。Optionally, after generating the correspondence between the first sub-storage space and the first type of intermediate data and the correspondence between the second sub-storage space and the second type of intermediate data, memartspool marks the plog resources corresponding to the first storage space as used plog resources.
可选的，由于每个plog会具有至少一个副本，第一存储空间中的每一个plog都有对应的备份plog，这些备份plog组成了第二存储空间。当每个plog写入数据后，器备份plog也会写入相同的数据，因此，由备份plog组成的第二存储空间也会在第一存储空间写入第一数据的时候写入第一数据。Optionally, since each plog has at least one copy, each plog in the first storage space has a corresponding backup plog, and these backup plogs constitute the second storage space. When each plog writes data, the backup plog will also write the same data, so the second storage space composed of the backup plogs will also write the first data when the first data is written to the first storage space.
本申请实施例中，由于第一存储空间与第二存储空间之间存在映射关系，在获取第一节点发送的第一数据后，根据第一应用与第一存储空间之间的对应关系和第一存储空间与第二存储空间之间的映射关系，将第一数据写入第二存储空间。第一存储空间与第二存储空间之间存在映射关系时，在获取到第一数据后，根据第一数据与第一存储空间之间的对应关系和第一存储空间与第二存储空间之间的映射关系将第一数据写入第二存储空间，作为第一存储空间中第一数据的备份，提升了存储系统中，数据存储的安全性。In the embodiment of the present application, since there is a mapping relationship between the first storage space and the second storage space, after obtaining the first data sent by the first node, the first data is written into the second storage space according to the corresponding relationship between the first application and the first storage space and the mapping relationship between the first storage space and the second storage space. When there is a mapping relationship between the first storage space and the second storage space, after obtaining the first data, the first data is written into the second storage space according to the corresponding relationship between the first data and the first storage space and the mapping relationship between the first storage space and the second storage space as a backup of the first data in the first storage space, thereby improving the security of data storage in the storage system.
可以理解的是，此处对生成第一子存储空间与第一类型的中间数据的对应关系和第二子存储空间与第二类型的中间数据之间的对应关系时，创建第一存储空间的方法的描述仅为示例，在实际应用中应结合具体的应用场景进行设置，此处不做限制。It can be understood that the description of the method of creating the first storage space when generating the correspondence between the first sub-storage space and the first type of intermediate data and the correspondence between the second sub-storage space and the second type of intermediate data is only an example. In actual application, it should be set in combination with the specific application scenario and is not limited here.
503、获取第一应用生成的第一数据；503. Acquire first data generated by the first application;
步骤503与前述图2中步骤203的操作类似，此处不再赘述，请参阅前述步骤203的介绍。The operation of step 503 is similar to that of step 203 in FIG. 2 , and will not be described again here. Please refer to the introduction of step 203 .
504、根据第一应用与第一存储空间之间的对应关系和中间数据的分类规则将第一数据写入第一存储空间；504. Write the first data into the first storage space according to the correspondence between the first application and the first storage space and the classification rule of the intermediate data;
Memartsstore在获取到第一数据后，memartspool根据中间数据的分类规则将第一数据进行分类，获得第一类型的第一数据和第二类型的第一数据，并根据第一应用与第一存储空间之间的对应关系将第一类型的第一数据写入第一子存储空间，第二类型的第一数据写入第二子存储空间，其中，第一数据为第一应用生成的中间数据。After Memartsstore obtains the first data, Memartspool classifies the first data according to the classification rule of the intermediate data to obtain the first data of the first type and the first data of the second type, and writes the first data of the first type into the first sub-storage space and the first data of the second type into the second sub-storage space according to the correspondence between the first application and the first storage space, wherein the first data is the intermediate data generated by the first application.
需要说明的是，此处对第一数据写入第一存储空间的方式仅为示例，在实际应用中，应当结合具体的应用场景对应的需求进行设置，此处不做限制。It should be noted that the method of writing the first data into the first storage space here is only an example. In actual application, it should be set in combination with the requirements corresponding to the specific application scenario, and is not limited here.
505、获取第一应用结束运行时发送的第一应用结束运行信号；505. Obtain a first application execution termination signal sent when the first application terminates execution;
步骤505与前述图2中步骤205的操作类似，此处不再赘述，请参阅前述步骤205的介绍。The operation of step 505 is similar to that of step 205 in FIG. 2 , and will not be described again here. Please refer to the introduction of step 205 .
506、删除第一子存储空间与第一类型的中间数据之间的对应关系、第二子存储空间与第二类型的中间数据之间的对应关系和第一存储空间中的第一数据。506. Delete the correspondence between the first sub-storage space and the intermediate data of the first type, the correspondence between the second sub-storage space and the intermediate data of the second type, and the first data in the first storage space.
Memartsstore在获取到第一应用结束运行信号后，向memartsstore中的memartspool转发该第一应用结束运行信号。Memartspool在接收到第一应用结束运行信号后，删除第一子存储空间与第一类型的中间数据之间的对应关系和第二子存储空间与第二类型的中间数据之间的对应关系，并且销毁第一存储空间中存储的所有第一数据。After receiving the signal that the first application has ended, Memartsstore forwards the signal to memartspool in memartsstore. After receiving the signal that the first application has ended, Memartspool deletes the correspondence between the first sub-storage space and the first type of intermediate data and the correspondence between the second sub-storage space and the second type of intermediate data, and destroys all first data stored in the first storage space.
进一步的，memartspool将第一存储空间中包含的plog标记为可分配资源。Furthermore, memartspool marks the plog contained in the first storage space as an allocatable resource.
可以理解的是，此处对memartsstore删除第一子存储空间与第一类型的中间数据之间的对应关系、第二子存储空间与第二类型的中间数据之间的对应关系和第一存储空间中的第一数据的具体操作仅为示例，在实际应用中，可以结合其他RSS技术实现，此处不做限制。It can be understood that the specific operation of memartsstore deleting the correspondence between the first sub-storage space and the first type of intermediate data, the correspondence between the second sub-storage space and the second type of intermediate data, and the first data in the first storage space is only an example. In actual applications, it can be implemented in combination with other RSS technologies and is not limited here.
本申请实施例中，在获取第一应用发送的创建存储空间请求后，根据创建存储空间请求中携带的中间数据的分类规则，生成第一应用的第一类型的中间数据与第一子存储空间的对应关系和第一应用的第二类型的中间数据与第二子存储空间的对应关系。在获得第一数据后将第一数据进行分类，将第一类型的第一数据写入第一子存储空间；第二类型的第一数据写入第二子存储空间。并且在获取第一应用在第一应用结束时发送的第一应用结束运行信号，在获取到该第一应用结束运行信号后，删除第一子存储空间与第一类型的中间数据之间的对应关系、第二子存储空间与第二类型的中间数据之间的对应关系和第一存储空间中的第一数据。其中，第一存储空间包括第一子存储空间和第二子存储空间。In an embodiment of the present application, after obtaining a storage space creation request sent by a first application, a correspondence between the first type of intermediate data of the first application and the first sub-storage space and a correspondence between the second type of intermediate data of the first application and the second sub-storage space are generated according to the classification rules of the intermediate data carried in the storage space creation request. After obtaining the first data, the first data is classified, and the first data of the first type is written into the first sub-storage space; the first data of the second type is written into the second sub-storage space. And after obtaining the first application end operation signal sent by the first application when the first application ends, after obtaining the first application end operation signal, the correspondence between the first sub-storage space and the first type of intermediate data, the correspondence between the second sub-storage space and the second type of intermediate data, and the first data in the first storage space are deleted. Among them, the first storage space includes the first sub-storage space and the second sub-storage space.
以上对本申请提出的方案中，以存储系统为DFS系统为例进行了介绍，当存储系统为Linux本地文件系统时，第一存储空间中资源分配的最小粒度为服务器的硬盘，而非plog。Memartspool在生成第一存储空间与第一应用之间的对应关系时，生成第一应用与指定的文件目录之间的对应关系，实现第一应用与硬盘之间的绑定。In the above solution proposed in this application, the storage system is introduced as a DFS system as an example. When the storage system is a Linux local file system, the minimum granularity of resource allocation in the first storage space is the hard disk of the server, not plog. When Memartspool generates the correspondence between the first storage space and the first application, it generates the correspondence between the first application and the specified file directory to achieve the binding between the first application and the hard disk.
本申请实施例中，通过将存储系统设置为Linux本地文件系统，降低了生成第一存储空间与第一应用的对应关系这一操作的难度，从而提升了方案的实施效率。In the embodiment of the present application, by setting the storage system as a Linux local file system, the difficulty of generating the corresponding relationship between the first storage space and the first application is reduced, thereby improving the implementation efficiency of the solution.
以上对本申请实施例提供的数据存储的方法进行了介绍，下面结合附图对本申请实施例提供的数据存储的设备进行介绍，请参阅图7，图7为本申请实施例提供的数据存储的设备的一种结构示意图。The above introduces the data storage method provided in the embodiment of the present application. The following introduces the data storage device provided in the embodiment of the present application in conjunction with the accompanying drawings. Please refer to Figure 7, which is a structural schematic diagram of the data storage device provided in the embodiment of the present application.
该数据存储的设备包括：The data storage devices include:
获取单元701，用于获取第一应用创建后第一节点发送的创建存储空间请求，第一节点为与存储系统以可通信方式连接的服务器；The acquisition unit 701 is used to acquire a storage space creation request sent by a first node after the first application is created, where the first node is a server connected to the storage system in a communicable manner;
生成单元702，用于生成第一应用与第一存储空间之间的对应关系，第一存储空间包括至少N个服务器的存储资源，存储系统中包括至少N+1个服务器，N为大于1的正整数；A generating unit 702 is configured to generate a correspondence between a first application and a first storage space, wherein the first storage space includes storage resources of at least N servers, and the storage system includes at least N+1 servers, where N is a positive integer greater than 1;
获取单元701，还用于获取第一节点发送的第一数据，第一数据为第一应用运行时生成的中间数据；The acquiring unit 701 is further configured to acquire first data sent by the first node, where the first data is intermediate data generated when the first application is running;
读写单元703，用于根据第一应用与第一存储空间之间的对应关系将第一数据写入第一存储空间。The read/write unit 703 is configured to write the first data into the first storage space according to the corresponding relationship between the first application and the first storage space.
可选的，获取单元701，还用于获取第一应用结束运行时，第一节点发送的第一应用结束运行信号；Optionally, the acquisition unit 701 is further configured to acquire a first application execution end signal sent by the first node when the first application finishes execution;
设备还包括删除单元704，用于删除第一应用与第一存储空间之间的对应关系和第一存储空间中存储的第一数据。The device further includes a deleting unit 704, configured to delete the correspondence between the first application and the first storage space and the first data stored in the first storage space.
可选的，获取单元701，还用于获取存储系统中可分配存储资源清单，可分配存储资源清单包括存储系统中可分配的存储资源；Optionally, the acquisition unit 701 is further configured to acquire a list of allocatable storage resources in the storage system, where the list of allocatable storage resources includes allocatable storage resources in the storage system;
设备还包括选取单元705，用于选取第一服务器的存储资源作为第一存储空间，第一服务器包含于存储系统，第一服务器的存储资源为可分配存储资源清单中的存储资源。The device also includes a selection unit 705, which is used to select storage resources of a first server as a first storage space, the first server being included in the storage system, and the storage resources of the first server being storage resources in the list of allocatable storage resources.
可选的，第一服务器为可分配存储资源清单中可分配存储资源最多的服务器。Optionally, the first server is a server with the most allocatable storage resources in the list of allocatable storage resources.
可选的，创建存储空间请求包括第一信息，第一信息指示第一应用所需的存储空间容量为第一容量，第一存储空间的容量大于或等于第一容量。Optionally, the request to create a storage space includes first information, where the first information indicates that the storage space capacity required by the first application is a first capacity, and the capacity of the first storage space is greater than or equal to the first capacity.
可选的，第一信息包括中间数据的分类规则，中间数据的分类规则包括第一应用的第一类型的中间数据所需的存储空间容量为第一子容量，第一应用的第二类型的中间数据所需的存储空间容量为第二子容量，第一容量为第一子容量和第二子容量的和，第一存储空间包括第一子存储空间和第二子存储空间；Optionally, the first information includes a classification rule for the intermediate data, the classification rule for the intermediate data includes that the storage space capacity required for the first type of intermediate data of the first application is the first sub-capacity, the storage space capacity required for the second type of intermediate data of the first application is the second sub-capacity, the first capacity is the sum of the first sub-capacity and the second sub-capacity, and the first storage space includes the first sub-storage space and the second sub-storage space;
选取单元705，具体用于：The selection unit 705 is specifically used for:
选取第一服务器的存储资源作为第一子存储空间，第一子存储空间的容量大于或等于第一子容量；Selecting a storage resource of the first server as a first sub-storage space, wherein the capacity of the first sub-storage space is greater than or equal to the first sub-capacity;
选取第二服务器的存储资源作为第二子存储空间，第二子存储空间的容量大于或等于第二子容量。The storage resources of the second server are selected as the second sub-storage space, and the capacity of the second sub-storage space is greater than or equal to the second sub-capacity.
可选的，第一应用与第一存储空间之间的对应关系包括第一类型的中间数据与第一子存储的对应关系和第二类型的中间数据与第二子存储的对应关系。Optionally, the correspondence between the first application and the first storage space includes a correspondence between the first type of intermediate data and the first sub-storage and a correspondence between the second type of intermediate data and the second sub-storage.
可选的，第一存储空间与第二存储空间之间存在映射关系；Optionally, there is a mapping relationship between the first storage space and the second storage space;
读写单元703，还用于根据第一应用与第一存储空间之间的对应关系和第一存储空间与第二存储空间之间的映射关系，将第一数据写入第二存储空间。The read-write unit 703 is further configured to write the first data into the second storage space according to the corresponding relationship between the first application and the first storage space and the mapping relationship between the first storage space and the second storage space.
除此以外，本申请实施例还提供了一种数据处理的设备，请参阅图8，图8为本申请实施例提供的数据存储的设备的另一种结构示意图。In addition, the embodiment of the present application further provides a data processing device. Please refer to FIG. 8 . FIG. 8 is another structural schematic diagram of the data storage device provided in the embodiment of the present application.
数据存储的设备800包括：总线802、处理器804、存储器806和通信接口808。处理器804、存储器806和通信接口808之间通过总线802通信。数据存储的设备800可以是服务器或终端设备。应理解，本申请不限定数据存储的设备800中的处理器、存储器的个数。The data storage device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other through the bus 802. The data storage device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the data storage device 800.
总线802可以是外设部件互连标准(peripheralcomponentinterconnect，PCI)总线或扩展工业标准结构(extendedindustrystandardarchitecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图8中仅用一条线表示，但并不表示仅有一根总线或一种类型的总线。总线802可包括在数据存储的设备800各个部件(例如，存储器806、处理器804、通信接口808)之间传送信息的通路。The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG8 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the data storage device 800 (e.g., the memory 806, the processor 804, and the communication interface 808).
处理器804可以包括中央处理器(centralprocessingunit，CPU)、图形处理器(graphicsprocessingunit，GPU)、微处理器(microprocessor，MP)或者数字信号处理器(digitalsignalprocessor，DSP)等处理器中的任意一种或多种。The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
存储器806可以包括易失性存储器(volatilememory)，例如随机存取存储器(random accessmemory，RAM)。处理器804还可以包括非易失性存储器(non-volatilememory)，例如只读存储器(read-onlymemory，ROM)，快闪存储器，机械硬盘(harddiskdrive，HDD)或固态硬盘(solidstatedrive，SSD)。The memory 806 may include a volatile memory, such as a random access memory (RAM). The processor 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
存储器806中存储有可执行的程序代码，处理器804执行该可执行的程序代码以分别实现前述获取单元701、生成单元702、读写单元703、删除单元704和选取单元705的功能，从而实现数据存储方法中的任一项方法。也即，存储器806上存有用于执行数据存储方法中任一项方法的指令。The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the functions of the aforementioned acquisition unit 701, generation unit 702, read-write unit 703, deletion unit 704, and selection unit 705, thereby implementing any one of the data storage methods. That is, the memory 806 stores instructions for executing any one of the data storage methods.
或者，存储器806中存储有可执行的代码，处理器804执行该可执行的代码以分别实现前述接入点的功能，从而实现数据存储方法中的任一项方法。也即，存储器806上存有用于执行数据存储方法中任一项方法的指令。Alternatively, the memory 806 stores executable codes, and the processor 804 executes the executable codes to respectively implement the functions of the aforementioned access points, thereby implementing any of the data storage methods. That is, the memory 806 stores instructions for executing any of the data storage methods.
通信接口808使用例如但不限于网络接口卡、收发器一类的收发模块，来实现计算设备800与其他设备或通信网络之间的通信。The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or communication networks.
请参阅图9，图9为本申请实施例提供的计算设备集群的一种结构示意图。Please refer to FIG. 9 , which is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application.
该计算设备集群包括至少一台计算设备。该计算设备可以是服务器，例如是中心服务器、边缘服务器，或者是本地数据中心中的本地服务器。在一些实施例中，计算设备也可以是台式机、笔记本电脑或者智能手机等终端设备。The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
如图9所示，所述计算设备集群包括至少一个数据存储的设备800。计算设备集群中的一个或多个数据存储的设备800中的存储器806中可以存有相同的用于执行数据存储的方法中任一项方法的指令。As shown in Fig. 9, the computing device cluster includes at least one data storage device 800. The memory 806 in one or more data storage devices 800 in the computing device cluster may store the same instructions for executing any one of the data storage methods.
在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备800的存储器806中也可以分别存有用于执行数据存储的方法中的任一项方法的部分指令。换言之，一个或多个数据存储的设备800的组合可以共同执行用于执行数据存储的方法中任一项方法的指令。In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing any one of the data storage methods. In other words, the combination of one or more data storage devices 800 may jointly execute instructions for executing any one of the data storage methods.
需要说明的是，计算设备集群中的不同的数据存储的设备800中的存储器806可以存储不同的指令，分别用于执行云管理平台的部分功能。也即，不同的数据存储的设备800中的存储器806存储的指令可以实现获取单元701、生成单元702、读写单元703、删除单元704和选取单元705中的一个或多个单元的功能。It should be noted that the memory 806 in different data storage devices 800 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the cloud management platform. That is, the instructions stored in the memory 806 in different data storage devices 800 can implement the functions of one or more units of the acquisition unit 701, the generation unit 702, the read-write unit 703, the deletion unit 704 and the selection unit 705.
在一些可能的实现方式中，计算设备集群中的一个或多个数据存储的设备可以通过网络连接。其中，所述网络可以是广域网或局域网等等。In some possible implementations, one or more data storage devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.
请参阅图10，图10为本申请实施例提供的计算设备集群的另一种结构示意图。Please refer to FIG. 10 , which is another schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application.
如图10所示，两个数据存储的设备800A和800B之间通过网络进行连接。具体地，通过各个计算设备中的通信接口与所述网络进行连接。在这一类可能的实现方式中，数据存储的设备800A中的存储器806中存有执行生成单元702、读写单元703和删除单元704的功能的指令。同时，数据存储的设备800B中的存储器806中存有执行获取单元701和选取单元705的功能的指令。As shown in FIG10 , two data storage devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the data storage device 800A stores instructions for executing the functions of the generation unit 702, the read-write unit 703, and the deletion unit 704. At the same time, the memory 806 in the data storage device 800B stores instructions for executing the functions of the acquisition unit 701 and the selection unit 705.
应理解，图10中示出的数据存储的设备800A的功能也可以由多个数据存储的设备800完成。同样，数据存储的设备800B的功能也可以由多个数据存储的设备800完成。It should be understood that the function of the data storage device 800A shown in Fig. 10 may also be completed by multiple data storage devices 800. Similarly, the function of the data storage device 800B may also be completed by multiple data storage devices 800.
本申请实施例还提供了另一种计算设备集群。该计算设备集群中各计算设备之间的连接关系可以类似的参考图9和图10所述计算设备集群的连接方式。不同的是，该计算设备集群中的一个或多个数据存储的设备800中的存储器806中可以存有相同的用于执行数据存储的方法中任一项方法的指令。The embodiment of the present application also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection mode of the computing device cluster described in Figures 9 and 10. The difference is that the memory 806 in one or more data storage devices 800 in the computing device cluster can store the same instructions for executing any method of the data storage method.
在一些可能的实现方式中，该计算设备集群中的一个或多个数据存储的设备800的存储器806中也可以分别存有用于执行数据存储的方法中任一项方法的部分指令。换言之，一个或多个数据存储的设备800的组合可以共同执行用于执行数据存储的方法中任一项方法的指令。In some possible implementations, the memory 806 of one or more data storage devices 800 in the computing device cluster may also store partial instructions for executing any of the data storage methods. In other words, the combination of one or more data storage devices 800 may jointly execute instructions for executing any of the data storage methods.
需要说明的是，计算设备集群中的不同的数据存储的设备800中的存储器806可以存储不同的指令，用于执行数据存储的方法的部分功能。It should be noted that the memory 806 in different data storage devices 800 in the computing device cluster may store different instructions for executing partial functions of the data storage method.
本申请实施例还提供一种计算机可读存储介质，包括计算机可读指令，当计算机可读指令在计算机上运行时，使得计算机执行如前述方法实施例所示任一项实现方式。An embodiment of the present application also provides a computer-readable storage medium, including computer-readable instructions. When the computer-readable instructions are executed on a computer, the computer executes any one of the implementation methods shown in the aforementioned method embodiments.
本申请实施例还提供的一种计算机程序产品，计算机程序产品包括计算机程序或指令，当计算机程序或指令在计算机上运行时，使得计算机执行如前述方法实施例所示任一项实现方式。The embodiments of the present application also provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer executes any one of the implementation methods shown in the aforementioned method embodiments.
本申请还提供一种芯片或芯片系统，该芯片可包括处理器。该芯片还可包括存储器(或存储模块)和/或收发器(或通信模块)，或者，该芯片与存储器(或存储模块)和/或收发器(或通信模块)耦合，其中，收发器(或通信模块)可用于支持该芯片进行有线和/或无线通信，存储器(或存储模块)可用于存储程序或一组指令，该处理器调用该程序或该组指令可用于实现上述方法实施例、方法实施例的任意一种可能的实现方式中由终端或者网络设备执行的操作。该芯片系统可包括以上芯片，也可以包含上述芯片和其他分立器件，如存储器(或存储模块)和/或收发器(或通信模块)。The present application also provides a chip or chip system, which may include a processor. The chip may also include a memory (or storage module) and/or a transceiver (or communication module), or the chip is coupled to a memory (or storage module) and/or a transceiver (or communication module), wherein the transceiver (or communication module) can be used to support the chip for wired and/or wireless communication, and the memory (or storage module) can be used to store a program or a set of instructions, and the processor calls the program or the set of instructions to implement the above method embodiment, and the operation performed by the terminal or network device in any possible implementation of the method embodiment. The chip system may include the above chip, and may also include the above chip and other discrete devices, such as a memory (or storage module) and/or a transceiver (or communication module).
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，read-onlymemory)、随机存取存储器(RAM，randomaccessmemory)、磁碟或者光盘等各种可以存储程序代码的介质。If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310152784.9ACN118567546A (en) | 2023-02-22 | 2023-02-22 | Method, device and computer readable storage medium for data storage |
| PCT/CN2024/077499WO2024174951A1 (en) | 2023-02-22 | 2024-02-19 | Data storage method, and device and computer-readable storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310152784.9ACN118567546A (en) | 2023-02-22 | 2023-02-22 | Method, device and computer readable storage medium for data storage |
| Publication Number | Publication Date |
|---|---|
| CN118567546Atrue CN118567546A (en) | 2024-08-30 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310152784.9APendingCN118567546A (en) | 2023-02-22 | 2023-02-22 | Method, device and computer readable storage medium for data storage |
| Country | Link |
|---|---|
| CN (1) | CN118567546A (en) |
| WO (1) | WO2024174951A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255962B (en)* | 2011-07-01 | 2013-11-06 | 华为数字技术（成都）有限公司 | Distributive storage method, device and system |
| CN110187970A (en)* | 2019-05-30 | 2019-08-30 | 北京理工大学 | A Distributed Big Data Parallel Computing Method Based on Hadoop MapReduce |
| EP4087212B1 (en)* | 2020-01-17 | 2025-08-27 | Huawei Technologies Co., Ltd. | Method and apparatus for cloning file system |
| CN111258978B (en)* | 2020-01-17 | 2024-04-19 | 广东小天才科技有限公司 | Data storage method |
| Publication number | Publication date |
|---|---|
| WO2024174951A1 (en) | 2024-08-29 |
| Publication | Publication Date | Title |
|---|---|---|
| JP5510556B2 (en) | Method and system for managing virtual machine storage space and physical hosts | |
| CN110704161B (en) | Virtual machine creation method and device and computer equipment | |
| US12307110B2 (en) | Resource allocation method, storage device, and storage system | |
| CN112000463B (en) | A CUDA-based GPU resource allocation method, system, terminal and storage medium | |
| CN111338779B (en) | Resource allocation method, device, computer equipment and storage medium | |
| CN113254222B (en) | Task allocation method and system for solid state disk, electronic device and storage medium | |
| WO2025161872A1 (en) | Network interface card configuration method and apparatus, and electronic device and storage medium | |
| CN113986846A (en) | Data processing method, system, device and storage medium | |
| CN115129621A (en) | Memory management method, device, medium and memory management module | |
| CN115794296A (en) | Link cloning method, system, equipment and storage medium based on hardware unloading | |
| CN118567546A (en) | Method, device and computer readable storage medium for data storage | |
| WO2024165064A1 (en) | Cloud phone hot starting method and cloud phone management platform | |
| WO2024125114A1 (en) | Resource allocation method and apparatus based on cloud service | |
| CN117891575A (en) | Task scheduling method, device, equipment and storage medium | |
| CN114722261B (en) | Resource processing method and device, electronic equipment and storage medium | |
| CN115114309A (en) | Data merging processing method, device, electronic device and storage medium | |
| CN114675995A (en) | Data backup method, device and electronic device | |
| CN114077493A (en) | Resource allocation method and related equipment | |
| CN112395076A (en) | Network data processing method, equipment and storage medium | |
| CN111930781B (en) | Method and device for processing data request of cache database | |
| CN120407201B (en) | Memory resource allocation method and device, electronic equipment, medium and product | |
| JPH06348666A (en) | Program execution load distribution method in computer system | |
| CN111399763B (en) | Method, device, system, equipment and storage medium for data storage | |
| CN119883595A (en) | Operation execution method and device | |
| CN119105903A (en) | Method, device, system and storage medium for updating data |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |