技术领域Technical Field
本公开涉及数据处理领域，尤其涉及一种数据处理方法、装置、电子设备及存储介质。The present disclosure relates to the field of data processing, and in particular to a data processing method, device, electronic device and storage medium.
背景技术Background Art
为了解决主机迁移过程中碎片文件传输效率低下的问题，源端迁移服务在传输前将多个迁移文件（也可以称为迁移数据）压缩成一个大文件，目标端迁移服务则在传输完成后通过解压方式还原源端文件结构。In order to solve the problem of inefficient transmission of fragmented files during host migration, the source-side migration service compresses multiple migration files (also called migration data) into a large file before transmission. The target-side migration service restores the source-side file structure by decompressing it after the transmission is completed.
然而，相关技术中的迁移方法迁移效率较低。However, the migration efficiency of the migration method in the related art is low.
发明内容Summary of the invention
本公开提供一种数据处理方法、装置、电子设备及存储介质，以解决相关技术中的问题。The present disclosure provides a data processing method, device, electronic device and storage medium to solve the problems in the related art.
本公开的第一方面实施例提出了一种数据处理方法，该方法包括：A first aspect of the present disclosure provides a data processing method, the method comprising:
基于发送端设备的负载波动信息，确定数据分割阈值；Determine a data segmentation threshold based on load fluctuation information of the sending end device;
基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型；Determine the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed;
利用所述数据类型对应的压缩策略，压缩所述待处理数据。The data to be processed is compressed using a compression strategy corresponding to the data type.
上述方案中，所述方法还包括：In the above solution, the method further comprises:
周期性获取所述发送端设备的运行信息和服务信息；所述运行信息包括所述发送端设备运行环境相关信息，所述服务信息包括所述发送端设备处理任务的信息；Periodically acquiring the operation information and service information of the sending end device; the operation information includes information related to the operation environment of the sending end device, and the service information includes information on the processing tasks of the sending end device;
基于所述运行信息和服务信息，确定所述负载波动信息。The load fluctuation information is determined based on the operation information and the service information.
上述方案中，所述数据分割阈值包括第一阈值和第二阈值；所述基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型，包括以下之一：In the above scheme, the data segmentation threshold includes a first threshold and a second threshold; and determining the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed includes one of the following:
所述待处理数据的数据大小小于或等于第一阈值，确定所述待处理数据为第一类数据；The data size of the data to be processed is less than or equal to a first threshold, and the data to be processed is determined to be first-category data;
所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值，确定所述待处理数据为第二类数据；The data size of the data to be processed is greater than the first threshold, and the data size of the data to be processed is less than or equal to the second threshold, determining that the data to be processed is the second category of data;
所述待处理数据的数据大小大于第二阈值，确定所述待处理数据为第三类数据。The data size of the data to be processed is greater than a second threshold, and it is determined that the data to be processed is third-category data.
上述方案中，所述利用所述数据类型对应的压缩策略，压缩所述待处理数据，包括：In the above solution, compressing the data to be processed by using the compression strategy corresponding to the data type includes:
所述待处理数据为第一类数据，将所述待处理数据存储至数据组；利用所述第一类数据对应的第一压缩策略，压缩所述数据组；The data to be processed is the first type of data, and the data to be processed is stored in a data group; the data group is compressed using a first compression strategy corresponding to the first type of data;
或者，or,
所述待处理数据为第二类数据，利用所述第二类数据对应的第二压缩策略，压缩所述待处理数据。The data to be processed is second-category data, and the data to be processed is compressed using a second compression strategy corresponding to the second-category data.
上述方案中，所述利用所述第一类数据对应的第一压缩策略，压缩所述数据组，包括：In the above solution, compressing the data group using the first compression strategy corresponding to the first type of data includes:
判断所述数据组中的数据大小是否超过第一阈值；Determining whether the data size in the data group exceeds a first threshold;
在所述数据组中的数据大小超过第一阈值的情况下，利用所述第一类数据对应的第一压缩策略，压缩所述数据组。When the data size in the data group exceeds a first threshold, the data group is compressed using a first compression strategy corresponding to the first type of data.
上述方案中，所述方法还包括以下之一：In the above scheme, the method further includes one of the following:
所述待处理数据为第一类数据，发送第一数据，所述第一数据包括标识信息和压缩后的所述数据组；The data to be processed is first type of data, and first data is sent, wherein the first data includes identification information and the compressed data group;
所述待处理数据为第二类数据，发送第二数据，所述第二数据包括标识信息和压缩后的所述待处理数据；The data to be processed is the second type of data, and second data is sent, where the second data includes identification information and the compressed data to be processed;
所述待处理数据为第三类数据，发送第三数据，所述第三数据包括标识信息和所述待处理数据；其中，The data to be processed is the third type of data, and the third data is sent, and the third data includes identification information and the data to be processed; wherein,
所述标识信息用于标识发送数据的数据类型。The identification information is used to identify the data type of the transmitted data.
上述方案中，所述待处理数据为第二类数据，所述方法还包括：In the above solution, the data to be processed is the second type of data, and the method further includes:
基于所述待处理数据的数据大小与所述第二阈值的差值，更新所述服务信息。The service information is updated based on the difference between the data size of the to-be-processed data and the second threshold.
本公开的第二方面实施例提出了一种数据处理装置，该装置包括：A second aspect of the present disclosure provides a data processing device, the device comprising:
确定单元，用于基于发送端设备的负载波动信息，确定数据分割阈值；以及，基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型；A determination unit, configured to determine a data segmentation threshold based on load fluctuation information of a transmitting end device; and determine a data type of the data to be processed based on the data segmentation threshold and a data size of the data to be processed;
处理单元，用于利用所述数据类型对应的压缩策略，压缩所述待处理数据。The processing unit is used to compress the data to be processed by using the compression strategy corresponding to the data type.
本公开的第三方面实施例提出了一种电子设备，包括：至少一个处理器；以及与至少一个处理器通信连接的存储器；其中，存储器存储有可被至少一个处理器执行的指令，指令被至少一个处理器执行，以使至少一个处理器能够执行本公开第一方面实施例中描述的方法。The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.
本公开的第四方面实施例提出了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，计算机指令用于使计算机执行本公开第一方面实施例中描述的方法。The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
本公开的第五方面实施例提出了一种芯片，该芯片包括一个或多个接口电路和一个或多个处理器；接口电路用于从电子设备的存储器接收信号，并向处理器发送信号，信号包括存储器中存储的计算机指令，当处理器执行计算机指令时，使得电子设备执行本公开第一方面实施例中描述的方法。The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.
综上，本公开提出的数据处理方法、装置、电子设备及存储介质，基于发送端设备的负载波动信息，确定数据分割阈值；基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型；利用所述数据类型对应的压缩策略，压缩所述待处理数据。本公开实施例提供的技术方案，根据发送端设备的负载波动情况，动态调整数据分割阈值，提高了数据类型划分结果的准确性，在此基础上，在利用不同压缩策略处理不同类型数据时，能够提高压缩策略与待处理数据的适配度，从而能够提高压缩效率，进而提升整体迁移效率。In summary, the data processing method, device, electronic device and storage medium proposed in the present disclosure determine the data segmentation threshold based on the load fluctuation information of the sending end device; determine the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed; and compress the data to be processed using the compression strategy corresponding to the data type. The technical solution provided by the embodiment of the present disclosure dynamically adjusts the data segmentation threshold according to the load fluctuation of the sending end device, thereby improving the accuracy of the data type division result. On this basis, when different compression strategies are used to process different types of data, the compatibility of the compression strategy with the data to be processed can be improved, thereby improving the compression efficiency and further improving the overall migration efficiency.
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理，并不构成对本公开的不当限定。The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
图1为相关技术中主机迁移方法流程示意图；FIG1 is a schematic diagram of a host migration method in the related art;
图2为本公开实施例提供的一种数据处理方法流程示意图；FIG2 is a flow chart of a data processing method provided by an embodiment of the present disclosure;
图3为本公开实施例提供的另一种数据处理方法流程示意图；FIG3 is a schematic diagram of another data processing method provided by an embodiment of the present disclosure;
图4为本公开应用实施例提供的主机迁移自适应压缩系统架构图；FIG4 is an architecture diagram of a host migration adaptive compression system provided by an application embodiment of the present disclosure;
图5为本公开应用实施例提供的自适应压缩方法流程示意图；FIG5 is a schematic diagram of a flow chart of an adaptive compression method provided by an application embodiment of the present disclosure;
图6为本公开实施例提供的数据处理装置结构示意图；FIG6 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;
图7为本公开实施例提供的电子设备结构示意图；FIG7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;
图8为本公开实施例提供的芯片结构示意图。FIG8 is a schematic diagram of a chip structure provided in an embodiment of the present disclosure.
具体实施方式DETAILED DESCRIPTION
下面详细描述本公开的实施例，实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本公开，而不能理解为对本公开的限制。Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
主机迁移是一种将源端服务器上的应用、数据迁移至目标端服务器的技术实现。相关技术中，如图1所示，主机迁移的流程可以包括三个步骤：Host migration is a technical implementation that migrates applications and data from a source server to a target server. In related technologies, as shown in Figure 1, the process of host migration can include three steps:
步骤1：迁移平台（或客户端）创建迁移任务，迁移任务的配置项包括传输方式、迁移带宽等。Step 1: The migration platform (or client) creates a migration task. The configuration items of the migration task include transmission mode, migration bandwidth, etc.
步骤2：源端服务器中的迁移服务从迁移平台拉取迁移任务，并执行迁移任务，如图1中步骤2.1和2.2所示。Step 2: The migration service in the source server pulls the migration task from the migration platform and executes the migration task, as shown in steps 2.1 and 2.2 in Figure 1.
步骤3：目标端服务器的迁移服务验证迁移数据。Step 3: The migration service on the target server verifies the migration data.
为了解决碎片文件传输效率低下问题，迁移任务通常配置为压缩传输的方式，即配置压缩策略，并利用压缩策略将多个迁移文件压缩成一个较大的文件进行传输，以提升带宽利用率，加速迁移进度。To solve the problem of low efficiency in transferring fragmented files, migration tasks are usually configured to use compressed transmission. That is, a compression policy is configured and used to compress multiple migration files into a larger file for transmission, thereby improving bandwidth utilization and accelerating migration progress.
然而，相关技术中，源端所有迁移文件均采用相同的压缩策略，由于不同文件适配的压缩条件不同，比如，大小不同的数据文件，对于压缩算法的压缩率、压缩速度、内存占用率的要求也不同，因此，同一种压缩策略与不同类型文件的适配度也不同，无法适应源端服务器中复杂的迁移场景，影响压缩效率和迁移效率；同时，相关技术中的压缩策略没有考虑到源端服务器负载的波动，与源端服务器的运行环境适配度不高，因此，容易出现负载不均衡，影响迁移效率，且在出现负载不均衡时，只能依赖于人工调研业务特性和分析源端文件，以进行参数调优，不仅效率低下，且会增加人力成本。However, in the related art, all migration files on the source side adopt the same compression strategy. Since different files are adapted to different compression conditions, for example, data files of different sizes have different requirements for the compression rate, compression speed, and memory usage of the compression algorithm. Therefore, the adaptability of the same compression strategy to different types of files is also different, and it cannot adapt to the complex migration scenarios in the source server, affecting the compression efficiency and migration efficiency. At the same time, the compression strategy in the related art does not take into account the fluctuation of the source server load, and is not highly adapted to the operating environment of the source server. Therefore, load imbalance is prone to occur, affecting the migration efficiency. When load imbalance occurs, it can only rely on manual research on business characteristics and analysis of source files to perform parameter tuning, which is not only inefficient but also increases labor costs.
为了解决相关技术中存在的问题，本公开提出一种数据处理方法、装置、电子设备及存储介质，根据发送端设备的负载波动情况，动态调整数据分割阈值，提高了数据类型划分结果的准确性，在此基础上，在利用不同压缩策略处理不同类型数据时，能够提高压缩策略与待处理数据的适配度，从而能够提高压缩效率，进而提升整体迁移效率。In order to solve the problems existing in the related technology, the present disclosure proposes a data processing method, device, electronic device and storage medium, which dynamically adjust the data segmentation threshold according to the load fluctuation of the sending device, thereby improving the accuracy of the data type division results. On this basis, when using different compression strategies to process different types of data, the adaptability of the compression strategy to the data to be processed can be improved, thereby improving the compression efficiency and then improving the overall migration efficiency.
图2提供了一种数据处理方法流程示意图。如图2所示，该方法可以包括：Figure 2 provides a schematic flow chart of a data processing method. As shown in Figure 2, the method may include:
步骤201：基于发送端设备的负载波动信息，确定数据分割阈值。Step 201: Determine a data segmentation threshold based on load fluctuation information of a transmitting end device.
实际应用时，所述负载波动信息能够可以表征所述发送端设备的负载环境波动情况，具体地，可以表征所述发送端设备的运行环境波动情况和任务处理波动情况。In actual application, the load fluctuation information can represent the load environment fluctuation of the sending end device, and specifically, can represent the operating environment fluctuation and task processing fluctuation of the sending end device.
实际应用时，可以通过获取源端文件列表，获知所述发送端设备的运行环境波动情况和任务处理波动情况，从而确定所述负载波动信息。In actual application, the load fluctuation information can be determined by obtaining the source file list to learn the operating environment fluctuation and task processing fluctuation of the sending end device.
实际应用时，所述发送端设备可以是服务器，具体地，可以是源端服务器。In actual application, the sending end device may be a server, specifically, a source end server.
步骤202：基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型。Step 202: Determine the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed.
实际应用时，所述数据分割阈值的数量可以是至少一个。In practical applications, the number of the data segmentation thresholds may be at least one.
实际应用时，可以将所述待处理数据的数据大小与所述数据分割阈值进行比较，并根据比较结果，确定所述待处理数据的类型。In actual application, the data size of the data to be processed may be compared with the data segmentation threshold, and the type of the data to be processed may be determined according to the comparison result.
在一些实施例中，所述数据分割阈值可以包括第一阈值和第二阈值，在所述待处理数据的数据大小小于或等于第一阈值时，确定所述待处理数据为第一类数据，在所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值时，确定所述待处理数据为第二类数据，在所述待处理数据的数据大小大于第二阈值时，确定所述待处理数据为第三类数据。In some embodiments, the data segmentation threshold may include a first threshold and a second threshold. When the data size of the data to be processed is less than or equal to the first threshold, the data to be processed is determined to be first-category data. When the data size of the data to be processed is greater than the first threshold and the data size of the data to be processed is less than or equal to the second threshold, the data to be processed is determined to be second-category data. When the data size of the data to be processed is greater than the second threshold, the data to be processed is determined to be third-category data.
步骤203：利用所述数据类型对应的压缩策略，压缩所述待处理数据。Step 203: compress the data to be processed using a compression strategy corresponding to the data type.
实际应用时，所述第一类数据为较小的数据，所述第二数据为较大的数据，可以根据这两类数据的特性，为所述第一类数据和所述第二类数据配置不同的压缩策略；而所述第三类数据为极大的数据，可以采用直传的方式，即，在主机迁移过程中直接传输第三类数据，而无需对其进行压缩。In actual application, the first category of data is smaller data, and the second category of data is larger data. Different compression strategies can be configured for the first category of data and the second category of data according to the characteristics of the two categories of data; and the third category of data is extremely large data, and a direct transmission method can be adopted, that is, the third category of data is directly transmitted during the host migration process without compressing it.
在一些实施例中，还可以发送压缩后的所述待处理数据，具体地，可以向接收端设备发送压缩后的所述待处理数据；所述待处理数据，也可以称为待处理文件。In some embodiments, the compressed data to be processed may also be sent. Specifically, the compressed data to be processed may be sent to a receiving device. The data to be processed may also be referred to as a file to be processed.
实际应用时，所述接收端设备可以是服务器，具体地，可以是目标端服务器。In actual application, the receiving end device may be a server, specifically, a target end server.
综上，本公开实施例提供的数据处理方法，根据发送端设备的负载波动情况，动态调整数据分割阈值，提高了数据类型划分结果的准确性，在此基础上，在利用不同压缩策略处理不同类型数据时，能够提高压缩策略与待处理数据的适配度，从而能够提高压缩效率，进而提升整体迁移效率。In summary, the data processing method provided by the embodiment of the present disclosure dynamically adjusts the data segmentation threshold according to the load fluctuation of the sending end device, thereby improving the accuracy of the data type division results. On this basis, when using different compression strategies to process different types of data, the adaptability of the compression strategy to the data to be processed can be improved, thereby improving the compression efficiency and further improving the overall migration efficiency.
基于图2所示的实施例，图3进一步示出本公开实施例提供的一种数据处理方法的流程示意图。如图3所示，该方法可以包括：Based on the embodiment shown in FIG2 , FIG3 further shows a flow chart of a data processing method provided by an embodiment of the present disclosure. As shown in FIG3 , the method may include:
步骤301：周期性获取所述发送端设备的运行信息和服务信息；所述运行信息包括所述发送端设备运行环境相关信息，所述服务信息包括所述发送端设备处理任务的信息。Step 301: periodically obtain the operation information and service information of the sending end device; the operation information includes information related to the operating environment of the sending end device, and the service information includes information on the processing tasks of the sending end device.
实际应用时，所述发送端设备可以是服务器，具体地，可以是源端服务器。In actual application, the sending end device may be a server, specifically, a source end server.
实际应用时，可以根据预设周期获取所述发送端设备的运行信息和服务信息；所述预设周期可以根据所述发送端设备的实际应用环境进行确定，本公开实施例对此不作限定。In actual application, the operation information and service information of the sending end device can be obtained according to a preset period; the preset period can be determined according to the actual application environment of the sending end device, and the embodiment of the present disclosure does not limit this.
在一些实施例中，所述预设周期可以是5min。In some embodiments, the preset period may be 5 minutes.
实际应用时，所述运行信息可以包括与所述发送端设备运行环境相关的信息，用于体现所述发送端设备运行环境的波动情况。In actual application, the operation information may include information related to the operation environment of the sending end device, which is used to reflect the fluctuation of the operation environment of the sending end device.
在一些实施例中，所述运行信息可以包括以下至少之一：In some embodiments, the operation information may include at least one of the following:
发送端设备中央处理器（Central Processing Unit，CPU）使用率；The CPU usage of the sending device;
发送端设备随机存取存储器（Random Access Memory，RAM）使用率；The sending device's random access memory (RAM) usage;
发送端设备带宽；Bandwidth of the sending device;
发送端设备操作系统信息；Operating system information of the sending device;
发送端设备磁盘类型信息。Disk type information of the sending device.
实际应用时，可以将针对不同操作系统配置的不同数值作为所述操作系统信息，比如，对于碎片数据较多的Windows系统，对应的操作系统信息配置为1，而对于碎片数据较少的非Windows系统（比如UNIX），对应的操作系统信息配置为2；所述操作系统信息也可以称为操作系统数值，本公开实施例对此不作限定。In actual application, different numerical values configured for different operating systems can be used as the operating system information. For example, for a Windows system with more fragmented data, the corresponding operating system information is configured as 1, and for a non-Windows system with less fragmented data (such as UNIX), the corresponding operating system information is configured as 2; the operating system information can also be called an operating system value, which is not limited to the embodiments of the present disclosure.
实际应用时，可以将针对不同磁盘类型配置的不同数值作为所述磁盘类型信息，比如，对于碎片数据较多的系统盘，对应的磁盘类型信息配置为1，而对于碎片数据较少的非系统盘，对应的磁盘类型信息配置为2；所述磁盘类型信息也可以称为磁盘类型数值，本公开实施例对此不作限定。In actual application, different numerical values configured for different disk types can be used as the disk type information. For example, for a system disk with more fragmented data, the corresponding disk type information is configured as 1, and for a non-system disk with less fragmented data, the corresponding disk type information is configured as 2; the disk type information can also be called a disk type value, which is not limited to the embodiments of the present disclosure.
实际应用时，所述服务信息可以包括与所述发送端设备处理任务相关的信息，用于体现所述发送端设备的处理能力波动情况。In actual application, the service information may include information related to the processing task of the sending end device, which is used to reflect the fluctuation of the processing capacity of the sending end device.
在一些实施例中，所述服务信息的初始值可以配置为0，在所述发送端设备处理任务过程中，可以根据所处理任务的大小确定所述服务信息；比如，在所述发送端设备当前正在处理较大的文件时，可以根据当前处理文件的大小与预设阈值的之间的差值确定所述服务信息；所述服务信息也可以称为服务权重，还可以称为源端服务权重，本公开实施例对此不作限定，只要能实现其功能即可。In some embodiments, the initial value of the service information can be configured as 0. During the task processing process of the sending device, the service information can be determined according to the size of the task being processed; for example, when the sending device is currently processing a larger file, the service information can be determined based on the difference between the size of the currently processed file and a preset threshold value; the service information can also be called a service weight, or a source-end service weight. The embodiments of the present disclosure do not limit this, as long as its function can be achieved.
步骤302：基于所述运行信息和服务信息，确定负载波动信息。Step 302: Determine load fluctuation information based on the operation information and service information.
实际应用时，可以先根据采集的运行信息中具有波动特性的信息（比如CPU使用率、RAM使用率），确定运行环境的波动情况；具体地，可以根据所述CPU使用率与CPU使用率阈值之间的第一差异，以及RAM使用率与RAM使用率阈值之间的第二差异，确定环境权重，即所述运行信息，以通过所述环境权重体现运行环境的波动情况；具体地，可以对第一差异和第二差异进行加权求和，确定所述环境权重。In actual application, the fluctuation of the operating environment can be determined based on the information with fluctuation characteristics in the collected operating information (such as CPU usage, RAM usage); specifically, the environmental weight, that is, the operating information, can be determined based on the first difference between the CPU usage and the CPU usage threshold, and the second difference between the RAM usage and the RAM usage threshold, so as to reflect the fluctuation of the operating environment through the environmental weight; specifically, the first difference and the second difference can be weighted and summed to determine the environmental weight.
示例性地，源端服务器环境权重可以通过以下公式确定：Exemplarily, the source server environment weight may be determined by the following formula:
；（1） ; (1)
其中，表示源端服务器环境权重，表示预设的源端服务器CPU偏转值，即第一差异在加权求和中的权重，本公开示例中，为0.5；表示CPU使用率阈值，本公开示例中，为0.3；表示CPU使用率；表示RAM偏转值，即第二差异在加权求和中的权重，本公开示例中，为0.5；表示RAM使用率阈值，本公开示例中，为0.5；表示RAM使用率。in, Indicates the source server environment weight. represents the preset source server CPU deflection value, that is, the weight of the first difference in the weighted sum. In the example disclosed in this disclosure, is 0.5; Indicates the CPU usage threshold. In this example, is 0.3; Indicates CPU usage; represents the RAM deflection value, that is, the weight of the second difference in the weighted summation. In the example disclosed in the present disclosure, is 0.5; Indicates the RAM usage threshold. In this example, is 0.5; Indicates RAM usage.
实际应用时，所述服务信息可以包括服务权重；在确定所述环境权重和所述服务权重后，根据所述环境权重、所述服务权重，以及所述运行信息中能够体现文件分布特性的信息（比如操作系统、磁盘类型），确定所述负载波动信息；所述负载波动信息也可以称为负载信息，还可以称为负载权重，本公开实施例对此不作限定，只要能实现其功能即可。In actual application, the service information may include a service weight; after determining the environment weight and the service weight, the load fluctuation information is determined according to the environment weight, the service weight, and the information in the operation information that can reflect the file distribution characteristics (such as operating system, disk type); the load fluctuation information may also be called load information, and may also be called load weight. The embodiment of the present disclosure does not limit this, as long as its function can be achieved.
示例性地，源端服务器负载权重可以通过以下公式确定：Exemplarily, the source server load weight may be determined by the following formula:
；（2） ; (2)
其中，表示负载权重，即负载波动信息；表示操作系统数值，即操作系统信息；表示磁盘类型数值，即磁盘类型信息；表示环境权重；表示服务权重；表示源端服务偏转值，本公开示例中，为0.5。in, Indicates load weight, i.e. load fluctuation information; Represents the operating system value, that is, the operating system information; Indicates the disk type value, that is, the disk type information; represents the environmental weight; Indicates the service weight; Indicates the source service deflection value. In this example, is 0.5.
需要说明的是，步骤301和步骤302为周期性确定负载波动信息的流程，由于负载波动信息能够体现所述发送端设备的运行环境和处理能力波动情况，因此，通过周期性确定的负载波动信息，能够准确获取所述发送端设备的负载波动情况，从而为后续数据分割阈值的确定提供更为准确的指导。It should be noted that step 301 and step 302 are the processes for periodically determining load fluctuation information. Since the load fluctuation information can reflect the operating environment and processing capacity fluctuations of the sending end device, the load fluctuation of the sending end device can be accurately obtained through the periodically determined load fluctuation information, thereby providing more accurate guidance for the subsequent determination of the data segmentation threshold.
步骤303：基于所述负载波动信息，确定数据分割阈值。Step 303: Determine a data segmentation threshold based on the load fluctuation information.
实际应用时，可以根据待迁移数据的特性，将待迁移的数据划分为三类数据；具体地，这三类型数据可以包括：In actual application, the data to be migrated can be divided into three types of data according to the characteristics of the data to be migrated; specifically, the three types of data may include:
第一类数据，表示较小的数据，也可以称为小文件；小文件大量存在于发送端设备系统盘中，尤其是微软窗口（Microsoft Windows）系统的系统盘中；小文件迁移场景中，小文件对服务器产生的影响较小，其他影响因素消耗时间基本一致，小文件迁移总时间与文件数量成正相关；The first type of data refers to relatively small data, which can also be called small files. A large number of small files exist in the system disk of the sending device, especially in the system disk of the Microsoft Windows system. In the small file migration scenario, small files have little impact on the server, and the time consumed by other influencing factors is basically the same. The total time of small file migration is positively correlated with the number of files.
第二类数据，表示较大的数据，也可以称为大文件；大文件数量较少，主要包括发送端设备数据盘中的业务数据；大文件迁移场景中，大文件迁移总时间与文件总大小成正比，压缩过程会对服务器产生一定影响；The second type of data refers to larger data, which can also be called large files. There are fewer large files, mainly including business data in the data disk of the sending device. In the large file migration scenario, the total migration time of large files is proportional to the total file size, and the compression process will have a certain impact on the server.
第三类数据，表示极大的数据，也可以称为其他文件；其他文件的文件大小极大或数据本身已压缩；其中，压缩或解压极大文件的行为会对服务器产生较大影响，甚至影响源端业务的正常运行；而对于已压缩文件，二次解压缩不能有效增快迁移进程，反而造成无效的时空浪费。The third type of data refers to extremely large data, which can also be called other files. The file size of other files is extremely large or the data itself has been compressed. Among them, the behavior of compressing or decompressing extremely large files will have a greater impact on the server and even affect the normal operation of the source-side business. For compressed files, secondary decompression cannot effectively speed up the migration process, but instead causes invalid waste of time and space.
在此基础上，可以根据确定的负载波动信息，确定多个数据分割阈值，从而利用多个数据分割阈值对不同类型的数据进行分类；由于负载波动信息根据所述发送端设备的负载变化而变化，因此，能够实现在所述发送端设备的不同负载状态下，动态调整数据分割阈值，从而实现对待迁移的数据的动态分类。On this basis, multiple data segmentation thresholds can be determined according to the determined load fluctuation information, so that different types of data can be classified using multiple data segmentation thresholds; since the load fluctuation information changes according to the load change of the sending end device, it is possible to dynamically adjust the data segmentation threshold under different load conditions of the sending end device, thereby realizing dynamic classification of the data to be migrated.
基于此，在一些实施例中，所述数据分割阈值包括第一阈值和第二阈值。Based on this, in some embodiments, the data segmentation threshold includes a first threshold and a second threshold.
实际应用时，所述发送端设备的带宽及实际负载情况能够反映发送端设备实际用于处理数据的能力，因此，可以基于所述发送端设备带宽和实际负载情况确定所述第一阈值和所述第二阈值。In actual application, the bandwidth and actual load of the sending device can reflect the actual ability of the sending device to process data. Therefore, the first threshold and the second threshold can be determined based on the bandwidth and actual load of the sending device.
基于此，在一些实施例中，所述基于发送端设备的负载波动信息，确定数据分割阈值，可以包括：Based on this, in some embodiments, determining the data segmentation threshold based on the load fluctuation information of the transmitting end device may include:
基于所述发送端设备的带宽和所述负载波动信息，确定所述第一阈值和所述第二阈值。The first threshold and the second threshold are determined based on the bandwidth of the transmitting end device and the load fluctuation information.
实际应用时，在所述发送端设备处理任务压力较小时，所述负载波动信息对不同类型的数据处理结果的影响较小，此时，可以根据所述发送端设备的带宽确定所述第一阈值和所述第二阈值；在所述发送端设备处理任务的压力较大时，所述负载波动信息对不同类型文件处理结果的影响较大，此时，可以根据所述发送设备的带宽和所述负载波动信息来确定所述第一阈值和所述第二阈值。In actual application, when the pressure of processing tasks on the sending end device is relatively small, the load fluctuation information has little impact on the processing results of different types of data. At this time, the first threshold and the second threshold can be determined according to the bandwidth of the sending end device; when the pressure of processing tasks on the sending end device is relatively large, the load fluctuation information has a greater impact on the processing results of different types of files. At this time, the first threshold and the second threshold can be determined according to the bandwidth of the sending device and the load fluctuation information.
示例性地，分割阈值可以通过以下公式确定：Exemplarily, the segmentation threshold may be determined by the following formula:
；（3） ; (3)
；（4） ; (4)
其中，表示小文件的分割阈值，即第一类数据的分割阈值，表示大文件的分割阈值，即第二类数据的分割阈值，表示源端服务器的带宽，表示负载权重，即负载波动信息。in, Indicates the segmentation threshold of small files, that is, the segmentation threshold of the first type of data, Indicates the segmentation threshold of large files, that is, the segmentation threshold of the second type of data, Indicates the bandwidth of the source server. Indicates the load weight, that is, load fluctuation information.
步骤304：基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型。Step 304: Determine the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed.
在一些实施例中，所述基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型，可以包括以下之一：In some embodiments, determining the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed may include one of the following:
所述待处理数据的数据大小小于或等于第一阈值，确定所述待处理数据为第一类数据；The data size of the data to be processed is less than or equal to a first threshold, and the data to be processed is determined to be first-category data;
所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值，确定所述待处理数据为第二类数据；The data size of the data to be processed is greater than the first threshold, and the data size of the data to be processed is less than or equal to the second threshold, determining that the data to be processed is the second category of data;
所述待处理数据的数据大小大于第二阈值，确定所述待处理数据为第三类数据。The data size of the data to be processed is greater than a second threshold, and it is determined that the data to be processed is third-category data.
实际应用时，由于第二类数据和第三类数据均为较大的数据，因此，在确定所述待处理数据数据类型的过程中，为了提高确定结果的准确性，可以加入对所述待处理数据类型的判断。In actual application, since the second and third types of data are both relatively large data, in the process of determining the type of data to be processed, in order to improve the accuracy of the determination result, the judgment of the type of data to be processed can be added.
基于此，在一些实施例中，所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值，确定所述待处理数据为第二类数据，可以包括：Based on this, in some embodiments, if the data size of the data to be processed is greater than a first threshold, and the data size of the data to be processed is less than or equal to a second threshold, determining that the data to be processed is the second category of data may include:
所述待处理数据的数据格式不是压缩数据格式，所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值，确定所述待处理数据为第二类数据。The data format of the data to be processed is not a compressed data format, the data size of the data to be processed is greater than a first threshold, and the data size of the data to be processed is less than or equal to a second threshold, and the data to be processed is determined to be second-category data.
实际应用时，所述压缩数据格式可以是压缩数据的格式，比如，zip、rar、7z等格式；所述压缩数据格式也可以称为压缩文件格式，还可以称为压缩格式，本公开实施例对此不作限定，只要能实现其功能即可。In actual application, the compressed data format can be a compressed data format, such as zip, rar, 7z and other formats; the compressed data format can also be called a compressed file format, or a compressed format. The embodiment of the present disclosure does not limit this, as long as its function can be achieved.
步骤305：利用所述数据类型对应的压缩策略，压缩所述待处理数据。Step 305: compress the data to be processed using a compression strategy corresponding to the data type.
实际应用时，由于所述第一类数据对服务器产生的影响较小，所述第二类数据在压缩过程中应尽量消除冗余数据、减少数据传输量，因此，可以针对所述第一类数据配置不同的压缩策略，以适应对应文件类型的特性；对于第三类数据，由于压缩或解压极大文件会对服务器产生较大影响，而对于已压缩文件进行二次解压缩会造成无效的时空浪费，因此，当所述待处理数据为第三类数据时，可以采用直传方式，即直接传输所述待处理数据。In actual application, since the first category of data has a smaller impact on the server, the second category of data should try to eliminate redundant data and reduce data transmission volume during the compression process. Therefore, different compression strategies can be configured for the first category of data to adapt to the characteristics of the corresponding file type; for the third category of data, since compressing or decompressing extremely large files will have a greater impact on the server, and secondary decompression of compressed files will cause invalid waste of time and space, therefore, when the data to be processed is the third category of data, direct transmission can be adopted, that is, directly transmitting the data to be processed.
基于此，在一些实施例中，所述利用所述数据类型对应的压缩策略，压缩所述待处理数据，可以包括：Based on this, in some embodiments, compressing the to-be-processed data by using the compression strategy corresponding to the data type may include:
所述待处理数据为第一类数据，将所述待处理数据存储至数据组；利用所述第一类数据对应的第一压缩策略，压缩所述数据组；The data to be processed is the first type of data, and the data to be processed is stored in a data group; the data group is compressed using a first compression strategy corresponding to the first type of data;
或者，or,
所述待处理数据为第二类数据，利用所述第二类数据对应的第二压缩策略，压缩所述待处理数据。The data to be processed is second-category data, and the data to be processed is compressed using a second compression strategy corresponding to the second-category data.
实际应用时，由于第一类数据对服务器产生的影响较小，因此，可以将多个小文件合并压缩成单个文件后再进行数据传输，提高整体带宽利用率。In actual application, since the first type of data has less impact on the server, multiple small files can be combined and compressed into a single file before data transmission, thereby improving overall bandwidth utilization.
基于此，在一些实施例中，所述利用所述第一类数据对应的第一压缩策略，压缩所述数据组，包括：Based on this, in some embodiments, compressing the data group using the first compression strategy corresponding to the first type of data includes:
判断所述数据组中的数据大小是否超过第一阈值；Determining whether the data size in the data group exceeds a first threshold;
在所述数据组中的数据大小超过第一阈值的情况下，利用所述第一类数据对应的第一压缩策略，压缩所述数据组。When the data size in the data group exceeds a first threshold, the data group is compressed using a first compression strategy corresponding to the first type of data.
实际应用时，所述数据组可以看作是一个数据容器，表示待迁移的小文件的集合，当所述数据组中数据大小未达到第一阈值时，表示所述数据组中数据的大小尚未超过容量阈值，此时，可以不压缩所述数据组，等待下一个文件的存入，避免压缩资源和传输资源的浪费；当所述数据组中数据大小达到第一阈值后，表示所述数据组中的数据大小已超过容量阈值，此时，对所述数据组进行压缩，以提高整体压缩效率和传输效率。In actual application, the data group can be regarded as a data container, which represents a collection of small files to be migrated. When the data size in the data group does not reach the first threshold, it means that the size of the data in the data group has not exceeded the capacity threshold. At this time, the data group may not be compressed and wait for the next file to be stored, so as to avoid the waste of compression resources and transmission resources; when the data size in the data group reaches the first threshold, it means that the data size in the data group has exceeded the capacity threshold. At this time, the data group is compressed to improve the overall compression efficiency and transmission efficiency.
实际应用时，可以通过获取所述发送端设备的文件列表来获取所述待处理数据的属性信息，比如，所述待处理数据的大小，所述待处理数据的数据格式等。In actual application, the attribute information of the data to be processed, such as the size of the data to be processed, the data format of the data to be processed, etc., can be obtained by obtaining the file list of the sending end device.
实际应用时，可以为所述待处理数据配置余量标识信息，所述余量标识信息用于标识所述待处理数据是否为源端文件列表中最后一个文件；在判断所述数据组中的数据大小是否超过第一阈值之前，可以先根据所述余量标识信息确定所述待处理数据是否为所述源端文件列表中最后一个文件，当所述待处理数据为最后一个文件时，可以直接压缩所述数据组，当所述待处理数据不为最后一个文件时，则可以判断所述数据组中的数据大小是否超过第一阈值。In actual application, the data to be processed can be configured with surplus identification information, and the surplus identification information is used to identify whether the data to be processed is the last file in the source file list; before judging whether the data size in the data group exceeds the first threshold, it can be determined whether the data to be processed is the last file in the source file list according to the surplus identification information; when the data to be processed is the last file, the data group can be directly compressed; when the data to be processed is not the last file, it can be judged whether the data size in the data group exceeds the first threshold.
实际应用时，所述数据组也可以称为文件组，还可以称为小文件组，本公开实施例对此不作限定，只要能实现其功能即可。In actual application, the data group may also be called a file group, or a small file group, and the embodiments of the present disclosure do not limit this, as long as the function can be achieved.
实际应用时，可以利用霍夫曼压缩算法、算数压缩算法等对第一类数据进行压缩，即，可以利用霍夫曼压缩算法、算数压缩算法等方法对所述数据组进行压缩；可以利用串表压缩算法（Lempel-Ziv-Welch Encoding，LZW）、LZ77压缩算法等对第二类数据进行压缩；具体采用哪种压缩算法，可以由实际应用需求进行确定，本公开实施例不作限定。In actual application, the first type of data can be compressed using Huffman compression algorithm, arithmetic compression algorithm, etc., that is, the data group can be compressed using Huffman compression algorithm, arithmetic compression algorithm, etc.; the second type of data can be compressed using string table compression algorithm (Lempel-Ziv-Welch Encoding, LZW), LZ77 compression algorithm, etc.; which compression algorithm to use can be determined by actual application requirements, and is not limited in the embodiments of the present disclosure.
实际应用时，由于压缩较大的第二类数据对发送端设备的处理能力影响较大，而处理第二类数据时，第二类数据与第二阈值之间的差值可以反应出发送端设备当前可提供的处理能力，因此，在压缩第二类数据时，可以通过待处理数据的数据大小与第二阈值之间的差值更新所述服务信息，用于知道后续数据处理中数据分割阈值的确定。In actual application, since the compression of the larger second type of data has a greater impact on the processing capability of the sending device, and when processing the second type of data, the difference between the second type of data and the second threshold can reflect the processing capability that the sending device can currently provide, therefore, when compressing the second type of data, the service information can be updated by the difference between the data size of the data to be processed and the second threshold, which is used to know the determination of the data segmentation threshold in subsequent data processing.
基于此，在一些实施例中，所述待处理数据为第二类数据时，所述方法还可以包括：Based on this, in some embodiments, when the data to be processed is the second type of data, the method may further include:
基于所述待处理数据的数据大小与所述第二阈值的差值，更新所述服务信息。The service information is updated based on the difference between the data size of the to-be-processed data and the second threshold.
示例性地，可以通过以下公式更新源端服务器的服务权重，即发送端设备的服务信息：Exemplarily, the service weight of the source server, that is, the service information of the sending device, can be updated by the following formula:
；（5） ; (5)
其中，表示待处理文件的大小，表示大文件分割阈值，即所述第二阈值，表示服务权重，即所述服务信息。in, Indicates the size of the file to be processed. represents the large file segmentation threshold, i.e., the second threshold, Represents the service weight, that is, the service information.
实际应用时， 数据迁移时间主要包括压缩时间、数据传输时间、解压时间和I/O连接时间，其中，所有文件的I/O连接时间基本一致，压缩时间和解压时间主要受服务器处理能力、文件大小、文件类型因素的影响，而数据传输时间取决于迁移服务带宽和文件大小；本公开实施例通过引入服务器负载情况，对所述待处理数据进行类型划分，并针对不同数据类型配置不同的压缩策略，从而能够提高所述待处理数据与应用的压缩策略之间的适配性，提高压缩效率，从而提高迁移效率。In actual application, data migration time mainly includes compression time, data transmission time, decompression time and I/O connection time, among which the I/O connection time of all files is basically the same, the compression time and decompression time are mainly affected by the server processing capacity, file size and file type, and the data transmission time depends on the migration service bandwidth and file size; the embodiment of the present disclosure introduces the server load situation, divides the data to be processed into types, and configures different compression strategies for different data types, so as to improve the adaptability between the data to be processed and the compression strategy of the application, improve the compression efficiency, and thus improve the migration efficiency.
当对所述待处理数据进行相应的处理后，即可进行迁移数据的传输；具体地，对于第一类数据，将压缩后的数据组传输至接收端设备，对于第二类数据，将压缩后的数据传输至接收端设备，对于第三类数据，直接将数据传输至接收端设备；实际应用时，为了使接收端设备获知传输文件的压缩策略，可以在传输的数据中携带用于标识压缩策略的信息。After the data to be processed is processed accordingly, the migration data can be transmitted; specifically, for the first category of data, the compressed data group is transmitted to the receiving device, for the second category of data, the compressed data is transmitted to the receiving device, and for the third category of data, the data is directly transmitted to the receiving device; in actual application, in order to enable the receiving device to know the compression strategy of the transmitted file, information for identifying the compression strategy can be carried in the transmitted data.
基于此，在一些实施例中，所述方法还可以包括以下之一：Based on this, in some embodiments, the method may further include one of the following:
所述待处理数据为第一类数据，发送第一数据，所述第一数据包括标识信息和压缩后的所述数据组；The data to be processed is first type of data, and first data is sent, wherein the first data includes identification information and the compressed data group;
所述待处理数据为第二类数据，发送第二数据，所述第二数据包括标识信息和压缩后的所述待处理数据；The data to be processed is the second type of data, and second data is sent, where the second data includes identification information and the compressed data to be processed;
所述待处理数据为第三类数据，发送第三数据，所述第三数据包括标识信息和所述待处理数据；其中，所述标识信息用于标识发送数据的数据类型。The data to be processed is third-category data, and third data is sent, wherein the third data includes identification information and the data to be processed; wherein the identification information is used to identify the data type of the sent data.
示例性地，将小文件组（即所述数据组）中所有文件压缩成单个迁移文件，并在迁移文件中添加压缩标识，用于指示该文件的类型；其中表示其他文件，即第三类数据，表示小文件，即第一类数据，表示大文件，即第二类数据。Exemplarily, all files in the small file group (ie, the data group) are compressed into a single migration file, and a compression mark is added to the migration file. , used to indicate the type of the file; Indicates other files, i.e. the third type of data, Indicates small files, i.e. the first type of data, Indicates large files, which are the second type of data.
在一些实施例中，所述方法还可以包括：In some embodiments, the method may further include:
所述接收端设备接收所述第一数据、第二数据、和/或第三数据。The receiving end device receives the first data, the second data, and/or the third data.
实际应用时，所述接收端设备接收到第一数据、第二数据或第三数据后，根据对应的标识信息，确定所接收数据的类型，从而可以根据所述第一压缩策略解压所述第一数据，或者，利用所述第二压缩策略解压所述第二数据。In actual application, after the receiving device receives the first data, the second data or the third data, it determines the type of the received data according to the corresponding identification information, so that it can decompress the first data according to the first compression strategy, or decompress the second data using the second compression strategy.
综上，本公开实施例提供的数据处理方法，根据发送端设备的负载波动情况，动态调整数据分割阈值，提高了数据类型划分结果的准确性，在此基础上，在利用不同压缩策略处理不同类型数据时，能够提高压缩策略与待处理数据的适配度，从而能够提高压缩效率，进而提升整体迁移效率。In summary, the data processing method provided by the embodiment of the present disclosure dynamically adjusts the data segmentation threshold according to the load fluctuation of the sending end device, thereby improving the accuracy of the data type division results. On this basis, when using different compression strategies to process different types of data, the adaptability of the compression strategy to the data to be processed can be improved, thereby improving the compression efficiency and further improving the overall migration efficiency.
下面结合具体应用示例对本公开技术方案作进一步详细说明。The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.
本公开应用实施例提供的一种主机迁移自适应压缩系统，所述主机迁移自适应压缩系统可以应用于图4所示的系统架构中。如图4所示，所述自适应压缩系统包括源端负载分析模块、源端文件分析模块和解压缩处理模块。下面对源端负载分析模块、源端文件分析模块和解压缩处理模块进行详细说明。The present application embodiment provides a host migration adaptive compression system, which can be applied to the system architecture shown in Figure 4. As shown in Figure 4, the adaptive compression system includes a source load analysis module, a source file analysis module, and a decompression processing module. The source load analysis module, the source file analysis module, and the decompression processing module are described in detail below.
源端负载分析模块，用于采集源端服务器的运行信息，根据源端运行信息分析源端服务器负载情况，计算源端负载权重值，即确定负载波动信息。The source-side load analysis module is used to collect the operating information of the source-side server, analyze the source-side server load according to the source-side operating information, and calculate the source-side load weight value, that is, to determine the load fluctuation information.
源端文件分析模块，用于基于源端负载权重值动态测算文件分割阈值， 结合文件大小、文件类型等文件信息，为迁移文件分配相应的文件处理区，即基于发送端设备的负载波动信息，确定数据分割阈值，并基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型。The source-side file analysis module is used to dynamically calculate the file segmentation threshold based on the source-side load weight value, and allocate the corresponding file processing area to the migrated file in combination with file information such as file size and file type, that is, to determine the data segmentation threshold based on the load fluctuation information of the sending-side device, and to determine the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed.
解压缩处理模块，用于将迁移文件送入源端文件分析模块判定的文件处理区，执行压缩或解压操作，即利用所述数据类型对应的压缩策略，压缩所述待处理数据。The decompression processing module is used to send the migration file to the file processing area determined by the source-side file analysis module and perform compression or decompression operations, that is, to compress the data to be processed using the compression strategy corresponding to the data type.
基于上述架构图，本公开应用实施例还提供一种自适应压缩方法。如图5所示，该方法包括：Based on the above architecture diagram, the application embodiment of the present disclosure also provides an adaptive compression method. As shown in FIG5 , the method includes:
步骤501：获取源端文件列表，然后执行步骤503。Step 501: Get a source file list, and then execute step 503.
获取源端文件列表具体可以包括以下步骤：Obtaining the source file list may specifically include the following steps:
拉取源端文件列表；根据用户在迁移平台设置的迁移任务，拉取待迁移的源端文件列表，循环顺序对每个文件执行迁移操作，从当前文件的属性信息中获取当前文件大小、当前文件类型；Pull the source file list; according to the migration task set by the user on the migration platform, pull the source file list to be migrated, perform migration operations on each file in a cyclic sequence, and start from the current file Get the current file size from the property information of , Current file type ;
设置余量标识；若当前文件为源端文件列表中最后一个文件，设置当前文件余量标识，否则；Set the margin flag; if the current file Set the current file remaining flag for the last file in the source file list ,otherwise ;
步骤502：更新源端负载权重，然后执行步骤503。Step 502: Update the source load weight, and then execute step 503.
其中，步骤502是单独运行的子程序，每隔分钟运行一次，本公开应用实施例中为5。Among them, step 502 is a subroutine that runs independently, and every The operation is performed once every minute. is 5.
更新源端负载权重具体可以包括以下步骤：Updating the source load weight may specifically include the following steps:
步骤5021：采集源端服务器运行信息；服务器运行信息包括源端服务器CPU使用率、源端服务器RAM使用率、源端服务器带宽、源端服务器操作系统和源端磁盘类型；源端服务器操作系统和源端磁盘类型的数值可以表示为：Step 5021: Collect source server operation information; server operation information includes source server CPU usage , RAM usage of the source server , Source server bandwidth , Source server operating system and the source disk type ; Source server operating system and the source disk type The value of can be expressed as:
； ;
。 .
步骤5022：更新源端环境权重；更新源端环境权重可以通过公式（1）计算得到，其中，是服务器CPU偏转值，取值为0.5；是服务器CPU使用率阈值，取值为0.3；是服务器RAM偏转值，取值为0.5；是服务器RAM使用率阈值，取值为0.5。Step 5022: Update the source environment weight ; Update the source environment weight It can be calculated by formula (1), where: is the server CPU deflection value, which is 0.5; is the server CPU usage threshold, the value is 0.3; is the server RAM deflection value, which is 0.5; is the server RAM usage threshold, which is set to 0.5.
步骤5023：计算源端负载权重；源端负载权重可以通过公式（2）计算得到，其中，是源端服务权重，初始值取0；是源端服务偏转值，取值为0.5。Step 5023: Calculate the source load weight ; Source load weight It can be calculated by formula (2), where: is the source service weight, the initial value is 0; It is the source service deflection value, and its value is 0.5.
步骤503：测算文件分割阈值，然后执行步骤504。Step 503: Calculate the file segmentation threshold, and then execute step 504.
测算文件分割阈值的过程具体可以包括：The process of calculating the file segmentation threshold may specifically include:
步骤5031：计算小文件分割阈值；小文件分割阈值可以利用公式（3）计算得到。Step 5031: Calculate the small file segmentation threshold ; Small file segmentation threshold It can be calculated using formula (3).
步骤5032：计算大文件分割阈值；大文件分割阈值可以利用公式（4）计算得到。Step 5032: Calculate large file segmentation threshold ; Large file segmentation threshold It can be calculated using formula (4).
步骤504：判断是否为小文件；若，则认定当前文件为小文件，执行步骤506，进行小文件处理；若，则执行步骤505，进一步判断文件规格，即数据类型。Step 504: Determine whether it is a small file; if , then the current file is identified If it is a small file, execute step 506 to process the small file; , then execute step 505 to further determine the file specification, that is, the data type.
步骤505：判断是否为大文件；若且，则认定当前文件为大文件，执行步骤507，进行大文件处理；反之，则认定当前文件为其他文件，无需压缩处理，当前文件即为迁移文件，迁移文件压缩标识，直接执行步骤508进行文件传输；其中，所述为压缩文件类型集合，如。Step 505: Determine whether it is a large file; if and , then the current file is identified If the file is a large file, execute step 507 to process the large file; otherwise, the current file is considered For other files, no compression is required. The current file Migration file , migration file compression flag , directly execute step 508 to transfer the file; wherein, A collection of compressed file types, such as .
步骤506：处理小文件。Step 506: Process small files.
处理小文件的过程具体包括：The process of processing small files specifically includes:
步骤5061：判断是否为余量文件；若当前文件余量标识，则当前文件不是余量文件，执行步骤5062；若当前文件余量标识，则当前文件是余量文件，执行步骤5063。Step 5061: Determine whether it is a surplus file; if the current file surplus mark , then the current file If the current file is not a surplus file, execute step 5062; , then the current file If it is a residual file, execute step 5063.
步骤5062：判断是否达到小文件组容量阈值；若，则表示当前未达小文件组容量阈值，执行步骤5064；若，则表示当前已超过小文件组容量阈值，执行步骤5063；其中，所述小文件组是待迁移小文件集合，小文件组容量是集合内所有文件大小之和。Step 5062: Determine whether the small file group has been reached Capacity threshold; if , it means that the current small file group capacity threshold has not been reached, and step 5064 is executed; if , it means that the current capacity threshold of the small file group has been exceeded, and step 5063 is executed; wherein the small file group It is a collection of small files to be migrated. The capacity of the small file group It is the sum of the sizes of all files in the collection.
步骤5063：压缩小文件组，然后执行步骤508。Step 5063: Compress small file groups , then execute step 508.
具体地，使用霍夫曼压缩算法将小文件组中所有文件压缩成单个迁移文件,迁移文件压缩标识；其中，所述霍夫曼压缩算法是一种常见压缩算法，适用于小文件合并压缩场景。Specifically, the Huffman compression algorithm is used to group small files into All files in the zip file are compressed into a single migration file ,Migration file compression flag ; Among them, the Huffman compression algorithm is a common compression algorithm, which is suitable for small file merging and compression scenarios.
步骤5064：更新小文件组，然后执行步骤509。Step 5064: Update the small file group, and then execute step 509.
具体地，更新小文件组，更新小文件组容量，。Specifically, update the small file group , Update the capacity of small file groups , .
步骤507：处理大文件。Step 507: Processing large files.
处理大文件的过程具体包括：The process of processing large files specifically includes:
步骤5071：更新源端服务权重，源端服务权重可以利用公式（5）计算得到，然后执行步骤5023，更新源端负载权重。Step 5071: Update source service weight , source service weight It can be calculated using formula (5), and then step 5023 is executed to update the source load weight.
步骤5072：压缩大文件，然后执行步骤508。Step 5072: Compress the large file, and then execute step 508.
具体地，使用LZW压缩算法将当前文件压缩成迁移文件,迁移文件压缩标识；其中，所述LZW算法是一种能有效减少冗余数据的压缩算法，适用于大文件压缩场景。Specifically, use the LZW compression algorithm to compress the current file Compressed into migration files ,Migration file compression flag ; Among them, the LZW algorithm is a compression algorithm that can effectively reduce redundant data and is suitable for large file compression scenarios.
步骤508：传输源端文件。Step 508: Transmit source files.
传输源端文件的过程具体包括：The process of transferring source files specifically includes:
步骤5081：传输迁移文件。Step 5081: Transfer migration files.
具体地，通过在迁移平台等客户端配置或服务端配置等方式，生成迁移任务，利用迁移任务生成的rsync迁移指令将迁移文件从源端服务器移动至目标端服务器，其中，rsync是一种数据镜像备份工具；若，表明迁移文件为其他文件，无需解压，目标端服务器执行步骤509；若，表明迁移文件为小文件组，目标端服务器执行步骤5082；若，表明迁移文件为大文件，目标端服务器执行步骤5083。Specifically, a migration task is generated by configuring the client or server on the migration platform, and the migration file is transferred using the rsync migration command generated by the migration task. Move from the source server to the target server, where rsync is a data mirroring backup tool; if , indicating that the migration file is other files and does not need to be decompressed, the target server executes step 509; if , indicating that the migration file is a small file group, the target server executes step 5082; if , indicating that the migration file is a large file, the target server executes step 5083.
步骤5082：解压小文件；具体地，使用霍夫曼压缩算法解压还原迁移文件，清空小文件组，执行步骤509。Step 5082: Decompress the small file; specifically, use the Huffman compression algorithm to decompress and restore the migration file , clear the small file group , execute step 509.
步骤5083：解压大文件；具体地，使用LZW算法解压还原迁移文件，然后执行步骤509。Step 5083: Decompress the large file; specifically, use the LZW algorithm to decompress and restore the migration file , then execute step 509.
步骤509：判断是否迁移完成；具体的，核对源端文件列表中是否存在未迁移文件，若存在未迁移文件，则执行步骤503，继续迁移流程；若不存在未迁移文件，反馈迁移平台当前迁移任务已成功。Step 509: Determine whether the migration is complete; specifically, check whether there are any unmigrated files in the source file list. If there are any unmigrated files, execute step 503 and continue the migration process; if there are no unmigrated files, feedback to the migration platform that the current migration task has been successful.
本公开应用实施例具有以下优点：The application embodiment of the present disclosure has the following advantages:
（1）能够根据源端文件特性智能调整文件压缩策略，能够适应源端服务器中不同规格文件掺杂的复杂迁移场景，有效提升服务器端资源利用率，减少了额外的人力投入。(1) It can intelligently adjust the file compression strategy according to the characteristics of the source files, and can adapt to complex migration scenarios where files of different specifications are mixed in the source server, effectively improving the resource utilization of the server and reducing additional manpower investment.
（2）迁移过程中充分考虑源端环境变化，能够根据解压缩处理模块服务压力动态调整文件规格划分，在保障源端正常运行的情况下优化迁移性能，增强迁移流程健壮性。(2) The migration process fully considers changes in the source environment and can dynamically adjust file size division according to the service pressure of the decompression processing module, thereby optimizing migration performance while ensuring normal operation of the source end and enhancing the robustness of the migration process.
为了实现本申请实施例的方法，本申请实施例还提供了一种数据处理装置，所述数据处理装置设置在电子设备上；如图6所示，该装置包括：In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides a data processing device, which is arranged on an electronic device; as shown in FIG6 , the device includes:
确定单元601，用于基于发送端设备的负载波动信息，确定数据分割阈值；以及，基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型；The determining unit 601 is used to determine a data segmentation threshold based on the load fluctuation information of the transmitting end device; and determine a data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed;
处理单元602，用于利用所述数据类型对应的压缩策略，压缩所述待处理数据Processing unit 602, used to compress the data to be processed using the compression strategy corresponding to the data type
在一些实施例中，所述确定单元601，还可以用于：In some embodiments, the determining unit 601 may also be used to:
周期性获取所述发送端设备的运行信息和服务信息；所述运行信息包括所述发送端设备运行环境相关信息，所述服务信息包括所述发送端设备处理任务的信息；Periodically acquiring the operation information and service information of the sending end device; the operation information includes information related to the operation environment of the sending end device, and the service information includes information on the processing tasks of the sending end device;
基于所述运行信息和服务信息，确定所述负载波动信息。The load fluctuation information is determined based on the operation information and the service information.
在一些实施例中，所述数据分割阈值包括第一阈值和第二阈值，所述确定单元601，具体可以用于以下之一：In some embodiments, the data segmentation threshold includes a first threshold and a second threshold, and the determination unit 601 can be specifically used for one of the following:
所述待处理数据的数据大小小于或等于第一阈值，确定所述待处理数据为第一类数据；The data size of the data to be processed is less than or equal to a first threshold, and the data to be processed is determined to be first-category data;
所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值，确定所述待处理数据为第二类数据；The data size of the data to be processed is greater than the first threshold, and the data size of the data to be processed is less than or equal to the second threshold, determining that the data to be processed is the second category of data;
所述待处理数据的数据大小大于第二阈值，确定所述待处理数据为第三类数据。The data size of the data to be processed is greater than a second threshold, and it is determined that the data to be processed is third-category data.
在一些实施例中，所述处理单元602，具体可以用于：In some embodiments, the processing unit 602 may be specifically configured to:
所述待处理数据为第一类数据，将所述待处理数据存储至数据组；利用所述第一类数据对应的第一压缩策略，压缩所述数据组；The data to be processed is the first type of data, and the data to be processed is stored in a data group; the data group is compressed using a first compression strategy corresponding to the first type of data;
或者，or,
所述待处理数据为第二类数据，利用所述第二类数据对应的第二压缩策略，压缩所述待处理数据。The data to be processed is second-category data, and the data to be processed is compressed using a second compression strategy corresponding to the second-category data.
在一些实施例中，所述处理单元602，具体可以用于：In some embodiments, the processing unit 602 may be specifically configured to:
判断所述数据组中的数据大小是否超过第一阈值；Determining whether the data size in the data group exceeds a first threshold;
在所述数据组中的数据大小超过第一阈值的情况下，利用所述第一类数据对应的第一压缩策略，压缩所述数据组。When the data size in the data group exceeds a first threshold, the data group is compressed using a first compression strategy corresponding to the first type of data.
在一些实施例中，所述装置还包括发送单元，所述发送单元用于以下之一：In some embodiments, the apparatus further comprises a sending unit, wherein the sending unit is configured to:
所述待处理数据为第一类数据，发送第一数据，所述第一数据包括标识信息和压缩后的所述数据组；The data to be processed is first type of data, and first data is sent, wherein the first data includes identification information and the compressed data group;
所述待处理数据为第二类数据，发送第二数据，所述第二数据包括标识信息和压缩后的所述待处理数据；The data to be processed is the second type of data, and second data is sent, where the second data includes identification information and the compressed data to be processed;
所述待处理数据为第三类数据，发送第三数据，所述第三数据包括标识信息和所述待处理数据；其中，The data to be processed is the third type of data, and the third data is sent, and the third data includes identification information and the data to be processed; wherein,
所述标识信息用于标识发送数据的数据类型。The identification information is used to identify the data type of the transmitted data.
在一些实施例中，所述待处理数据为第二类数据，所述处理单元602还可以用于：In some embodiments, the data to be processed is the second type of data, and the processing unit 602 may also be used to:
基于所述待处理数据的数据大小与所述第二阈值的差值，更新所述服务信息。The service information is updated based on the difference between the data size of the to-be-processed data and the second threshold.
实际应用时，所述确定单元601和发送单元可由数据处理装置中的处理器结合通信接口实现，所述处理单元602可由数据处理装置中的处理器实现。In actual application, the determining unit 601 and the sending unit may be implemented by a processor in a data processing device in combination with a communication interface, and the processing unit 602 may be implemented by a processor in the data processing device.
需要说明的是：上述实施例提供的数据处理装置在进行数据处理时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将装置的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的数据处理装置与数据处理方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。It should be noted that: when the data processing device provided in the above embodiment performs data processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
基于上述程序模块的硬件实现，且为了实现本申请实施例的方法，本申请实施例还提供了一种电子设备，如图7所示，该电子设备700可以包括：Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present application, the embodiment of the present application further provides an electronic device, as shown in FIG. 7 , the electronic device 700 may include:
通信接口701，能够发送和接收信号；例如，用于发送第一数据或第二数据；Communication interface 701, capable of sending and receiving signals; for example, used to send first data or second data;
处理器702，与所述通信接口701连接，以实现信号的发送和接收，用于运行计算机程序时，执行上述一个或多个技术方案提供的数据处理方法；A processor 702, connected to the communication interface 701 to implement signal sending and receiving, and used to execute the data processing method provided by one or more of the above technical solutions when running a computer program;
存储器703，所述计算机程序存储在所述存储器703上。A memory 703 , on which the computer program is stored.
具体地，所述处理器702，用于：Specifically, the processor 702 is configured to:
基于发送端设备的负载波动信息，确定数据分割阈值；Determine a data segmentation threshold based on load fluctuation information of the sending end device;
基于所述数据分割阈值和待处理数据的数据大小，确定所述待处理数据的数据类型；Determine the data type of the data to be processed based on the data segmentation threshold and the data size of the data to be processed;
利用所述数据类型对应的压缩策略，压缩所述待处理数据。The data to be processed is compressed using a compression strategy corresponding to the data type.
在一些实施例中，所述处理器702，还可以用于：In some embodiments, the processor 702 may also be configured to:
周期性获取所述发送端设备的运行信息和服务信息；所述运行信息包括所述发送端设备运行环境相关信息，所述服务信息包括所述发送端设备处理任务的信息；Periodically acquiring the operation information and service information of the sending end device; the operation information includes information related to the operation environment of the sending end device, and the service information includes information on the processing tasks of the sending end device;
基于所述运行信息和服务信息，确定所述负载波动信息。The load fluctuation information is determined based on the operation information and the service information.
在一些实施例中，所述数据分割阈值包括第一阈值和第二阈值；所述处理器702，具体可以用于以下之一：In some embodiments, the data segmentation threshold includes a first threshold and a second threshold; the processor 702 may be specifically configured to do one of the following:
所述待处理数据的数据大小小于或等于第一阈值，确定所述待处理数据为第一类数据；The data size of the data to be processed is less than or equal to a first threshold, and the data to be processed is determined to be first-category data;
所述待处理数据的数据大小大于第一阈值，且所述待处理数据的数据大小小于或等于第二阈值，确定所述待处理数据为第二类数据；The data size of the data to be processed is greater than the first threshold, and the data size of the data to be processed is less than or equal to the second threshold, determining that the data to be processed is the second category of data;
所述待处理数据的数据大小大于第二阈值，确定所述待处理数据为第三类数据。The data size of the data to be processed is greater than a second threshold, and it is determined that the data to be processed is third-category data.
在一些实施例中，所述处理器702，具体可以用于：In some embodiments, the processor 702 may be specifically configured to:
所述待处理数据为第一类数据，将所述待处理数据存储至数据组；利用所述第一类数据对应的第一压缩策略，压缩所述数据组；The data to be processed is the first type of data, and the data to be processed is stored in a data group; the data group is compressed using a first compression strategy corresponding to the first type of data;
或者，or,
所述待处理数据为第二类数据，利用所述第二类数据对应的第二压缩策略，压缩所述待在一些实施例中，所述处理器702，具体可以用于：The data to be processed is the second type of data, and the second compression strategy corresponding to the second type of data is used to compress the data to be processed. In some embodiments, the processor 702 may be specifically configured to:
判断所述数据组中的数据大小是否超过第一阈值；Determining whether the data size in the data group exceeds a first threshold;
在所述数据组中的数据大小超过第一阈值的情况下，利用所述第一类数据对应的第一压缩策略，压缩所述数据组。When the data size in the data group exceeds a first threshold, the data group is compressed using a first compression strategy corresponding to the first type of data.
在一些实施例中，所述处理器702，还可以用于以下之一：In some embodiments, the processor 702 may also be used for one of the following:
所述待处理数据为第一类数据，利用所述通信接口701发送第一数据，所述第一数据包括标识信息和压缩后的所述数据组；The data to be processed is the first type of data, and the communication interface 701 is used to send the first data, wherein the first data includes identification information and the compressed data group;
所述待处理数据为第二类数据，利用所述通信接口701发送第二数据，所述第二数据包括标识信息和压缩后的所述待处理数据；The data to be processed is the second type of data, and the communication interface 701 is used to send the second data, wherein the second data includes identification information and the compressed data to be processed;
所述待处理数据为第三类数据，利用所述通信接口701发送第三数据，所述第三数据包括标识信息和所述待处理数据；其中，The data to be processed is the third type of data, and the third data is sent using the communication interface 701, and the third data includes identification information and the data to be processed; wherein,
所述标识信息用于标识发送数据的数据类型。The identification information is used to identify the data type of the transmitted data.
在一些实施例中，所述处理器702，还可以用于：In some embodiments, the processor 702 may also be configured to:
基于所述待处理数据的数据大小与所述第二阈值的差值，更新所述服务信息。The service information is updated based on the difference between the data size of the to-be-processed data and the second threshold.
需要说明的是，处理器702的具体处理过程可参照上述方法理解。It should be noted that the specific processing process of the processor 702 can be understood by referring to the above method.
当然，实际应用时，所述电子设备700中的各个组件通过总线系统704耦合在一起。Of course, in actual application, the various components in the electronic device 700 are coupled together via the bus system 704 .
可理解，总线系统704用于实现这些组件之间的连接通信。总线系统704除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图8中将各种总线都标为总线系统704。It is understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as the bus system 704 in FIG. 8 .
本申请实施例中的存储器703用于存储各种类型的数据以支持电子设备700的操作。这些数据的示例包括：用于在电子设备700上操作的任何计算机程序。The memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 700. Examples of such data include: any computer program used to operate on the electronic device 700.
上述本申请实施例揭示的方法可以应用于所述处理器702中，或者由所述处理器702实现。所述处理器702可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过所述处理器702中的硬件的集成逻辑电路或者软件形式的指令完成。上述的所述处理器702可以是通用处理器、数字信号处理器(DSP，Digital SignalProcessor)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。所述处理器702可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本申请实施例所公开的方法的步骤，可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中，该存储介质位于存储器703，所述处理器702读取存储器703中的信息，结合其硬件完成前述方法的步骤。The method disclosed in the above embodiment of the present application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 702 or an instruction in the form of software. The above-mentioned processor 702 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 703, and the processor 702 reads the information in the memory 703 and completes the steps of the above method in combination with its hardware.
在示例性实施例中，电子设备700可以被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，ProgrammableLogic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable Logic Device)、现场可编程门阵列(FPGA，Field-Programmable Gate Array)、通用处理器、控制器、微控制器(MCU，Micro Controller Unit)、微处理器(Microprocessor)、或者其他电子元件实现，用于执行前述方法。In an exemplary embodiment, the electronic device 700 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.
在示例性实施例中，还提供了一种包括指令的非临时性计算机可读存储介质，例如包括指令的存储器703，上述指令可由电子设备700的处理器702执行以完成上述方法。例如，非临时性计算机可读存储介质可以是ROM、RAM、CD-ROM、磁带、软盘和光数据存储设备等。In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 703 including instructions, and the instructions can be executed by a processor 702 of an electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
本公开的实施例还提出了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，计算机指令用于使计算机执行本公开上述实施例中描述的数据处理方法。The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data processing method described in the above embodiments of the present disclosure.
本公开的实施例还提出一种计算机程序产品，包括计算机程序，计算机程序在被处理器执行本公开上述实施例中描述的数据处理方法。The embodiments of the present disclosure further provide a computer program product, including a computer program, which executes the data processing method described in the above embodiments of the present disclosure when a processor is used.
本公开的实施例还提出了一种芯片，可参见图8所示的芯片的结构示意图。图8所示的芯片包括处理器801和接口电路802。其中，处理器801的数量可以是一个或多个，接口电路802的数量可以是一个或多个。The embodiment of the present disclosure also proposes a chip, which can be seen in the schematic diagram of the chip structure shown in Figure 8. The chip shown in Figure 8 includes a processor 801 and an interface circuit 802. The number of processors 801 can be one or more, and the number of interface circuits 802 can be one or more.
可选的，芯片还包括存储器803，存储器803用于存储必要的计算机程序和数据；接口电路802用于从存储器803接收信号，并向处理器701发送信号，信号包括存储器703中存储的计算机指令，当处理器801执行计算机指令时，使得电子设备执行本公开上述实施例中描述的数据处理方法。Optionally, the chip also includes a memory 803, which is used to store necessary computer programs and data; the interface circuit 802 is used to receive signals from the memory 803 and send signals to the processor 701, and the signals include computer instructions stored in the memory 703. When the processor 801 executes the computer instructions, the electronic device executes the data processing method described in the above embodiments of the present disclosure.
需要说明的是，本公开的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本公开的实施例能够以除了在这里图示或描述的那些以外的顺序实施。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置和方法的例子。It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
在本说明书的描述中，参考术语“一个实施方式”、“一些实施方式”、“示意性实施方式”、“示例”、“具体示例”或“一些示例”等的描述意指结合实施方式或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施方式或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本发明的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本发明的实施例所属技术领域的技术人员所理解。Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理模块的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(控制方法)，便携式计算机盘盒(磁装置)， RAM，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得程序，然后将其存储在计算机存储器中。The logic and/or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), RAM, read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic device, and portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
应当理解，本发明的实施方式的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
此外，在本发明的各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。上述提到的存储介质可以是只读存储器，磁盘或光盘等。In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
尽管上面已经示出和描述了本发明的实施方式，可以理解的是，上述实施方式是示例性的，不能理解为对本发明的限制，本领域的普通技术人员在本发明的范围内可以对上述实施实施进行变化、修改、替换和变型。Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311183689.1ACN116932464B (en) | 2023-09-14 | 2023-09-14 | Data processing methods, devices, electronic equipment and storage media |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311183689.1ACN116932464B (en) | 2023-09-14 | 2023-09-14 | Data processing methods, devices, electronic equipment and storage media |
| Publication Number | Publication Date |
|---|---|
| CN116932464Atrue CN116932464A (en) | 2023-10-24 |
| CN116932464B CN116932464B (en) | 2023-11-28 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311183689.1AActiveCN116932464B (en) | 2023-09-14 | 2023-09-14 | Data processing methods, devices, electronic equipment and storage media |
| Country | Link |
|---|---|
| CN (1) | CN116932464B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110351336A (en)* | 2019-06-10 | 2019-10-18 | 西安交通大学 | A kind of edge service moving method based on docker container |
| CN113986119A (en)* | 2021-09-30 | 2022-01-28 | 贵州电网有限责任公司 | Simplified data reading method and system for accelerating cloud migration speed |
| CN114039973A (en)* | 2021-12-14 | 2022-02-11 | 中国建设银行股份有限公司 | File transmission method, device and storage medium |
| CN114564457A (en)* | 2022-02-24 | 2022-05-31 | 滨州东方地毯有限公司 | A storage space optimization method and system for database files |
| CN115048179A (en)* | 2021-03-08 | 2022-09-13 | 中国电信股份有限公司 | Migration optimization method, source end device and virtual machine migration management system |
| CN116567079A (en)* | 2022-01-30 | 2023-08-08 | 华为技术有限公司 | Data compression method and device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110351336A (en)* | 2019-06-10 | 2019-10-18 | 西安交通大学 | A kind of edge service moving method based on docker container |
| CN115048179A (en)* | 2021-03-08 | 2022-09-13 | 中国电信股份有限公司 | Migration optimization method, source end device and virtual machine migration management system |
| CN113986119A (en)* | 2021-09-30 | 2022-01-28 | 贵州电网有限责任公司 | Simplified data reading method and system for accelerating cloud migration speed |
| CN114039973A (en)* | 2021-12-14 | 2022-02-11 | 中国建设银行股份有限公司 | File transmission method, device and storage medium |
| CN116567079A (en)* | 2022-01-30 | 2023-08-08 | 华为技术有限公司 | Data compression method and device |
| CN114564457A (en)* | 2022-02-24 | 2022-05-31 | 滨州东方地毯有限公司 | A storage space optimization method and system for database files |
| Publication number | Publication date |
|---|---|
| CN116932464B (en) | 2023-11-28 |
| Publication | Publication Date | Title |
|---|---|---|
| US20210342692A1 (en) | Technologies for scaling deep learning training | |
| CN110677402B (en) | Data integration method and device based on intelligent network card | |
| CN109918018B (en) | Data storage method and storage device | |
| US20160070910A1 (en) | Platform based verification of contents of input-output devices | |
| US12073102B2 (en) | Method and apparatus for compressing data of storage system, device, and readable storage medium | |
| JP2008065834A (en) | Fusion memory device and method | |
| US20080263244A1 (en) | Compression control device and method | |
| WO2021114025A1 (en) | Incremental data determination method, incremental data determination apparatus, server and terminal device | |
| EP4160951B1 (en) | Data transmission methods, devices, system, and medium | |
| CN103455363A (en) | Command processing method, device and physical host of virtual machine | |
| CN1584875B (en) | Ergodic compressing and decompressing method for batched computer document | |
| CN117857536A (en) | File uploading method and computer equipment | |
| CN112035401A (en) | Model data processing method, apparatus, electronic device and readable medium | |
| JP7075077B2 (en) | Backup server, backup method, program, storage system | |
| CN116932464B (en) | Data processing methods, devices, electronic equipment and storage media | |
| CN113656364B (en) | Sensor data processing method, device and computer readable storage medium | |
| US12346308B2 (en) | Data compression method and apparatus | |
| CN109710581B (en) | Method for decompressing compressed data in qcow image file | |
| CN116048424B (en) | IO data processing method, device, equipment and medium | |
| CN107124615A (en) | A kind of method and device of WebP lossy compression methods | |
| CN106063192A (en) | Transmission method for wireless ethernet interface hard disk, related device, and system | |
| CN116132546A (en) | Method, device, equipment and medium for data transmission | |
| JP6183101B2 (en) | Information processing system, control program, and control method | |
| JP7140705B2 (en) | Storage system and data recording method | |
| CN107346271A (en) | The method and calamity of Backup Data block are for end equipment |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |