Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
An implementation environment related to the service data processing method provided by the embodiment of the present application will be briefly described below.
The implementation environment includes a service data processing system, which may include a data source server 101, an index server 102, a plurality of service servers, and an HDFS (Hadoop Distributed FILE SYSTEM ) 104 as shown in fig. 1, it should be noted that fig. 1 only illustrates a target server 103, where the target server 103 is one of the plurality of service servers.
Wherein, the data source server 101 may communicate with the index server 102 and with the HDFS104 through a network, the index server 102 may communicate with the data source server 101, with the target server 103, with the HDFS104 and with the client 105 through a network, and the target server 103 may communicate with the index server 102, with the HDFS104 and with the client 105 through a network.
The data source server 101 and the index server 102 may be one server, or may be a server cluster formed by a plurality of servers, and the data source server 101, the index server 102 and the target server 103 may be tower servers, rack servers, blade servers, high-density servers, single-path servers, two-path servers, or multiple-path servers, and the client 105 may be a personal computer, a notebook computer, a media player, a smart television, a smart phone, a tablet computer, a portable wearable device, or the like.
In one embodiment, as shown in fig. 2, a service data processing method is provided, and the service data processing system shown in fig. 1 is taken as an example and applied to the method, where the service data processing system may include an index server and a plurality of service servers, and the target server is one service server of the plurality of service servers. The method comprises the steps of 201, 202, 203 and 204:
in step 201, the index server obtains the original incremental service data, and performs data slicing on the original incremental service data to obtain incremental sliced service data.
In the service data processing system, a large amount of offline service data exists, the index server processes the service data correspondingly, then builds a library of the processed service data, and the processed service data is stored in an HDFS (Hadoop Distributed FILE SYSTEM ) in a distributed manner. The target server can directly pull the processed service data from the HDFS and load the service data into the memory for the client to access.
The service data processing system can continuously generate newly added service data, namely original incremental service data, along with the time, the index server acquires the original incremental service data, and performs data slicing on the original incremental service data to obtain incremental sliced service data. In one possible implementation manner, the index server can divide the original incremental service data into a plurality of fragments through the map-reduce task, so that each fragment can be stored in the HDFS in a distributed manner after being processed by the index server, and distributed management of the original incremental service data is realized.
Referring to fig. 3, fig. 3 shows an exemplary service data fragmentation process schematic. As shown in FIG. 3, the index server divides the original delta service data into two slices, one delta slice service data for each slice, by the map-reduce task.
Step 202, the index server processes the incremental fragment service data to obtain incremental service data with a preset data structure, and stores the incremental service data in the distributed file system HDFS.
In the embodiment of the application, the index server processes and stores the incremental fragment service data, and processes the incremental fragment service data to obtain the incremental service data with a preset data structure, wherein the preset data structure can be in a hash table form, so that the incremental service data can be searched quickly.
In one possible implementation of step 202, the index server may process the incremental fragment service data using a zipper method to obtain incremental service data having a preset data structure.
The zipper method, also called a chain address method, is to put key (synonym) values with the same hash address in the same single-chain table (synonym linked list). The index server adopts a zipper method to hash the incremental fragment service data, and incremental service data corresponding to the incremental fragment service data is generated. Incremental business data generated by adopting a zipper method supports change insertion, so that automatic data updating of the incremental business data can be realized.
The index server processes the incremental slicing service data to obtain the incremental service data with a preset data structure, and then the incremental service data is stored in the distributed file system HDFS, so that the automatic storage and updating of the incremental service data are realized, the incremental slicing service data after slicing are processed through data slicing and then are stored in the distributed file system HDFS, the distributed management of the original incremental service data can be realized, even if the service data is increased sharply, the distributed management of the service data can be realized through data slicing, and the expandability of the service data processing system is enhanced.
In step 203, the index server updates the data version information in the target server based on the incremental service data, and sends a data version update notification to the target server.
Wherein the target server is one of a plurality of service servers.
After the index server stores the incremental service data in the HDFS, the index server updates the data version information of the incremental service data to the target server, and sends a data version update notification to the target server to prompt the target server to pull the incremental service data from the HDFS according to the data version information of the incremental service data.
In the embodiment of the application, the data version information may represent version information of the service data, for example, the data version information of the incremental service data may be a version number of the incremental service data.
In step 204, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, and loads the incremental service data into the memory for the client to access.
After receiving the data version update notification sent by the index server, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information (i.e., the data version information of the incremental service data updated to the target server by the index server).
In one possible implementation manner, after the index server stores the incremental service data in the HDFS, the index server may further generate first data basic information according to updated data version information of the incremental service data and a first storage address of the incremental service data in the HDFS, where the updated data version information is used for the target server to obtain the first data basic information according to the updated data version information, and the first data basic information is used for the target server to pull the incremental service data according to a storage location of the HDFS indicated by the first storage address. The target server may implement the process of step 204 by performing steps a11 and a12 as follows:
and step A11, the target server acquires the first data basic information from the index server according to the updated data version information.
And step A12, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.
In this way, when the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, the first data basic information may be obtained from the index server according to the updated data version information, and then the incremental service data corresponding to the updated data version information may be pulled from the HDFS storage location indicated by the first storage address included in the first data basic information.
After the target server pulls the incremental business data from the HDFS, the incremental business data is automatically loaded to the memory of the target server, so that the automatic loading of the incremental business data is realized, and the automation capacity of the business data processing system is improved.
If the target server receives the access request for the incremental service data sent by the client, the target server responds to the data access request based on the incremental service data in the memory, so that the client accesses the incremental service data loaded into the memory, and the offline service data is automatically converted into online service, thereby improving the service data processing efficiency.
Optionally, the target server according to the embodiment of the present application may be deployed in a kubernetes isochoric container cluster. Taking kubernetes as an example, the target server serves as a service node in kubernetes, and a container deployed in the pod in the target server is started to provide data access service for the client, and the container can provide services such as reading and writing of incremental business data for the client when running.
Optionally, an index server may be deployed in kubernetes, where the index server performs service discovery through consul, and consul is an open source tool developed based on GO language, and mainly faces to a distributed, served system to provide functions of service registration, service discovery and configuration management.
In one possible implementation, referring to fig. 4, the process of the index server for service discovery through consul may include steps 401, 402, 403, and 404 shown in fig. 4:
In step 401, if the index server receives the data address acquisition request sent by the client, the index server determines the service address of the target server according to the data address acquisition request.
The client sends a data address acquisition request to the index server to acquire an address of a service server that can be accessed, and the index server determines a target server for providing services to the client from a plurality of service servers through consul.
Referring to fig. 5, fig. 5 is a schematic diagram illustrating service discovery by an exemplary index server. As shown in fig. 5, the servers may be service servers, consul periodically perform health check on each server, detect whether each server operates normally, add SERVER LIST (server list) the servers that operate normally, remove servers that do not operate normally from the server list, and ensure that the service servers in the server list can provide services normally.
The index server obtains the service address of the server operating normally from consul, and the service address can be IP: the index server can establish mapping relation between each key code and each service address in the form of Port, different key codes correspond to different service addresses, and the key codes can be in the form of character strings.
In one possible implementation manner, the index server may analyze the data address obtaining request to obtain a key code corresponding to the data address obtaining request, and determine, according to the key code and the established mapping relationship, a service address of a target server corresponding to the key code from the plurality of service servers, so as to implement a process of determining, by consul, the target server for providing services to the client from the plurality of service servers. According to the embodiment of the application, the index server performs service discovery through consul, so that decoupling of the client and the service server is realized, and the service processing efficiency is improved.
The index server returns the service address to the client, step 402.
Step 403, the client sends an access request for the incremental service data to the target server according to the service address.
In step 404, the target server receives the access request and responds to the access request based on the incremental transaction data loaded in the memory.
And after the index server determines the service address of the target server from the plurality of service servers, the index server returns the service address to the client. And the client sends an access request for the incremental service data to the service address, and after receiving the access request, the target server responds to the data access request based on the incremental service data loaded in the memory, so that the client can access the incremental service data. As an embodiment, the client may access the traffic data from the target server via an http-proxy interface.
In the embodiment of the application, the client can send the data address acquisition request comprising the key code, acquire the returned service address of the target server from the index server, and access the incremental service data in the target server based on the service address, thereby realizing the k-v (key-value) query access of the client.
In other embodiments, the index server may also build a vector database during database creation, so that the client may implement vector-based service data query access, which is not particularly limited herein.
In the service data processing system of the embodiment of the application, after the original incremental service data is obtained by the index server, the original incremental service data is subjected to data slicing to obtain the incremental sliced service data, the incremental sliced service data is processed by the index server to obtain the incremental service data with a preset data structure, and the incremental service data is stored in the HDFS, so that the automatic storage of the incremental service data is realized, the sliced incremental sliced service data is processed and then distributed and stored in the HDFS, the distributed management of the original incremental service data can be realized, the distributed management of the service data can be realized through the data slicing even if the service data is rapidly increased, and the expandability of the service data processing system is enhanced; and then, the index server updates the data version information in the target server based on the incremental service data, and sends a data version update notification to the target server, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, and loads the incremental service data to the memory for the client to access, so that the target server directly pulls the incremental service data from the HDFS according to the updated data version information, loads the pulled incremental service data to the memory for the client to access, the automatic loading of the incremental service data is realized, and the automation capability of the service data processing system is improved, thereby avoiding the problem that in the traditional technology, the service data needs to be manually imported into the Redis in the memory to provide the automation capability of the service data processing system caused by on-line access. In addition, in the embodiment of the application, the incremental service data is stored in the HDFS, and the target server automatically pulls the incremental service data and loads the incremental service data into the memory without manually importing the incremental service data into the Redis, so that the RDB based on the Redis is not required to realize persistence, and the memory waste caused by that the RDB needs to copy one part of service data in the memory into the memory and then persistence the copied data is avoided.
In one embodiment, based on the embodiment shown in fig. 2, the present embodiment refers to a process how the index server obtains the original incremental service data and how the index server performs data normalization on the incremental service data. The service data processing system of this embodiment further includes a data source server, as shown in fig. 6, and the service data processing method provided in this embodiment further includes step 205:
in step 205, the data source server obtains the original incremental service data, stores the original incremental service data to the HDFS, and sends the second data base information of the original incremental service data to the index server.
Wherein the second data base information at least comprises a second storage address of the original incremental business data in the HDFS.
Correspondingly, step 201 may include step 2011 as shown in fig. 6:
In step 2011, the index server pulls the original incremental service data from the HDFS storage location indicated by the second storage address according to the second storage address included in the second data basic information, and performs data slicing on the original incremental service data to obtain incremental sliced service data.
With continued reference to fig. 6, after step 2011, the method for processing service data provided in this embodiment may further include step 206:
And 206, analyzing the incremental slicing service data by the index server according to a preset data format to obtain the analyzed incremental slicing service data.
In the embodiment of the application, after the index server performs data slicing on the original incremental business data to obtain the incremental sliced business data, the index server can analyze the incremental sliced business data according to a preset data format to obtain the analyzed incremental sliced business data. The preset data format may be, for example, a Protobuf format.
Correspondingly, step 202 may include step 2021 as shown in fig. 6:
in step 2021, the index server processes the parsed incremental fragment service data to obtain incremental service data with a preset data structure, and stores the incremental service data in the HDFS.
The index server may hash the parsed incremental fragment traffic data using a zipper method to generate incremental traffic data to implement the process of step 202. Therefore, the analyzed incremental fragment business data are in a uniform data format, and the user can take structured data, so that the aim of normalizing the data is fulfilled. The problem of the traditional technique that the data maintenance difficulty is high because the data source is complex and the data definition is various is avoided. The embodiment reduces the maintenance difficulty of the incremental business data.
In one embodiment, based on the embodiment shown in fig. 2 above, this embodiment relates to a process how the target server provides client access based on the full traffic data, and to a process how the target server recovers the traffic data in case the target server is restarted. In the embodiment of the application, the total business data can be historical business data, and the incremental business data is newly added business data.
Referring to fig. 7, the target server may implement a procedure of providing client access based on the full traffic data by performing steps 701 and 702 as shown in fig. 7:
in step 701, the target server obtains the full-volume data version information, and pulls the full-volume service data corresponding to the full-volume data version information from the HDFS according to the full-volume data version information.
Wherein the full-volume data version information is updated in the target server by the index server based on the full-volume service data.
In the embodiment of the application, the full-volume service data can be obtained by corresponding processing of the original full-volume service data by the index server, such as slicing processing, hash processing and the like. After the index server carries out corresponding processing on the original full-volume service data, the obtained full-volume service data is stored into the HDFS.
The index server updates the full-volume data version information of the full-volume service data to the target server, informs the target server to pull the full-volume service data, and pulls the full-volume service data corresponding to the full-volume data version information from the HDFS according to the full-volume data version information.
In step 702, the target server loads the full traffic data into the memory for the client to access.
The target server loads the full traffic data into the memory, and when the client accesses the full traffic data and/or the incremental traffic data, the target server can provide services for the client based on the traffic data loaded in the memory. Therefore, offline business data is automatically converted into online service, and business processing efficiency is improved.
With continued reference to fig. 7, the target server may implement a procedure for recovering service data in the event of restart of the target server by performing step 703 and step 704 shown in fig. 7:
in step 703, if the target server detects a restart trigger event, the target server pulls incremental service data from the HDFS according to the updated data version information, and pulls full service data from the HDFS according to the full data version information.
In step 704, the target server loads the pulled incremental traffic data and the full traffic data into the memory for access by the client.
In the embodiment of the application, if the target server is restarted due to unexpected downtime and other conditions, the target server acquires the full-volume data version information and the updated data version information from the index server again, and pulls the incremental service data from the HDFS according to the updated data version information and pulls the full-volume service data from the HDFS according to the full-volume data version information.
The target server loads the pulled incremental service data and the full service data into the memory, and when the client accesses the full service data and/or the incremental service data, the target server can provide services for the client based on the service data loaded in the memory. In this way, even if the target server is restarted, no data is lost.
In the conventional technology, in order to avoid service data loss during restarting of a server, data persistence is generally performed through RDB of Redis, and the problem of double memory occupation exists in the data persistence process, so that memory resource waste is caused. In the embodiment of the application, the service data can be pulled from the HDFS according to the updated data version information and the full data version information after the target server is restarted, so that the extra memory of the target server is not required to be occupied for data persistence, and the memory resource of the target server is saved.
In one embodiment, referring to fig. 8, the present embodiment refers to a process of indexing server database for full traffic data, based on the embodiment shown in fig. 7. As shown in fig. 8, the process may include step 801, step 802, and step 803:
in step 801, the index server obtains full-volume shard service data from the HDFS.
In an embodiment of the present application, the service data processing system further includes a data source server, and in a possible implementation manner, the data source server derives a full-volume sliced service data, where the full-volume sliced service data is obtained after performing data slicing on the original full-volume service data, and the data source server pushes the full-volume sliced service data to the HDFS.
In another possible implementation, the data source server derives the original full traffic data, the data source server may push the original full traffic data to the HDFS, and then the index server performs data slicing on the original full traffic data to obtain full sliced traffic data, and stores the full traffic data in the HDFS.
Thus, the index server can obtain the full-volume fragment service data from the HDFS.
Step 802, the index server processes the full-volume fragmented service data by adopting an open addressing method to generate the full-volume service data with a preset data structure.
The preset data structure may be in the form of a hash table, for example, an open addressing method is a method for processing conflicts by starting from the unit where the conflict occurs, searching a free storage unit from the hash table according to a certain order, and storing the element to be inserted where the conflict occurs into the unit.
In the embodiment of the application, the index server adopts an open addressing method to hash the full-volume fragmented service data to generate the full-volume service data corresponding to the full-volume fragmented service data. The whole business data generated by adopting the open addressing method is compact, which is beneficial to reducing the storage occupation space of the whole business data.
In step 803, the index server stores the full traffic data to the HDFS, and generates third data base information according to the full data version information of the full traffic data and the third storage address.
The full-volume data version information is used for the target server to acquire third data basic information according to the full-volume data version information, and the third data basic information is used for the target server to pull full-volume service data corresponding to the full-volume data version information according to an HDFS storage position indicated by the third storage address, so that database construction of the full-volume service data is realized.
In one possible implementation, after step 803, the target server may implement the procedure of step 701 by performing steps a21 and a22 as follows:
And step A21, the target server acquires the third data basic information from the index server according to the full data version information.
And step A22, the target server pulls the full-volume service data corresponding to the full-volume data version information from the HDFS storage position indicated by the third storage address according to the third storage address included in the third data basic information.
In the embodiment of the application, after the index server stores the full-volume service data into the HDFS, the index server generates the third data basic information according to the full-volume data version information of the full-volume service data and the third storage address, and the index server updates the data version information in the target server and informs the target server to pull the data. The target server acquires full-volume data version information, acquires third data basic information corresponding to the full-volume data version information from the index server according to the full-volume data version information, and then pulls full-volume service data corresponding to the full-volume data version information from an HDFS storage position indicated by the third storage address according to a third storage address included in the third data basic information.
Therefore, the target server automatically pulls the full traffic data from the HDFS and loads the full traffic data to the memory to provide data access service by establishing the full traffic data in the HDFS, thereby realizing the automatic establishment of the offline traffic data and achieving the effect of automatically converting the offline traffic data into the online service.
Optionally, after step 801, the service data processing method of this embodiment may further include step a31:
and step A31, the index server analyzes the full-volume fragmented service data according to a preset data format to obtain the analyzed full-volume fragmented service data.
In the embodiment of the application, after the index server obtains the full-volume fragmented service data from the HDFS, the index server analyzes the full-volume fragmented service data according to a preset data format to obtain the analyzed full-volume fragmented service data. The predetermined data format may be Protobuf (Google Protocol Buffer), for example.
Correspondingly, the index server may perform the process of step 802 by performing hash processing on the parsed full-volume fragment service data by using an open addressing method to generate the full-volume service data. Thus, the analyzed full-volume fragment business data are all in a uniform data format, and the structured data are obtained by the user, so that the purpose of normalizing the data is achieved. The problem of the traditional technique that the data maintenance difficulty is high because the data source is complex and the data definition is various is avoided. The embodiment reduces the maintenance difficulty of the whole business data.
In one embodiment, based on the embodiment shown in fig. 8 and referring to fig. 9, this embodiment relates to how the target server builds a database of offline business data in the service expansion and contraction process. As shown in fig. 9, the process may include the steps of:
Step 901, the index server obtains the fragmentation information of the original full-volume service data, and detects whether the fragmentation information is the same as the stored local fragmentation information.
In an actual application scenario, with the change of the service, the number of service servers often needs to be expanded or contracted, and after the expansion or contraction occurs, the number of service servers in the service data processing system will change.
In consideration of the situation of the expansion and contraction of the volume, in the embodiment of the application, before the index server builds the library, the index server acquires the fragmentation mode information of the original full volume service data, detects whether the fragmentation mode information is the same as the stored local fragmentation mode information, and the local fragmentation mode information is related to the number of a plurality of service servers, for example, the local fragmentation mode information can be the number of the service servers in the service data processing system after the expansion and contraction of the volume.
Taking the capacity expansion scenario as an example, for example, the index server segments the original full-volume service data according to 2 segments, then the information of the segmentation mode of the original full-volume service data obtained by the index server is 2, the number of service servers in the service data processing system after capacity expansion is 4, and the index server detects whether 2 and 4 are the same.
If the fragmentation information is the same as the local fragmentation information, the number of the service servers after the fragmentation information and the expansion capacity representing the original total amount of service data is matched, and the index server performs steps 801, 802 and 803 as above to build a database for the offline service data.
If the fragmentation information is different from the local fragmentation information, the fragmentation representing the original full-volume service data is performed according to the number of service servers before the expansion and contraction capacity is not performed, so that the original full-volume service data needs to be re-fragmented to match the number of service servers after the expansion and contraction capacity, and the index server performs step 902:
And 902, performing data slicing on the original full-volume service data according to the local slicing mode information by the index server to obtain the full-volume sliced service data.
And the server performs data slicing on the original full-volume service data according to the local slicing mode information, namely, re-performs data slicing on the original full-volume service data according to the number of the expanded and contracted machines to obtain full-volume slicing service data, and the index server stores the full-volume slicing service data in the HDFS.
After re-slicing, the index server performs steps 801, 802 and 803 as described above on the basis of step 902 to build a library for offline service data.
In the embodiment of the application, the index server can automatically build the library according to the combination of the expansion and contraction situations when building the library, thereby realizing the automatic management of offline service data and better meeting the requirements of actual service on expansion and contraction.
In one embodiment, referring to fig. 10, based on the embodiment shown in fig. 2, this embodiment refers to a process in which the target server loads incremental service data into the memory. In the embodiment of the application, the target server comprises a first sliced data copy and a second sliced data copy, and the first sliced data copy and the second sliced data copy are used for providing the same data access service. As shown in fig. 10, the process may include steps 2041 and 2042:
In step 2041, the target server performs data update on the first sliced data copy according to the incremental service data, and in the process of data update of the first sliced data copy, the target server provides data access service through the second sliced data copy, and prohibits the first sliced data copy in the process of data update from providing data access service.
In the embodiment of the application, each node has multiple copies, and the node is a service server. The first and second sharded data copies may be two copies of the last version of the incremental business data (assumed to be historical incremental business data), both copies providing access to the historical incremental business data externally.
If the historical incremental business data is updated, the updated data is the incremental business data according to the embodiment of the application, and after the target server acquires the incremental business data, the incremental business data is required to be loaded into the memory so as to update the historical incremental business data by adopting the incremental business data.
In the embodiment of the application, the target server updates the partitioned data copies one by one, so that at least one partitioned data copy can be ensured to provide service.
As one implementation, the target server is deployed in Kubernetes, and the first sliced data copy and the second sliced data copy provide services externally through different pod. When the target server updates the data of the first sliced data copy according to the incremental service data, the target server changes the state of the pod corresponding to the first sliced data copy from running to stopping running so as to inhibit the first sliced data copy in the data updating process from providing the data access service. And if the access request of the client is received, the target server provides data access service for the client through the pod corresponding to the second fragmented data copy.
In step 2042, after the data of the first sliced data copy is updated, the target server performs data update on the second sliced data copy according to the incremental service data, and in the process of data update of the second sliced data copy, the target server provides data access service through the first sliced data copy after data update, and prohibits the second sliced data copy in the process of data update from providing data access service.
After the target server finishes updating the data of the first fragment data copy according to the incremental service data, the target server updates the data of the second fragment data copy according to the incremental service data,
Similarly, the target server changes the state of the pod corresponding to the second sliced data copy from running to stopping running so as to inhibit the second sliced data copy in the data updating process from providing the data access service. And if the access request of the client is received, the target server provides data access service for the client through the pod corresponding to the first fragmented data copy after data updating. Therefore, the on-line service is not influenced in the process of loading the service data into the memory, the service quality of the on-line service is ensured, and the problem of unstable service caused by externally providing the service while updating the service data in the traditional technology is avoided.
Referring to fig. 11, fig. 11 is an exemplary sliced data schematic. As shown in fig. 11, the first and second slices may be different traffic slice data, e.g., the first and second slices may be different incremental traffic data. The first service server loads the first fragment into the memory of the first service server, the second service server loads the second fragment into the memory of the second service server, the pod 0 and the pod 1 are two fragment data copies of the first fragment, and the pod 2 and the pod 3 are two fragment data copies of the second fragment.
In the data updating process, the first service server updates pod 0 and provides service through pod 1, the second service server updates pod 2 and provides service through pod 3, and the timestamps of pod 1 and pod 3 are consistent, namely the data version information is consistent, for example, 20200401.
After the update of pod 0 and pod 2 is completed, the data version information of pod 0 and pod 2 is updated to the latest version information, for example to 20200402, the first service server updates pod 1 and the second service server updates pod 3, so that the updated pod 0 and pod 2 provide services.
Therefore, for the data access of the same batch, the business data processing system is based on version control, and each accessed business server provides the fragmented data copy with the same timestamp (data version information) for the client to access, so that the data access of the same batch is ensured, the business data of the same version is accessed, and the consistency of the data access is ensured.
In one embodiment, referring to fig. 12, a flow chart of a business data processing method is provided, which can be applied in the implementation environment shown in fig. 1. As shown in fig. 12, the method may include the steps of:
in step 1001, the data source server exports the full-size fragmented traffic data and pushes the full-size fragmented traffic data to the HDFS.
The full-volume slicing service data is obtained after data slicing is carried out on the original full-volume service data.
In step 1002, the data source server obtains the original incremental service data, stores the original incremental service data to the HDFS, and sends the second data base information of the original incremental service data to the index server.
Wherein the second data basic information at least comprises a second storage address of the original incremental business data; the data source server may continuously obtain the original incremental business data.
In step 1003, the index server pulls the original incremental service data from the HDFS storage location indicated by the second storage address according to the second storage address included in the second data basic information, and performs data slicing on the original incremental service data to obtain incremental sliced service data.
In step 1004, the index server obtains the fragmentation information of the original full-volume service data and the original incremental service data, and detects whether the fragmentation information is the same as the local fragmentation information.
If the same, the index server analyzes the full-size fragment service data according to the preset data format to obtain the analyzed full-size fragment service data in step 1005.
In step 1006, the index server processes the parsed full-volume fragment service data by adopting an open addressing method to generate full-volume service data with a preset data structure.
In step 1007, the index server stores the full traffic data to the HDFS, generates third data basic information according to the full data version information of the full traffic data and the third storage address, and notifies the target server of pulling the data.
In step 1008, the target server obtains the full-size data version information, and obtains the third data basic information corresponding to the full-size data version information from the index server according to the full-size data version information.
In step 1009, the target server pulls the full service data corresponding to the full data version information from the HDFS storage location indicated by the third storage address according to the third storage address included in the third data base information.
In step 1010, the index server parses the incremental fragment service data according to a preset data format to obtain parsed incremental fragment service data.
In step 1011, the index server processes the parsed incremental fragment service data by adopting a zipper method to generate incremental service data with a preset data structure.
In step 1012, the index server stores the incremental service data to the HDFS, and generates first data basic information according to the updated data version information of the incremental service data and the first storage address of the incremental service data in the HDFS.
In step 1013, the index server updates the data version information in the target server based on the incremental service data, and sends a data version update notification to the target server.
In step 1014, if the target server receives the data version update notification sent by the index server, the target server obtains the first data basic information from the index server according to the updated data version information.
In step 1015, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS storage location indicated by the first storage address according to the first storage address included in the first data base information.
In step 1016, the target server loads the full traffic data and incremental traffic data into memory for access by the client.
In the process that the target server loads the increment service data into the memory, the target server performs data updating on the first sliced data copy according to the increment service data, in the process of data updating of the first sliced data copy, the target server provides data access service through the second sliced data copy, and the first sliced data copy in the data updating process is forbidden to provide data access service; the target server comprises a first sliced data copy and a second sliced data copy, and the first sliced data copy and the second sliced data copy are used for providing the same data access service.
After the data of the first sliced data copy is updated, the target server performs data updating on the second sliced data copy according to the incremental service data, and in the process of the data updating of the second sliced data copy, the target server provides data access service through the first sliced data copy after the data updating, and prohibits the second sliced data copy in the data updating process from providing data access service.
In step 1017, if the index server receives the data address acquisition request sent by the client, the index server analyzes the data address acquisition request to obtain a key code corresponding to the data address acquisition request, and determines a service address of a target server corresponding to the key code from the plurality of service servers according to the key code, and the index server returns the service address to the client.
Step 1018, the client sends the access request of the service data to the target server according to the service address.
In step 1019, if the target server receives the access request of the service data sent by the client, the target server responds to the data access request based on the full service data and/or the incremental service data in the memory.
In step 1020, if the target server detects a restart trigger event, the target server pulls incremental service data from the HDFS according to the updated data version information, and pulls full service data from the HDFS according to the full data version information.
In step 1021, the target server loads the pulled incremental service data and the full service data into the memory for the client to access.
In an embodiment of the present application, as an implementation manner, the index server may include an incremental data slicing DELTA SHARDING component and an update peta update component, and the index server performs the steps of the index server based on the multiple components. The target server may include consul components and a controller component, based on which the target server performs the steps of the target server.
The index server, the target server, the data source server, the HDFS and the components can be accessed through interfaces. Referring to FIG. 13, FIG. 13 is a schematic diagram of an exemplary component-based business data processing system in which clients access a server via PETA CLIENT. The data interaction process based on interfaces among the index server, the target server, the data source server, the HDFS and each component is as follows:
1) The data source server ts derives the full-size fragment traffic data base, and ts pushes the full-size fragment traffic data to the HDFS.
2) Ts obtains the delta of the original incremental business data, stores the original incremental business data into the HDFS, and calls the add_delta_file interface of the index server to send the second data basic information of the original incremental business data to the index server.
Wherein, the second data basic information at least includes a second storage address of the original incremental service data, and ts can continuously acquire the original incremental service data.
3) DELTA SHARDING accessing an index server at fixed time, acquiring second data basic information through a get_delta_ sharding _file interface and a set_delta_ sharding _file_working interface, performing data slicing on original incremental service data by running a Map/Reduce task to obtain incremental sliced service data after slicing, DELTA SHARDING storing the incremental sliced service data in an HDFS, and calling an add_delta_ sharded _file to push file information of the incremental sliced service data to the index server, wherein the file information of the incremental sliced service data comprises data version information and storage path information of the incremental sliced service data.
4) Peta update before library building, calling an add_and_get_ shard _count interface of the index server to acquire the fragmentation mode information of the original full-volume service data and the original incremental service data, and detecting whether the fragmentation mode information is identical with the local fragmentation mode information.
5) If the total amount of the fragment service data is the same as the preset data format, peta update is used for analyzing the total amount of the fragment service data to obtain the analyzed total amount of the fragment service data. peta update processing the analyzed full-volume fragmented service data by adopting an open addressing method to generate full-volume service data with a preset data structure. peta update stores the full traffic data to the HDFS, and generates third data base information according to the full data version information of the full traffic data and the third storage address.
6) Peta update according to a preset data format, analyzing the incremental fragment service data to obtain the analyzed incremental fragment service data. peta update processing the analyzed incremental fragment service data by adopting a zipper method to generate incremental service data with a preset data structure. peta update stores the incremental service data to the HDFS and generates first data base information according to the updated data version information of the incremental service data and the first storage address.
7) Peta update updates the data version information on consul and informs the controller of the pull data.
8) The controller calls get_base_and_delta_file_list of the index server to acquire full-volume data version information, and acquires third data basic information corresponding to the full-volume data version information from the index server according to the full-volume data version information. And the controller pulls the full service data corresponding to the full data version information from the HDFS storage position indicated by the third storage address according to the third storage address included in the third data basic information.
9) The controller calls get_base_and_delta_file_list of the index server to acquire updated data version information. And the controller acquires the first data basic information corresponding to the updated data version information from the index server according to the updated data version information. And the controller pulls the incremental service data corresponding to the updated data version information from the HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.
10 The controller triggers the server to load data, and the server loads the full traffic data and the incremental traffic data into the memory for the client to access.
In the process that the server loads the increment service data into the memory, the server performs data updating on the first sliced data copy according to the increment service data, in the process of data updating of the first sliced data copy, the server provides data access service through the second sliced data copy, and the first sliced data copy in the data updating process is forbidden to provide data access service; the server comprises a first sliced data copy and a second sliced data copy, and the first sliced data copy and the second sliced data copy are used for providing the same data access service.
After the data of the first sliced data copy is updated, the server performs data updating on the second sliced data copy according to the incremental service data, and in the process of the data updating of the second sliced data copy, the server provides data access service through the first sliced data copy after the data updating, and prohibits the second sliced data copy in the data updating process from providing data access service.
11 If the index server receives the data address acquisition request sent by the client, the index server analyzes the data address acquisition request to obtain a key code corresponding to the data address acquisition request, and the index server determines the service address of the target server corresponding to the key code from a plurality of service servers according to the key code.
12 Index server returns the service address to the client.
13 The client sends an access request for the incremental business data to the server according to the service address.
14 If the server receives the access request of the service data sent by the client, the server responds to the data access request based on the total service data and/or the incremental service data in the memory.
15 If the controller detects a restart trigger event, the controller pulls incremental service data from the HDFS according to the updated data version information, and pulls full service data from the HDFS according to the full data version information.
16 The controller triggers the server to reload the data, and the server loads the pulled incremental service data and the full service data into the memory for the client to access.
It should be understood that, although the steps in the above-described flowcharts are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described above may include a plurality of steps or stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of execution of the steps or stages is not necessarily sequential, but may be performed in turn or alternately with at least a part of other steps or stages.
In one embodiment, a business data processing system is provided, the business data processing system comprising an index server and a plurality of business servers, a target server being one of the plurality of business servers;
The index server is used for acquiring original incremental service data, performing data slicing on the original incremental service data to obtain incremental sliced service data, processing the incremental sliced service data to obtain incremental service data with a preset data structure, and storing the incremental service data in a distributed file system (HDFS);
The index server is further used for updating the data version information in the target server based on the incremental service data and sending a data version update notification to the target server;
and the target server is used for pulling the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, and loading the incremental service data into a memory for the client to access.
In one embodiment, the index server is further configured to generate first data basic information according to the updated data version information of the incremental service data and a first storage address of the incremental service data in the HDFS;
Correspondingly, the target server is specifically configured to obtain the first data basic information from the index server according to the updated data version information; and pulling the incremental business data corresponding to the updated data version information from an HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.
In one embodiment, the index server is specifically configured to process the incremental fragment service data by using a zipper method, so as to obtain the incremental fragment service data with a preset data structure.
In one embodiment, the service data processing system further includes a data source server, configured to obtain the original incremental service data, store the original incremental service data to the HDFS, and send second data base information of the original incremental service data to the index server, where the second data base information includes at least a second storage address of the original incremental service data in the HDFS;
Correspondingly, the index server is specifically configured to pull the original incremental service data from the HDFS storage location indicated by the second storage address according to the second storage address included in the second data base information.
In one embodiment, the index server is specifically configured to parse the incremental fragment service data according to a preset data format to obtain parsed incremental fragment service data; and processing the analyzed increment fragment service data to obtain the increment service data with a preset data structure.
In one embodiment, the target server is further configured to obtain full-volume data version information, and pull full-volume service data corresponding to the full-volume data version information from the HDFS according to the full-volume data version information, where the full-volume data version information is updated in the target server by the index server based on the full-volume service data; the target server is further configured to load the full traffic data into a memory for access by a client.
In one embodiment, the target server is further configured to pull the incremental service data from the HDFS according to the updated data version information and pull the full service data from the HDFS according to the full data version information if a restart trigger event is detected; and loading the pulled incremental service data and the pulled full service data into a memory for the client to access.
In one embodiment, the index server is further configured to obtain full-volume fragmented service data from the HDFS, where the full-volume fragmented service data is obtained by performing data fragmentation on original full-volume service data; processing the full-volume fragmented service data by adopting an open addressing method to generate the full-volume service data with a preset data structure; storing the full traffic data to the HDFS, and generating third data basic information according to the full data version information of the full traffic data and a third storage address;
Correspondingly, the target server is specifically configured to obtain the third data basic information from the index server according to the full-volume data version information; and pulling the full service data corresponding to the full data version information from an HDFS storage position indicated by the third storage address according to the third storage address included in the third data basic information.
In one embodiment, the index server is further configured to obtain fragmentation information of the original full-volume service data, and detect whether the fragmentation information is the same as local fragmentation information, where the local fragmentation information is related to the number of the plurality of service servers; if the fragmentation mode information is the same as the local fragmentation mode information, the index server executes the step that the index server acquires the full-quantity fragmentation service data from the HDFS; and if the slicing mode information is different from the local slicing mode information, the index server performs data slicing on the original full-volume service data according to the local slicing mode information to obtain the full-volume slicing service data, and the index server acquires the full-volume slicing service data from the HDFS.
In one embodiment, the target server includes a first sliced data copy and a second sliced data copy, where the first sliced data copy and the second sliced data copy are used to provide the same data access service, and the target server is specifically used to perform data update on the first sliced data copy according to the incremental service data, and in the process of data update of the first sliced data copy, the target server provides the data access service through the second sliced data copy, and forbids the first sliced data copy in the process of data update from providing the data access service; after the data of the first sliced data copy is updated, the target server performs data updating on the second sliced data copy according to the incremental service data, and in the process of the data updating of the second sliced data copy, the target server provides data access service through the first sliced data copy after the data updating, and prohibits the second sliced data copy in the process of the data updating from providing data access service.
In one embodiment, the index server is further configured to determine, if a data address acquisition request sent by the client is received, a service address of the target server according to the data address acquisition request; the index server returns the service address to the client; the client sends an access request for the incremental service data to the target server according to the service address;
The target server is further configured to receive the access request, and respond to the access request based on the incremental service data loaded in the memory.
In one embodiment, the index server is specifically configured to parse the data address obtaining request to obtain a key code corresponding to the data address obtaining request; and determining the service address of the target server corresponding to the key code from the plurality of service servers according to the key code.
For specific limitations of the service data processing system, reference may be made to the above limitation of the service data processing method, and no further description is given here. The various modules in the business data processing system described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or independent of a processor in a server, or may be stored in software in a memory in the server, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a server is provided, the internal structure of which may be as shown in FIG. 14. The server includes a processor, memory, and a network interface connected by a system bus. Wherein the processor of the server is configured to provide computing and control capabilities. The memory of the server includes nonvolatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the server is used for storing business data processing data. The network interface of the server is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a business data processing method.
It will be appreciated by those skilled in the art that the structure shown in fig. 14 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and is not limiting of the server to which the present inventive arrangements are applied, and that a particular server may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a server is provided that includes a memory and a processor, the memory having a computer program stored therein, the processor when executing the computer program performing the steps of:
Acquiring original incremental business data, and performing data slicing on the original incremental business data to obtain incremental sliced business data;
processing the incremental fragment service data to obtain incremental service data with a preset data structure, and storing the incremental service data in a distributed file system (HDFS);
Updating data version information in a target server based on the incremental service data, and sending a data version update notification to the target server;
The target server is one service server of the plurality of service servers, the data version update notification is used for notifying the target server, the incremental service data corresponding to the updated data version information is pulled from the HDFS according to the updated data version information, and the incremental service data is loaded into a memory for access by a client.
In one embodiment, the processor, when executing the computer program, further performs the following steps;
Generating first data basic information according to the updated data version information of the incremental service data and a first storage address of the incremental service data in the HDFS; the first data basic information is used for the target server to acquire the first data basic information from the index server according to the updated data version information; the first data basic information is used for the target server to pull the incremental service data corresponding to the updated data version information from an HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.
In one embodiment, the processor when executing the computer program further performs the steps of:
And processing the incremental fragment service data by adopting a zipper method to obtain the incremental service data with a preset data structure.
In one embodiment, the service data processing system further includes a data source server, where the data source server is configured to obtain the original incremental service data, store the original incremental service data to the HDFS, and send second data basic information of the original incremental service data to the index server, where the second data basic information includes at least a second storage address of the original incremental service data in the HDFS;
Correspondingly, the processor, when executing the computer program, also implements the steps of:
And pulling the original incremental business data from the HDFS storage position indicated by the second storage address according to the second storage address included in the second data basic information.
In one embodiment, the processor when executing the computer program further performs the steps of:
analyzing the incremental fragment service data according to a preset data format to obtain analyzed incremental fragment service data;
And processing the analyzed increment fragment service data to obtain the increment service data with a preset data structure.
In one embodiment, the target server is configured to obtain full-volume data version information, and pull full-volume service data corresponding to the full-volume data version information from the HDFS according to the full-volume data version information, where the full-volume data version information is updated in the target server by the index server based on the full-volume service data; and the target server loads the full service data into a memory for the client to access.
In one embodiment, the target server is configured to pull the incremental service data from the HDFS according to the updated data version information and pull the full service data from the HDFS according to the full data version information if a restart trigger event is detected;
and the target server loads the pulled incremental service data and the full service data into a memory for the client to access.
In one embodiment, the processor when executing the computer program further performs the steps of:
Acquiring full-volume fragmented service data from the HDFS, wherein the full-volume fragmented service data is obtained by performing data fragmentation on original full-volume service data;
Processing the full-volume fragmented service data by adopting an open addressing method to generate the full-volume service data with a preset data structure;
Storing the full traffic data to the HDFS, and generating third data basic information according to the full data version information of the full traffic data and a third storage address;
The third data basic information is used for the target server to acquire the third data basic information from the index server according to the full data version information; and the target server pulls the full service data corresponding to the full data version information from an HDFS storage position indicated by the third storage address according to the third storage address included in the third data basic information.
In one embodiment, the processor when executing the computer program further performs the steps of:
Acquiring the fragmentation mode information of the original full-volume service data, and detecting whether the fragmentation mode information is the same as local fragmentation mode information or not, wherein the local fragmentation mode information is related to the number of the plurality of service servers;
if the fragmentation mode information is the same as the local fragmentation mode information, executing the step that the index server acquires the full-quantity fragmentation service data from the HDFS;
And if the slicing mode information is different from the local slicing mode information, performing data slicing on the original full-volume service data according to the local slicing mode information to obtain the full-volume slicing service data, and executing the step that the index server acquires the full-volume slicing service data from the HDFS.
In one embodiment, the target server includes a first sliced data copy and a second sliced data copy, where the first sliced data copy and the second sliced data copy are used to provide the same data access service, and the target server is specifically used to perform data update on the first sliced data copy according to the incremental service data, and in the process of data update of the first sliced data copy, the target server provides the data access service through the second sliced data copy, and forbids the first sliced data copy in the process of data update from providing the data access service; after the data of the first sliced data copy is updated, the target server performs data updating on the second sliced data copy according to the incremental service data, and in the process of the data updating of the second sliced data copy, the target server provides data access service through the first sliced data copy after the data updating, and prohibits the second sliced data copy in the process of the data updating from providing data access service.
In one embodiment, the processor when executing the computer program further performs the steps of:
If a data address acquisition request sent by the client is received, determining a service address of the target server according to the data address acquisition request;
Returning the service address to the client; the service address is used for the client to send an access request for the incremental service data to the target server according to the service address; and the target server receives the access request and responds to the access request based on the increment business data loaded in the memory.
In one embodiment, the processor when executing the computer program further performs the steps of:
Analyzing the data address acquisition request to obtain a key code corresponding to the data address acquisition request;
and determining the service address of the target server corresponding to the key code from the plurality of service servers according to the key code.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
Acquiring original incremental business data, and performing data slicing on the original incremental business data to obtain incremental sliced business data;
processing the incremental fragment service data to obtain incremental service data with a preset data structure, and storing the incremental service data in a distributed file system (HDFS);
Updating data version information in a target server based on the incremental service data, and sending a data version update notification to the target server;
The target server is one service server of the plurality of service servers, the data version update notification is used for notifying the target server, the incremental service data corresponding to the updated data version information is pulled from the HDFS according to the updated data version information, and the incremental service data is loaded into a memory for access by a client.
In one embodiment, the computer program when executed by a processor further performs the following steps;
Generating first data basic information according to the updated data version information of the incremental service data and a first storage address of the incremental service data in the HDFS; the first data basic information is used for the target server to acquire the first data basic information from the index server according to the updated data version information; the first data basic information is used for the target server to pull the incremental service data corresponding to the updated data version information from an HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.
In one embodiment, the computer program when executed by the processor further performs the steps of:
And processing the incremental fragment service data by adopting a zipper method to obtain the incremental service data with a preset data structure.
In one embodiment, the service data processing system further includes a data source server, where the data source server is configured to obtain the original incremental service data, store the original incremental service data to the HDFS, and send second data basic information of the original incremental service data to the index server, where the second data basic information includes at least a second storage address of the original incremental service data in the HDFS;
correspondingly, the computer program when executed by the processor also realizes the following steps:
And pulling the original incremental business data from the HDFS storage position indicated by the second storage address according to the second storage address included in the second data basic information.
In one embodiment, the computer program when executed by the processor further performs the steps of:
analyzing the incremental fragment service data according to a preset data format to obtain analyzed incremental fragment service data;
And processing the analyzed increment fragment service data to obtain the increment service data with a preset data structure.
In one embodiment, the target server is configured to obtain full-volume data version information, and pull full-volume service data corresponding to the full-volume data version information from the HDFS according to the full-volume data version information, where the full-volume data version information is updated in the target server by the index server based on the full-volume service data; and the target server loads the full service data into a memory for the client to access.
In one embodiment, the target server is configured to pull the incremental service data from the HDFS according to the updated data version information and pull the full service data from the HDFS according to the full data version information if a restart trigger event is detected;
and the target server loads the pulled incremental service data and the full service data into a memory for the client to access.
In one embodiment, the computer program when executed by the processor further performs the steps of:
Acquiring full-volume fragmented service data from the HDFS, wherein the full-volume fragmented service data is obtained by performing data fragmentation on original full-volume service data;
Processing the full-volume fragmented service data by adopting an open addressing method to generate the full-volume service data with a preset data structure;
Storing the full traffic data to the HDFS, and generating third data basic information according to the full data version information of the full traffic data and a third storage address;
The third data basic information is used for the target server to acquire the third data basic information from the index server according to the full data version information; and the target server pulls the full service data corresponding to the full data version information from an HDFS storage position indicated by the third storage address according to the third storage address included in the third data basic information.
In one embodiment, the computer program when executed by the processor further performs the steps of:
Acquiring the fragmentation mode information of the original full-volume service data, and detecting whether the fragmentation mode information is the same as local fragmentation mode information or not, wherein the local fragmentation mode information is related to the number of the plurality of service servers;
if the fragmentation mode information is the same as the local fragmentation mode information, executing the step that the index server acquires the full-quantity fragmentation service data from the HDFS;
And if the slicing mode information is different from the local slicing mode information, performing data slicing on the original full-volume service data according to the local slicing mode information to obtain the full-volume slicing service data, and executing the step that the index server acquires the full-volume slicing service data from the HDFS.
In one embodiment, the target server includes a first sliced data copy and a second sliced data copy, where the first sliced data copy and the second sliced data copy are used to provide the same data access service, and the target server is specifically used to perform data update on the first sliced data copy according to the incremental service data, and in the process of data update of the first sliced data copy, the target server provides the data access service through the second sliced data copy, and forbids the first sliced data copy in the process of data update from providing the data access service; after the data of the first sliced data copy is updated, the target server performs data updating on the second sliced data copy according to the incremental service data, and in the process of the data updating of the second sliced data copy, the target server provides data access service through the first sliced data copy after the data updating, and prohibits the second sliced data copy in the process of the data updating from providing data access service.
In one embodiment, the computer program when executed by the processor further performs the steps of:
If a data address acquisition request sent by the client is received, determining a service address of the target server according to the data address acquisition request;
Returning the service address to the client; the service address is used for the client to send an access request for the incremental service data to the target server according to the service address; and the target server receives the access request and responds to the access request based on the increment business data loaded in the memory.
In one embodiment, the computer program when executed by the processor further performs the steps of:
Analyzing the data address acquisition request to obtain a key code corresponding to the data address acquisition request;
and determining the service address of the target server corresponding to the key code from the plurality of service servers according to the key code.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.