Disclosure of Invention
The embodiment of the disclosure provides a data processing method and device, which can solve the problem of low supervision efficiency in the data exchange process. The technical scheme is as follows:
According to a first aspect of embodiments of the present disclosure, there is provided a data processing method, the method comprising:
Applied to a file system, the method comprises:
acquiring target data and identification information corresponding to the target data;
Comparing the target data with the original data, and determining a target type corresponding to the target data in preset types according to a comparison result, wherein the preset types comprise: new type, delete type, modify type;
storing the target data to a corresponding target file directory in a preset file directory according to the target type and the identification information;
and generating a creation index instruction according to the target file directory, wherein the creation index instruction is used for indicating to create an index of the target data according to the target file directory corresponding to the target data.
In one embodiment, the method of the present disclosure further comprises, prior to acquiring the target data:
acquiring first data, wherein the first data comprises credit data;
copying the first data, generating the target data, and transmitting the target data to a file system;
the first data is stored to a first database or transmitted to a second database.
In one embodiment, the method of the present disclosure further comprises, prior to generating the create index instruction:
And creating a preset index directory according to the preset file directory, wherein the preset file directory is generated according to the identification information, the type and the time stamp corresponding to the historical data.
In one embodiment, the method of the present disclosure further comprises, after creating the index of the target data:
and generating an index file according to the index of the target data, wherein the index file comprises historical iteration information corresponding to the target data and classification statistical information corresponding to the target data.
In one embodiment, the disclosed method further comprises:
Receiving a user query instruction, wherein the query instruction comprises marking information of target data, and the user query instruction is used for indicating to query at least one item of historical iteration information classification statistical information corresponding to the target data or the target data;
and according to the query instruction, querying the index file, acquiring target data or at least one item of historical iteration information classification statistical information corresponding to the target data, and sending the information.
When the first data is acquired, the data processing method provided by the embodiment of the disclosure copies the first data to generate target data and sends the target data to a file system; on the other hand, the first data is sent to a database; after the file system acquires the target data, storing the target data into a target file directory according to the type and the identification information corresponding to the target data, and generating an index of the target data according to the target file directory; and finally, generating an index file according to the index of the target data, and searching the target data and information related to the target data according to the index file by a user.
According to a second aspect of embodiments of the present disclosure, there is provided a data processing apparatus comprising:
the device comprises an acquisition module, a comparison module, a storage module and a creation module;
The acquisition module is used for acquiring target data and identification information corresponding to the target data;
The comparison module is used for comparing the target data with the original data and determining a target type corresponding to the target data in a preset type according to a comparison result, wherein the preset type comprises: new type, delete type, modify type;
The storage module is used for storing the target data into a corresponding target file directory in a preset file directory according to the target type and the identification information, and generating a new file adding message;
the creation module is used for generating an index creation instruction according to the new file message, wherein the index creation instruction is used for indicating to create an index of the target data according to the target file directory corresponding to the target data.
In one embodiment, the present disclosure further provides the data processing apparatus, including a processor and a memory, where the memory stores at least one computer instruction, where the instruction is loaded and executed by the processor to implement the steps performed in the data processing method provided in the first aspect and any embodiment of the first aspect.
In one embodiment, the apparatus further comprises a computer readable storage medium having stored therein at least one computer instruction loaded and executed by a processor to implement the steps performed in the data processing method provided in the first aspect and any embodiment of the first aspect.
According to a third aspect of embodiments of the present disclosure, there is provided a data processing system comprising:
a first device and a second device, the first device and the second device being connected;
the first device is configured to obtain first data, where the first data includes credit data;
Copying the first data, generating the target data, and transmitting the target data to the second device;
storing the first data in a first database or transmitting the first data to a second database;
The second device is used for acquiring target data and identification information corresponding to the target data through a file system;
Comparing the target data with the original data, and determining a target type corresponding to the target data in preset types according to a comparison result, wherein the preset types comprise: new type, delete type, modify type;
storing the target data into a corresponding target file directory in a preset file directory according to the target type and the identification information, and generating a new file adding message;
And generating a creation index instruction according to the newly added file message, wherein the creation index instruction is used for indicating to create an index of the target data according to a target file directory corresponding to the target data.
In one embodiment, the system further comprises: the third device is arranged to be connected to the first device,
The third device is configured to generate an index file according to an index of the target data, where the index file includes historical iteration information corresponding to the target data and classification statistical information corresponding to the target data to obtain the index file;
Receiving a user query instruction, wherein the query instruction comprises marking information of target data, and the user query instruction is used for indicating to query at least one item of historical iteration information classification statistical information corresponding to the target data or the target data;
and inquiring the index file according to the inquiry instruction, and acquiring at least one item of target data or historical iteration information classification statistical information corresponding to the target data.
The embodiment of the disclosure provides a data processing system, when first equipment acquires first data, on one hand, the first data is copied to generate target data and sent to second equipment; on the other hand, the first data is sent to a database; after the second device acquires the target data, the second device processes the target data through the file system: storing the target data into a target file directory according to the type and the identification information corresponding to the target data, and generating an index of the target data according to the target file directory; the third device generates an index file from the index of the target data.
According to the system provided by the disclosure, after the data are acquired, the data are respectively stored to the first equipment and the second equipment, and the programs for data storage and data analysis are separated by adopting distributed deployment, so that the coupling degree is reduced, and the processing efficiency is improved. Meanwhile, a search engine of the target data is established through the third equipment, and an index file is generated, so that an analysis result and a change result of the target data can be quickly and accurately inquired, and the data inquiry efficiency is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
Example 1
An embodiment of the present disclosure provides a data processing method, as shown in fig. 1, where the data processing method is applied to a file system, and includes the following steps:
101. and acquiring target data and identification information corresponding to the target data.
The identification information corresponding to the target data comprises name information, service information and the time stamp corresponding to the target data. For example, the target data is business license data of the enterprise a, and when the time for acquiring the target data is 09 minutes 09 seconds of 09 months 09 of 2019, name information in the target data, namely, the name of the business license of the enterprise a; business information in the target data, namely content information of business license, the timestamp in the target data is 09 min 09 s of 09 s of 2019.
In an alternative embodiment, the method provided by the present disclosure further includes, before acquiring the target data:
The method comprises the steps of obtaining first data, wherein the first data comprises credit data, and the credit data comprises personal credit information data and market subject credit information data of enterprises, farmer professional cooperation communities, individual industrial merchants and the like.
Copying the first data, generating the target data, and transmitting the target data to a file system;
the first data is stored to a first database or transmitted to a second database.
For the above explanation, after the first data is acquired, the method provided by the present disclosure copies the first data to generate the target data and sends the target data to the file system, and stores the first data to the first database, so that the data storage and the data analysis are respectively processed in the database and the file system, thereby reducing the storage pressure of the service database; further, the file system and the first database can be arranged on one device or two devices, so that the data processing efficiency is further improved.
102. And comparing the target data with the original data, and determining the corresponding target type of the target data in the preset types according to the comparison result.
The preset types comprise a new type, a modified type, a deleted type and the like.
In an optional embodiment, when comparing the target data with the original data, the method provided by the disclosure may be according to identification information corresponding to the target data, where the identification information includes information such as name information, service information, and time stamp of the target data; searching the original data corresponding to the identification information; and finally, comparing the target data with the original data. Further, the original data can be searched through database technology SQL.
In an optional embodiment, the method provided by the present disclosure further includes creating preset type information, where the preset type information may include a new type, a modified type, and a deleted type; wherein the newly created type means that the target data is newly added based on the original data; the modification type refers to that the target data is modified based on the original data, and the deletion type refers to that the target data is deleted with a part of the original data based on the original data; when the target data is of a deletion type, the method provided by the present disclosure also obtains identification information of the deleted data, for example, data ID information, a name to which the data belongs, and data permission information.
In an alternative embodiment, after comparing the target data with the original data, the method provided by the present disclosure may create an intermediate temporary table according to the comparison result, and store the target data and the preset type information corresponding to the target data.
Further, the intermediate temporary table may be created in a server storing the target data, or may be stored in another server. And the service data and the exchange process data are stored separately, so that the storage pressure of the service database is reduced.
103. And storing the target data to a corresponding target file directory in a preset file directory according to the identification information and the target type.
In an alternative embodiment, after the target data is stored in a target file directory corresponding to the preset file directory, the method provided by the present disclosure may generate a new file adding message, where the new file adding message includes target file directory information and target data corresponding to the new target data.
The preset file directory and the identification information have a mapping relation.
In an alternative embodiment, the method provided by the present disclosure further includes, prior to generating the create index instruction:
And creating a preset index directory according to the preset file directory, wherein the preset file directory is generated according to the identification information, the type and the time stamp corresponding to the historical data.
In an alternative embodiment, the method provided by the present disclosure further includes creating a preset file directory, which may include multiple levels of subdirectories.
When the preset file directory is created, the preset file directory can be created based on identification information, types and time stamps corresponding to user requirements and historical data, for example, the preset file directory is a primary directory based on the province of the data, the preset file directory is a secondary directory based on the service of the data, and the preset file directory is a tertiary directory based on the type of the data.
If province information in the identification information of the target data is Shaanxi province, the business information is an A enterprise mining license, and the type of the data is a newly added type; then the target file directory corresponding to the target data is: newly added data folders in the mining license folder in the Shaanxi province folder.
The method provided by the disclosure stores data exchange process data according to a timestamp file catalog by storing target data in a preset folder, stores business data and exchange process data separately, and reduces the storage pressure of the business database
104. And generating a creation index instruction according to the target file directory.
The index creating instruction is configured to instruct a target file directory corresponding to the target data to create an index of the target data.
The creating rule of the preset index file may be that an index directory is created according to a preset file directory, and an index of the target data is created according to a type corresponding to the target data.
In an alternative embodiment, the method provided by the present disclosure creates an index of the target data when the newly added file message file is received, i.e., the target data and the index establish a mapping relationship.
In an alternative embodiment, the method provided by the present disclosure further includes, after creating the index of the target data:
and generating an index file according to the index of the target data, wherein the index file comprises historical iteration information corresponding to the target data and classification statistical information corresponding to the target data.
The example illustrates that the historical iteration information corresponding to the target data, for example, the target data includes credit data of the B enterprise, and then the historical iteration information of the target data may include information about a change process of the credit data of the B enterprise in a preset period of time: credit rating in credit data of business B2018 is good; the credit rating in the credit data of business B in 2019 is excellent. When the user selects the credit data of the B enterprise, the user can check the change process of the credit data of the B enterprise, and the aim of supervising the data exchange is fulfilled.
The listing examples illustrate classification statistics corresponding to the target data, for example, the target data includes credit data of C enterprise, credit data of D enterprise, and credit data of E person, and the classification standard of the classification statistics corresponding to the target data may be classified according to enterprise or person, and then the target data may be classified into two types: enterprise information and personal information, wherein the enterprise information comprises credit data of a C enterprise and credit data of a D enterprise; the personal information includes credit data of the E person.
The index file further includes: data exchange information, which may include: the data exchange information can obtain data exchange information according to the comparison result of the target data compared with the original data. If the target data comprises data A, data B and data C, the original data matched with the target data is data A and data B; according to the comparison result, the data A is the same data, the data B is the same data, the data C is the newly added data, and according to the preset rule, only the newly added data is received, and the total amount information of the exchanged data is the exchanged data at this time: the data exchange success information comprises the following data A, data B and data C: the successfully exchanged data is C data, and the failed information of the exchanged data is: and the data which are successfully exchanged are data A and data B.
The data quality information can be further generated for the data exchange information, for example, the data quality information is generated according to the success and failure ratio of the current received target data. If the quality of the data quality information display of some data is lower than a preset threshold value, a warning message can be generated to prompt the data sending equipment.
In an alternative embodiment, the method provided by the present disclosure further includes, after creating the index file:
Receiving a query instruction, wherein the query instruction comprises marking information of target data, and the user query instruction is used for indicating to query at least one item of historical iteration information classification statistical information corresponding to the target data;
and according to the query instruction, querying the index file, acquiring target data or at least one item of historical iteration information classification statistical information corresponding to the target data, and sending the information.
The query instruction may include an accurate search and a fuzzy search, for example, the user may only input the name of the a enterprise, that is, may retrieve the related data of the a enterprise, and the related data may include the data of the current a enterprise and the data of the historical a enterprise. Further data for the historic a enterprise may include the number of historic changes stored according to the data entry timestamp, as well as detailed information for each change.
The method provided by the disclosure comprises the steps of creating an index directory according to a file directory structure, and creating an index of target data according to a target file directory corresponding to the target data. The distributed deployment is adopted, so that analysis of data exchange results and a program for recording exchange data are separated, the coupling degree is reduced, and the processing efficiency is improved
When the first data is acquired, the data processing method provided by the embodiment of the disclosure copies the first data to generate target data and sends the target data to a file system; on the other hand, the first data is sent to a database; after the file system acquires the target data, storing the target data into a target file directory according to the type and the identification information corresponding to the target data, and generating an index of the target data according to the target file directory; and finally, generating an index file according to the index of the target data, and searching the target data and information related to the target data according to the index file by a user.
According to the method, after the data are acquired, the data are respectively stored in the database and the file system, and the programs for data storage and data analysis are separated by adopting distributed deployment, so that the coupling degree is reduced, and the processing efficiency is improved. Meanwhile, by establishing a search engine of the target data, an index file is generated, so that an analysis result and a change result of the target data can be quickly and accurately inquired, and the data inquiry efficiency is improved.
Example two
Based on the data processing method described in the foregoing corresponding embodiment of fig. 1, the following is an embodiment of the apparatus of the present disclosure, which may be used to execute an embodiment of the method of the present disclosure.
An embodiment of the present disclosure provides a data processing apparatus, as shown in fig. 2, the data processing apparatus 20 includes: comprising the following steps: an acquisition module 201, a comparison module 202, a storage module 203 and a creation module 204;
The acquiring module 201 is configured to acquire target data and identification information corresponding to the target data;
the comparing module 202 is configured to compare the target data with the original data, and determine, according to a comparison result, a target type corresponding to the target data in a preset type, where the preset type includes: new type, delete type, modify type;
The storage module 203 is configured to store the target data into a corresponding target file directory in a preset file directory according to the target type and the identification information, and generate a new file adding message;
The creation module 204 is configured to generate, according to the new file message, a creation index instruction, where the creation index instruction is used to instruct to create, according to a target file directory corresponding to the target data, an index of the target data.
In one embodiment, the disclosed apparatus further comprises a data processing apparatus, wherein the data processing apparatus comprises a processor and a memory, the memory having stored therein at least one computer instruction that is loaded and executed by the processor to implement the steps performed in the data processing method provided in the first and any examples of the embodiments.
In one embodiment, the disclosed apparatus further includes a computer readable storage medium having stored therein at least one computer instruction that is loaded and executed by a processor to implement the steps performed in the data processing method provided in the first embodiment and any of the embodiments.
When the data processing device provided by the embodiment of the disclosure acquires first data, on one hand, the first data is copied to generate target data and sent to a file system; on the other hand, the first data is sent to a database; after the file system acquires the target data, storing the target data into a target file directory according to the type and the identification information corresponding to the target data, and generating an index of the target data according to the target file directory; and generating an index file according to the index of the target data.
According to the device provided by the disclosure, after the data is acquired, the data are respectively stored in the database and the file system, and the programs for data storage and data analysis are separated by adopting distributed deployment, so that the coupling degree is reduced, and the processing efficiency is improved. Meanwhile, by establishing a search engine of the target data, an index file is generated, so that an analysis result and a change result of the target data can be quickly and accurately inquired, and the data inquiry efficiency is improved.
Example III
Based on the data processing method described in the above corresponding embodiment of fig. 1, the following is an embodiment of the system of the present disclosure, which may be used to execute the embodiment of the method of the present disclosure.
The disclosed embodiments provide a data processing system, as shown in FIG. 3, the data processing system 30, comprising: the first device 301 and the second device 302 are connected, and the connection manner may include a hardware connection and a software connection.
The first device 301 may be a terminal device, a network server, or the like, and the second device 302 may be a terminal device, a network server, or the like.
The first device 301 is configured to obtain first data, where the first data includes credit data;
copying the first data, generating the target data, and transmitting the target data to the second device 302;
storing the first data in a first database or transmitting the first data to a second database;
The second device 302 is configured to obtain, through a file system, target data and identification information corresponding to the target data;
Comparing the target data with the original data, and determining a target type corresponding to the target data in preset types according to a comparison result, wherein the preset types comprise: new type, delete type, modify type;
storing the target data into a corresponding target file directory in a preset file directory according to the target type and the identification information, and generating a new file adding message;
And generating a creation index instruction according to the newly added file message, wherein the creation index instruction is used for indicating to create an index of the target data according to a target file directory corresponding to the target data.
In one embodiment, the present system 30 further comprises: a third device 303, which may be a terminal device or a network server or the like,
The third device 303 is configured to generate an index file according to an index of the target data, where the index file includes historical iteration information corresponding to the target data and classification statistical information corresponding to the target data;
Receiving a user query instruction, wherein the query instruction comprises marking information of target data, and the user query instruction is used for indicating to query at least one item of historical iteration information classification statistical information corresponding to the target data or the target data;
and inquiring the index file according to the inquiry instruction, and acquiring at least one item of target data or historical iteration information classification statistical information corresponding to the target data.
In an alternative embodiment, the third device 303 may be the same device as the second device 302, or may not be the same device as the second device 302, that is, the target data and the index file are not stored in the same device, so as to improve the processing efficiency of the data and reduce the pressure of data storage.
The embodiment of the disclosure provides a data processing system, when first equipment acquires first data, on one hand, the first data is copied to generate target data and sent to second equipment; on the other hand, the first data is sent to a database; after the second device acquires the target data, the second device processes the target data through the file system: storing the target data into a target file directory according to the type and the identification information corresponding to the target data, and generating an index of the target data according to the target file directory; the third device generates an index file from the index of the target data.
According to the system provided by the disclosure, after the data are acquired, the data are respectively stored to the first equipment and the second equipment, and the programs for data storage and data analysis are separated by adopting distributed deployment, so that the coupling degree is reduced, and the processing efficiency is improved. Meanwhile, a search engine of the target data is established through the third equipment, and an index file is generated, so that an analysis result and a change result of the target data can be quickly and accurately inquired, and the data inquiry efficiency is improved.
Based on the data processing method described in the above embodiment corresponding to fig. 1, the embodiment of the present disclosure further provides a computer readable storage medium, for example, a non-transitory computer readable storage medium may be a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. The storage medium stores computer instructions for executing the data processing method described in the corresponding embodiment of fig. 1, which is not described herein.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.