Disclosure of Invention
In view of this, the present invention provides a method for querying data information, which can improve the efficiency of querying data based on a relational database; another object of the present invention is to provide a data information query apparatus, a device and a computer readable storage medium, all of which have the above advantages.
In order to solve the above technical problem, the present invention provides a method for querying data information, including:
screening data to be dumped from data information pre-stored in a relational database;
storing the data to be dumped into a non-relational database, and deleting the data to be dumped in the relational database;
when a query instruction is received, judging whether target data information corresponding to the query instruction exists in the relational database or not;
if yes, performing information feedback on the query instruction by using the target data information;
if not, determining target data information corresponding to the query instruction from the non-relational database, and feeding back the information of the query instruction by using the target data information.
Preferably, the process of storing the data to be dumped into the non-relational database specifically includes:
acquiring a data identifier of the data to be dumped;
determining corresponding map information according to each key value pair of the data to be dumped;
transmitting the data identification and the map information into a put data type of Phoenix to obtain a corresponding data record;
and calling an insertion method in the Phoenix to insert the data record into the non-relational database.
Preferably, after the calling the insertion method in Phoenix to insert the data record into the non-relational database, the method further comprises:
and performing information verification on the data records, and entering the step of deleting the data to be dumped in the relational database under the condition that the data records pass the verification.
Preferably, before the step of transmitting the data identifier and the map information into a put data type of Phoenix to obtain a corresponding data record, the method further includes:
judging whether the data record is prestored in the non-relational database;
if yes, deleting the data to be dumped corresponding to the data records in the relational database;
if not, the step of calling the insertion method in the Phoenix to insert the data record into the non-relational database is carried out.
Preferably, the process of screening data to be dumped from data information pre-stored in the relational database specifically includes:
and screening the data to be dumped from the data information pre-stored in the relational database according to a preset time period.
Preferably, the process of storing the data to be dumped into the non-relational database specifically includes:
grouping the data to be dumped according to the preset data quantity of each group;
and utilizing a plurality of threads to store the data to be dumped corresponding to each group into the non-relational database respectively in parallel.
Preferably, the preset data amount is 1000.
In order to solve the above technical problem, the present invention further provides an apparatus for querying data information, including:
the screening module is used for screening data to be dumped from data information pre-stored in the relational database;
the storage module is used for storing the data to be dumped into a non-relational database and deleting the data to be dumped in the relational database;
the first judgment module is used for judging whether target data information corresponding to a query instruction exists in the relational database or not when the query instruction is received; if yes, calling a first query module; if not, calling a second query module;
the first query module is used for performing information feedback on the query instruction by using the target data information;
and the second query module is used for determining target data information corresponding to the query instruction from the non-relational database and feeding back the information of the query instruction by using the target data information.
In order to solve the above technical problem, the present invention further provides a data information query device, including:
a memory for storing a computer program;
and the processor is used for realizing the steps of any data information inquiry method when the computer program is executed.
In order to solve the above technical problem, the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps of any one of the above data information query methods.
The data information query method provided by the embodiment of the invention comprises the steps of screening data to be dumped from a relational database in advance, storing the data to be dumped into a non-relational database, deleting the data to be dumped from the relational database, querying corresponding target data information from the relational database when a query instruction is received, and querying corresponding target data information from the non-relational database under the condition that the target data information is not found; the data volume of the data information stored in the relational database is reduced, so that the response speed of inquiring the target data information from the relational database can be improved, and the data inquiry efficiency based on the relational database can be improved; and the speed of inquiring data information from the non-relational database is high, so that the overall data inquiring efficiency can be improved.
In order to solve the technical problem, the invention also provides a data information query device, equipment and a computer readable storage medium, which have the beneficial effects.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The core of the embodiment of the invention is to provide a data information query method which can improve the data query efficiency based on a relational database; another core of the present invention is to provide a data information query apparatus, a device and a computer readable storage medium, all having the above beneficial effects.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Fig. 1 is a flowchart of a data information query method according to an embodiment of the present invention. As shown in fig. 1, a method for querying data information includes:
s10: screening data to be dumped from data information pre-stored in a relational database;
s20: and storing the data to be dumped into a non-relational database, and deleting the data to be dumped in the relational database.
In this embodiment, the data information is pre-stored in the relational database, and the data to be dumped is screened from the data information, stored in the non-relational database, and then deleted from the relational database, so as to transfer and store the data to be dumped from the relational database into the non-relational database.
It should be noted that the relational database refers to a database that organizes data using a relational model; the relational model refers to a two-dimensional table model, and the corresponding relational database is a data organization composed of database tables and the relations between the database tables and the relational database tables. A non-relational database refers to a non-relational, distributed data storage system; a non-relational database stores data information in key-value pairs, each piece of data information may have a different field and is not limited to a fixed structure. In addition, the relational database generally uses a hard disk to store data, so that the storage cost is low; the non-relational database mainly stores data information in a memory, so that the storage cost is high.
More specifically, the relational database includes Oracle, SQL Server, db2, infomix, Sybase, MySQL, Postpresql, Access, SQL Anywhere, sqlite, interbase, etc.; the non-relational database comprises Redis, Memcached, Apache Ignite, Riak, Hbase and the like; the present embodiment does not limit the specific types of the relational database and the non-relational database.
In this embodiment, the data information is divided into cold data and hot data by predicting how frequently the data information is used, and the cold data is stored as data to be dumped in the non-relational database. More specifically, assuming that the data information is historical order data, setting a preset time point for distinguishing cold data from hot data, and classifying the data exceeding the preset time point into cold data; for example, hot data may be data within a half year that users often query; cold data is data half a year ago that users may rarely use.
It should be noted that if the data to be dumped is data information such as historical order data and operation records which is not frequently queried and is hardly changed, hbase (hadoop database) may be preferably used as a non-relational database for storing the data to be dumped; HBase is a distributed storage system with high reliability, high performance, orientation and scalability, and supports high availability and mass data and large instant write quantity; HBase is a non-relational database, but uses a hard disk to store it, so that storage cost can be further reduced.
It should be noted that, in this embodiment, MyBitis is preferably used to read corresponding data to be dumped from the relational database, and MyBitis has an advantage that most of codes can be automatically generated, and is easy to implement.
S30: when a query instruction is received, judging whether target data information corresponding to the query instruction exists in the relational database or not; if yes, go to S40; if not, go to S50;
s40: performing information feedback on the query instruction by using the target data information;
s50: and determining target data information corresponding to the query instruction from the non-relational database, and performing information feedback on the query instruction by using the target data information.
Specifically, when a query instruction is received, it indicates that target data information corresponding to the query instruction needs to be searched from pre-stored data information at present; firstly, searching data from a relational database according to a query instruction, and judging whether target data information corresponding to the query instruction exists in the relational database; if yes, performing information feedback on the query instruction by using the searched target data information; if not, data searching is further needed from the non-relational database according to the query instruction, whether target data information corresponding to the query instruction exists in the non-relational database or not is judged, and if yes, the searched target data information is utilized to perform information feedback on the query instruction; if not, the relational database and the non-relational database do not have target data information corresponding to the query instruction, so that invalid report information can be fed back for query according to the query instruction.
In addition, in the actual operation, the program code corresponding to "screen the data to be dumped from the data information pre-stored in the relational database" and the program code corresponding to "store the data to be dumped into the non-relational database, and delete the data to be dumped from the relational database" may be written in one Java method or separately written in two Java methods, so as to implement the above steps in one Java web project, thereby making the operation more convenient; or the program codes for realizing the steps can be separately written in two Java methods and deployed to two servers, and the steps can be realized by using two JavaWeb projects; and in this case two servers running java web items each transmit data via Json messages.
The data information query method provided by the embodiment of the invention comprises the steps of screening data to be dumped from a relational database in advance, storing the data to be dumped into a non-relational database, deleting the data to be dumped from the relational database, querying corresponding target data information from the relational database when a query instruction is received, and querying corresponding target data information from the non-relational database under the condition that the target data information is not found; the data volume of the data information stored in the relational database is reduced, so that the response speed of inquiring the target data information from the relational database can be improved, and the data inquiry efficiency based on the relational database can be improved; and the speed of inquiring data information from the non-relational database is high, so that the overall data inquiring efficiency can be improved.
On the basis of the foregoing embodiment, this embodiment further describes and optimizes the technical solution, and specifically, in this embodiment, the process of storing the data to be dumped into the non-relational database specifically includes:
acquiring a data identifier of data to be dumped;
determining corresponding map information according to each key value pair of the data to be dumped;
transmitting the data identification and the map information into a put data type of Phoenix to obtain a corresponding data record;
and calling an insertion method in Phoenix to insert the data records into the non-relational database.
In this embodiment, first, a data identifier of data to be dumped needs to be obtained, and corresponding map information is determined according to each key value pair of the data to be dumped. Specifically, the data identifier is used to uniquely identify each different data to be dumped, such as a data ID; the Key-Value pair (Key-Value) includes a field name and an attribute Value; wherein, the field names are different attributes of the data to be dumped, for example, in the historical order data, the field names include information such as transaction time, price, quantity, etc.; the attribute value is a specific value corresponding to a field name (attribute). That is, the map information is generated by individual key-value pairs composed of field names and attribute values in the data to be dumped.
And then, storing the data to be dumped into a non-relational database by utilizing Phoenix. Specifically, the data identification and the map information are transmitted into a put data type of Phoenix to obtain a corresponding data record, and an insertion method in Phoenix is called to insert the data record into a non-relational database. The Phoenix has the advantages that compared with other tools, the Phoenix packages the bottom layer of a non-relational database such as HBase by one layer, is more similar to SQL of common relational data in use, and is low in learning cost.
In addition, when the time information is stored in the relational database Mysql in a Datetime type, in order to avoid a dump error when the time information is dumped into the non-relational database, in an actual operation, the time information may be first converted into milliseconds and then stored.
Therefore, according to the method of the embodiment, the data to be dumped can be efficiently transferred from the relational database to the non-relational database.
On the basis of the foregoing embodiment, this embodiment further describes and optimizes the technical solution, and specifically, in this embodiment, the process of storing the data to be dumped into the non-relational database specifically includes:
grouping the data to be dumped according to the preset data quantity of each group;
and utilizing a plurality of threads to respectively store the data to be dumped corresponding to each group into a non-relational database in parallel.
It should be noted that, in this embodiment, in consideration of the situation that the data amount of the data to be dumped is large, after the data to be dumped is screened out from the data information, the data to be dumped is firstly grouped according to the preset data amount, that is, the data to be dumped is divided into a plurality of groups, where the data amount in each group is the preset data amount; and then, a plurality of threads are utilized to execute dump operation in parallel, each thread corresponds to one group, and the data to be dumped in the group is stored in a non-relational database.
In a preferred embodiment, the predetermined number of data is 1000.
It should be noted that when the Phoenix inserts a data record into the HBase, one HBase link needs to be opened every time of insertion, and the link needs to be closed after the insertion is completed, so that if too many data records are inserted every time, the insertion speed is affected; because resources of HBase are consumed for opening and closing the link, if the number of data records inserted after the link is opened is too small, more resources are consumed for opening and closing the link for multiple times; therefore, the preset data quantity is preferably set to 1000 in the embodiment, that is, when the inserting method in the Phoenix is called to insert the data record into the non-relational database, the Phoenix sends 1000 pieces of data to the HBase each time; the method can efficiently insert the data to be dumped into the HBase and relatively save resources.
On the basis of the above embodiment, the present embodiment further describes and optimizes the technical solution, and specifically, after the inserting method in Phoenix is called to insert the data record into the non-relational database, the present embodiment further includes:
and performing information verification on the data records, and entering a step of deleting the data to be dumped in the relational database if the data records pass the verification.
It should be noted that, in this embodiment, after the data record is inserted into the non-relational database, information verification may be further performed on the data record, including verifying the integrity of the key value pair in the data record and verifying the security of the data record, and the specific content of the information verification is not limited in this embodiment. If the verification is passed, the current dump is successful, and the corresponding data to be dumped in the relational database can be deleted; if the verification is not passed, the current dump is not successful, and the data dump operation needs to be performed again.
It should be noted that, a specific verification method may be to obtain a standard field name corresponding to the data record in advance, and then determine whether the current field name in the currently received data record corresponds to the standard field name one by one, and whether each attribute value corresponding to the current field name is empty, if so, it indicates that the key value pair in the data record is complete.
Therefore, the safety and reliability of storing the data to be dumped into the non-relational database can be further guaranteed by further performing information verification on the data records.
It should be noted that, before the data identifier and the map information are transmitted into the put data type of Phoenix to obtain the corresponding data record, the method further includes:
judging whether a data record is prestored in the non-relational database;
if so, deleting the data to be dumped corresponding to the data records in the relational database;
if not, the step of calling an insertion method in Phoenix to insert the data record into the non-relational database is carried out.
In this embodiment, after the data identifier and the map information are transmitted into a put data type of Phoenix to obtain a corresponding data record, further verifying whether the data record is stored in the non-relational database in advance; if so, directly deleting the data to be dumped corresponding to the data records in the relational database without repeatedly storing the corresponding data records; otherwise, the data record is stored in the non-relational database for the first time, and therefore the step of calling the insertion method in Phoenix to insert the data record into the non-relational database is carried out.
Therefore, in the embodiment, by further judging whether the non-relational database stores the data records in advance, repeated data record storage can be avoided, storage resources of the non-relational database are saved, and storage efficiency is improved.
On the basis of the foregoing embodiment, this embodiment further describes and optimizes the technical solution, and specifically, in this embodiment, a process of screening data to be dumped from data information pre-stored in a relational database specifically includes:
and screening data to be dumped from the data information pre-stored in the relational database according to a preset time period.
It should be noted that, in actual operation, different database reading methods are called according to different ways of starting tasks, and corresponding dump data is determined from the relational database. Specifically, data to be dumped can be screened from data information pre-stored in the relational database in a manual starting or timing task starting mode, so that data dumping operation can be performed subsequently.
Specifically, the manual starting refers to that a corresponding interface of the Javeweb service is called according to a triggering instruction of a user, a database reading method of a Java bottom layer is triggered and called, and corresponding data information in a relational database is read; the instruction of manual start is "dump data information from 5 month 1 day 0 to 6 month 1 day 0 to HBase". The starting of the timing task refers to presetting a time period for starting the timing task, generating a trigger instruction in each time period, calling a corresponding interface of the Javeweb service according to the trigger instruction, triggering and calling a database reading method at the bottom layer of the Java, and reading corresponding data information in a relational database; instructions for starting a timing task such as "automatically dumping an order before 100 days to HBase at 8 o' clock each day"; in addition, in the actual operation, the time length of the preset time period is set according to the actual requirement, which is not limited in this embodiment.
The embodiment preferably screens the data to be dumped from the data information pre-stored in the relational database according to a preset time period, so that manual operation of a user can be avoided, the convenience of data dumping is improved, and the convenience of data query is further improved.
The above detailed description is given for the embodiments of the method for querying data information provided by the present invention, and the present invention further provides a device, an apparatus, and a computer-readable storage medium for querying data information corresponding to the method.
Fig. 2 is a structural diagram of an apparatus for querying data information according to an embodiment of the present invention, and as shown in fig. 2, the apparatus for querying data information includes:
thescreening module 21 is configured to screen data to be dumped from data information pre-stored in the relational database;
thestorage module 22 is configured to store data to be dumped into a non-relational database, and delete the data to be dumped in the relational database;
the first judgingmodule 23 is configured to, when receiving the query instruction, judge whether target data information corresponding to the query instruction exists in the relational database; if yes, calling thefirst query module 24; if not, thesecond query module 25 is called;
thefirst query module 24 is configured to perform information feedback on the query instruction by using the target data information;
and thesecond query module 25 is configured to determine target data information corresponding to the query instruction from the non-relational database, and perform information feedback on the query instruction by using the target data information.
The data information query device provided by the embodiment of the invention has the beneficial effects of the data information query method.
As a preferred embodiment, the storage module specifically includes:
the obtaining submodule is used for obtaining a data identifier of data to be dumped;
the determining submodule is used for determining corresponding map information according to each key value pair of the data to be dumped;
the transmitting submodule is used for transmitting the data identification and the map information into a put data type of Phoenix to obtain a corresponding data record;
and the inserting submodule is used for calling an inserting method in Phoenix to insert the data records into the non-relational database.
As a preferred embodiment, an apparatus for querying data information further includes:
and the verification module is used for performing information verification on the data records and calling the storage module to delete the data to be dumped in the relational database under the condition that the data records are verified to be passed.
As a preferred embodiment, an apparatus for querying data information further includes:
the second judgment module is used for judging whether the non-relational database stores data records in advance; if yes, calling a first execution module; if not, calling an insertion submodule;
and the first execution module is used for deleting the data to be dumped corresponding to the data records in the relational database.
As a preferred embodiment, the storage module specifically includes:
the grouping submodule is used for grouping the data to be dumped according to the preset data quantity of each group;
and the storage submodule is used for utilizing a plurality of threads to respectively store the data to be dumped corresponding to each group into the non-relational database in parallel.
Fig. 3 is a structural diagram of a data information query device according to an embodiment of the present invention, and as shown in fig. 3, the data information query device includes:
amemory 31 for storing a computer program;
and aprocessor 32, configured to implement the steps of the data information query method as described above when executing the computer program.
The data information query device provided by the embodiment of the invention has the beneficial effects of the data information query method.
In order to solve the above technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the above data information query method.
The computer-readable storage medium provided by the embodiment of the invention has the beneficial effect of the data information query method.
The present invention provides a method, an apparatus, a device and a computer readable storage medium for querying data information. The principles and embodiments of the present invention are explained herein using specific examples, which are set forth only to help understand the method and its core ideas of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.