CN103034650A - System and method for processing data - Google Patents
System and method for processing dataDownload PDFInfo
- Publication number
- CN103034650A CN103034650ACN2011103007259ACN201110300725ACN103034650ACN 103034650 ACN103034650 ACN 103034650ACN 2011103007259 ACN2011103007259 ACN 2011103007259ACN 201110300725 ACN201110300725 ACN 201110300725ACN 103034650 ACN103034650 ACN 103034650A
- Authority
- CN
- China
- Prior art keywords
- data
- user behavior
- query
- analysis
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545processingMethods0.000titleclaimsabstractdescription30
- 238000000034methodMethods0.000titleclaimsabstractdescription12
- 238000004458analytical methodMethods0.000claimsabstractdescription56
- 230000006399behaviorEffects0.000claimsdescription180
- 230000002093peripheral effectEffects0.000claimsdescription45
- 238000007405data analysisMethods0.000claimsdescription17
- 238000004364calculation methodMethods0.000claimsdescription15
- 238000012544monitoring processMethods0.000claimsdescription6
- 238000010586diagramMethods0.000description6
- 238000003672processing methodMethods0.000description5
- 230000009286beneficial effectEffects0.000description4
- 238000011161developmentMethods0.000description2
- 238000013459approachMethods0.000description1
- 230000005540biological transmissionEffects0.000description1
- 238000004891communicationMethods0.000description1
- 238000007418data miningMethods0.000description1
- 230000007547defectEffects0.000description1
- 238000005516engineering processMethods0.000description1
- 238000004880explosionMethods0.000description1
- 238000013508migrationMethods0.000description1
- 230000005012migrationEffects0.000description1
- 238000012986modificationMethods0.000description1
- 230000004048modificationEffects0.000description1
- 238000011160researchMethods0.000description1
- 238000007619statistical methodMethods0.000description1
- 230000009897systematic effectEffects0.000description1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The invention relates to the technical field of internet, in particular to a data processing system and a data processing method.
Background
In the current information explosion era, analysis processing on massive data is generally required. Such as data analysis of large websites in the internet, and data computation in scientific research.
In general data analysis processing, a highly configured relational database is used for storing data, and the data is cleaned depending on the computing power of the database. When this approach relies on a relational database, there are the following disadvantages: the system load is large, the bottleneck of the system is easily caused, and the upgrading and the expansion are not easy to realize; poor calculation power for data above TB level; the storage cost of the original data is high, and the utilization and the migration are not facilitated; less intelligent operation and high manual participation; in the relational data table, the relevance between the records is poor, that is, a plurality of behaviors of the same user need to be stored as a plurality of records, so that the relationship between the user and the behaviors is relatively loose, the analysis and statistics of the conditions of multiple behaviors are not facilitated, and the query of a single user behavior is not facilitated.
Disclosure of Invention
The invention provides a data processing system which is strong in computing capability, easy to expand, high in automation degree and beneficial to query.
The invention also provides a data processing method, which improves the computing power of the data processing system, and ensures that the system is easy to expand, has high automation degree and is more beneficial to query.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
the invention discloses a data processing system, which comprises: the system comprises a data acquisition module, an analysis system, a user behavior database, a network database, a data cache module, a key value cache module and a query module;
the data acquisition module is used for receiving data pushed by the first type of peripheral system in real time, monitoring the second type of peripheral system, and actively acquiring the data from the second type of peripheral system when the second type of peripheral system is idle; sending the data to an analysis system;
the analysis system is used for analyzing the user behavior data and the non-user behavior data from the received data; analyzing the user behavior data to obtain user, behavior and result data; establishing a corresponding relation among the user, the behavior and the result; wherein the result is a specific behavior content; analyzing the non-user behavior data according to the structure attribute of the network database;
the user behavior database is used for storing the corresponding relation among the user, the behavior and the result in a tree structure to obtain a user behavior data tree; wherein, the user is a root node, the behavior is a branch node under the root node, and the result is a branch node under the behavior node;
the network database is used for storing the non-user behavior data according to the structure attribute;
the key value cache module is used for storing query key values of data stored in the user behavior database and the network database;
the data cache module is used for storing the queried query key values and the corresponding data;
the query module is used for receiving a query request of an external application, analyzing a query key value in the query request, querying whether the query key value exists in the key value cache module, if not, returning a prompt of illegal query to the external application, if so, further querying whether the query key value and corresponding data exist in the data cache module, if so, returning the corresponding data in the data cache to the external application, otherwise, querying in a user behavior database and a network database, and returning the queried data to the external application.
The invention also discloses a data processing method, which comprises the following steps:
receiving data pushed by a first type of peripheral system in real time, monitoring a second type of peripheral system, and actively acquiring the data from the second type of peripheral system when the second type of peripheral system is idle;
analyzing user behavior data and non-user behavior data from the data; analyzing the user behavior data to obtain user, behavior and result data; establishing a corresponding relation among the user, the behavior and the result; wherein the result is a specific behavior content; analyzing the non-user behavior data according to the structure attribute of the network database;
storing the corresponding relation among the user, the behavior and the result in a tree structure to obtain a user behavior data tree, and storing the user behavior data tree in a user behavior database; wherein, the user is a root node, the behavior is a branch node under the root node, and the result is a branch node under the behavior node;
storing the non-user behavior data into a network database according to the structure attribute;
caching query key values of data stored in a user behavior database and a network database in a key value cache, and caching the queried query key values and corresponding data in a data cache;
when a query request of an external application is received, analyzing a query key value in the query request, querying whether the query key value exists in a key value cache, if not, returning a prompt of illegal query to the external application, if so, further querying whether the query key value and corresponding data exist in a data cache, if so, returning the corresponding data in the data cache to the external application, otherwise, querying in a user behavior database and a network database, and returning the queried data to the external application.
As can be seen from the above, the data processing system of the present invention includes: the data acquisition module is used for sending the acquired data to the analysis system; the analysis system analyzes the user behavior data and the non-user behavior data from the received data; the user behavior database stores the user behavior data in a tree structure to obtain a user behavior data tree; the network database stores the non-user behavior data according to the structure attribute; the key value cache module stores the query key values of the data stored in the database; the data cache module stores the queried query key values and corresponding data; the query module analyzes and searches the query key value in the key value cache module, returns an illegal query prompt if the query key value does not exist, and queries in the data cache module and returns the query prompt to an external application or queries in the database if the query key value exists. According to the technical scheme, the computing capacity of the data processing system is improved, and the system is easy to expand, high in automation degree and more beneficial to query.
Drawings
FIG. 1 is a schematic diagram of an improved data analysis processing system of the prior art;
FIG. 2 is a block diagram of a data processing system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data processing system, its peripheral systems, and the flow of data in the present invention;
fig. 4 is a flow chart of a data processing method in the present invention.
Detailed Description
In implementing the present invention, the inventor has conducted a systematic study on a data analysis processing technology solution, for which distributed cloud computing can be implemented by using an inexpensive PC, and an HIVE is used to simulate an operation mode of a relational database, as shown in fig. 1.
FIG. 1 is a schematic diagram of an improved data analysis processing system. As shown in fig. 1:
1. the data access platform transmits the received data to the hadoop platform at regular time;
2. and synchronizing the information of the completion of the data transmission to the application system of the data platform. Informing a data platform by adopting a protoBuffer communication mode of google;
3. the application system of the data platform reads information from the scheduling system after receiving the information, and judges whether to start calling Hive statements for calculation and the calling sequence;
4. the Hive system reads metadata and data of the hadoop platform from the mysql relational database according to the received information and associates the metadata and the data;
5. hive reads data stored by the computing hadoop platform and generates a result file;
6. importing a result file generated by Hive calculation into a mysql database;
7. extracting partial data from the MyS QL database and putting the data into the memcache;
8. the data display platform reads data from the memcache to display;
9. and the data display platform reads data from the MyS QL database for display.
Although the scheme shown in fig. 1 has a certain improvement, the reason for thinking position inherits the existing operation mode, resulting in poor expansibility; the real-time data is not specially processed, so that the scheme is only suitable for partial fields of data mining services; the data query performance is poor, and the automation degree of the system is not enough.
Therefore, the invention provides a brand-new data processing system to overcome the defects of the existing system.
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific embodiments.
Fig. 2 is a schematic structural diagram of a data processing system according to an embodiment of the present invention. As shown in fig. 2, the system includes: the system comprises adata routing module 201, adata acquisition module 202, aparsing system 203, auser behavior database 204, anetwork database 205, adata caching module 206, a keyvalue caching module 207 and aquery module 208; wherein,
thedata acquisition module 202 is configured to receive data pushed by a first type of peripheral system in real time, monitor a second type of peripheral system, and actively acquire data from the second type of peripheral system when the second type of peripheral system is idle; sending the data to theparsing system 203;
specifically, thedata obtaining module 202 receives data pushed by the first type of peripheral system in real time through thedata routing module 201, and actively obtains user behavior data from the second type of peripheral system through thedata routing module 201;
thedata routing module 201 is used for converting the data from the first type peripheral system and the second type peripheral system into data conforming to the data processing system and then transmitting the data to thedata acquisition module 202;
theanalysis system 203 is used for analyzing the user behavior data and the non-user behavior data from the received data; analyzing the user behavior data to obtain user, behavior and result data; establishing a corresponding relation among the user, the behavior and the result; wherein the result is a specific behavior content; analyzing the non-user behavior data according to the structure attribute of the network database;
the data pushed by the first type of peripheral system in real time comprises user behavior data, and the data format of the user behavior data is data in a format of 'user, behavior and result'; the data formats acquired from the second type of peripheral system are various, and theanalysis system 203 needs to analyze the user behavior data in various formats to obtain data in a unified format, such as "user, behavior, and result".
In an embodiment of the present invention, theanalysis system 203 is a Hadoop cluster system, and adopts a greenplus calculation method.
Theuser behavior database 204 is used for storing the corresponding relation among the user, the behavior and the result in a tree structure to obtain a user behavior data tree; wherein, the user is a root node, the behavior is a branch node under the root node, and the result is a branch node under the behavior node;
thenetwork database 205 is used for storing the non-user behavior data according to the structure attribute;
a keyvalue cache module 207, configured to store query key values of data stored in the user behavior database and the network database;
adata cache module 206, configured to store queried query key values and corresponding data;
thequery module 208 is configured to receive a query request of an external application, analyze a query key value in the query request, query whether the query key value exists in the keyvalue cache module 207, if the query key value does not exist, return a prompt of an illegal query to the external application, if the query key value exists, further query whether the query key value and corresponding data exist in thedata cache module 206, if the query key value and corresponding data exist, return the corresponding data in the data cache to the external application, otherwise, query in theuser behavior database 204 and thenetwork database 205, and return the queried data to the external application.
In the system shown in figure 2 of the drawings,
thedata acquisition module 202 receives data pushed by the first type of peripheral system in real time in a service interface mode;
thedata obtaining module 202 is configured to monitor an application log system, an application system backup library, and a web crawler system, and actively obtain data from the application log system, the application system backup library, and the web crawler system when the systems are idle.
As shown in fig. 2, the system further includes adata analysis module 209, configured to traverse the user behavior data tree in theuser behavior database 204 and/or query the structural attributes of the non-user behavior data in thenetwork database 205, query and locate pre-analysis data according to the analysis requirement, and perform analysis calculation on the pre-analysis data.
Thedata analysis module 209 comprises: the user behaviorquantity counting unit 2091 is configured to traverse the user behavior data tree, query and locate results corresponding to each behavior, count the quantity of the results corresponding to each behavior, and establish a corresponding relationship between the behavior, the results, and the quantity;
theuser behavior database 204 further includes: the result storage unit is used for storing the corresponding relation among the behaviors, the results and the quantity in a tree structure to obtain a user behavior quantity tree; the behavior is a root node, the result is a branch node below the root node, and the quantity is a branch node below the result node. The result storage unit in theuser behavior database 204 is not shown in fig. 2.
In the system shown in figure 2 of the drawings,
thedata analysis module 209 further comprises: thequery analysis unit 2092 is configured to, when thequery module 208 does not query the query key value and the corresponding data in thedata cache 206, traverse the user behavior data tree and/or query the structural attribute of the non-user behavior data according to the query key value obtained by analyzing by thequery module 208, and locate pre-analysis data; analyzing and calculating the pre-analysis data according to the query request; storing the analysis calculation results in theuser behavior database 204 and/or thenetwork database 205; so that thequery module 208 queries theuser behavior database 204 and/or thenetwork database 205 to obtain a query result, and returns the query result to the external application.
In the system shown in fig. 2, thedata analysis module 209 specifically includes a plurality ofdata analysis sub-modules 2093, which are arranged on a plurality of devices in a distributed manner, and eachdata analysis sub-module 2093 is configured to traverse the user behavior data tree and/or query the structural attributes of non-user behavior data in a distributed calculation manner, query and locate pre-analysis data according to an analysis requirement, and perform analysis calculation on the pre-analysis data;
and/or theuser behavior database 204 stores data in a distributed storage manner.
In one embodiment of the present invention, for example, in an application of statistical analysis of user access behavior of a large website, the data processing system of the present invention: the service interface can be called to obtain real-time data, an application log system, an application system backup library and a web crawler system are monitored, and the data are actively obtained when the application log system, the application system backup library and the web crawler system are idle. The analysis system adopts a Hadoop cluster and adopts greenplus calculation. The user behavior database adopts a non-relational NOSQL database, and the network database adopts a relational database. Such a system may be as shown in fig. 3.
FIG. 3 is a schematic diagram of a data processing system, its peripheral systems, and data flow in the present invention. As shown in fig. 3, data may be acquired from a plurality of data sources such as a service interface, an application log, a backup library, a web crawler, and the like, and after being processed by data routing, Hadoop cluster/greenplus calculation, and the like, the processed data is respectively placed in an NOSQL database and a relational database according to the type thereof, and a key value cache and a data cache are established. After receiving an external application request (such as a statistical result of a certain index queried by a user) the SQL router analyzes the content of the request, queries whether the query exists in the key value cache, and returns an illegal query if the query does not exist. If yes, continuing to query the corresponding data in the database or the data cache (such as standard SQL statement query).
In the system shown in fig. 3:
1. except for the original way of pushing data by each system, the data sources of various channels such as service interface calling, application logs, backup libraries of each application system, web crawlers and the like are added. The resources of application servers such as application logs, application system backup libraries, web crawlers and the like are monitored, and data synchronization is performed only when the application servers are idle, so that contention for system resources among applications is avoided, and the system utilization rate is improved; the real-time data is called and obtained through the service interface, the processing of supporting the real-time data is realized, and the data is more accurate and timely.
2. And converting the peripheral data into system usable data through the data routing module.
3. After the data enters the Hadoop cluster, the speed of calculation by using greenplus is 3-4 times faster than that of hive or mapReduce of the existing scheme.
a. Compared with the existing scheme adopting a relational database, the scheme has the advantages that the NOSQL database does not have the senses of tables and does not need to create fields in advance, so that the tables do not need to be reconstructed when new attributes are added, and the expansibility is enhanced. And the function of customizing the attribute by the user is provided, so that the system is more intelligent.
b. And entering the calculation result of the common indexes into a relational database.
4. All possible query key values are cached in the key value cache. The method can avoid the stress on the data processing system caused by the query of the non-existent data by the external application. And the queried data is cached in the data cache, so that the pressure of the database is reduced, and the speed of querying the data from the database is higher than 10 times, thereby ensuring the support of high concurrent graph display.
5. The SQL router is adopted in the system, the SQL router is applied to the outside to open an interface and is used for receiving the query condition of a user, after the SQL router receives the query condition of the user, the condition of external application self-defined query can be analyzed to obtain a query key value, whether the query key value exists in a key value cache is firstly queried, if the query key value does not exist, an illegal query prompt is directly returned, if the query key value exists, specific query is realized in a data cache (similarly, if the query key value does not exist in the data cache, the query key value is then inquired in an NOSQL database/relational database), and the query result is fed back to the user. Compared with the existing mode that a user puts forward the passive development requirement, the intelligent passive development system is more intelligent and more flexible.
It can be seen from the above that in the data processing system of the present invention, since the tree structure is used to manage the user behavior data, the data cache module is used to store the queried content, and the greenplus calculation mode is used, the operation speed is greatly increased. By combining the reasons, the invention can support the flexible setting of the query conditions, realize the rapid positioning search according to the query conditions, and calculate the required query result at any time when necessary, so that the supported query mode is more flexible. The problem that in the prior art, only the analysis result set by the system can be queried and the system is inflexible is solved, and user experience and application scenes are further improved.
Fig. 4 is a flow chart of a data processing method in the present invention. As shown in fig. 4, the method includes:
401, receiving data pushed by the first type of peripheral system in real time, monitoring the second type of peripheral system, and actively acquiring the data from the second type of peripheral system when the second type of peripheral system is idle.
402, analyzing user behavior data and non-user behavior data from the data; analyzing the user behavior data to obtain user, behavior and result data; establishing a corresponding relation among the user, the behavior and the result; wherein the result is a specific behavior content; and analyzing the non-user behavior data according to the structure attribute of the network database.
403, storing the corresponding relationship among the user, the behavior and the result in a tree structure to obtain a user behavior data tree, and storing the user behavior data tree in a user behavior database; wherein, the user is a root node, the behavior is a branch node under the root node, and the result is a branch node under the behavior node.
And 404, storing the non-user behavior data into a network database according to the structure attribute.
405, caching query key values of data stored in the user behavior database and the network database in the key value cache, and caching the queried query key values and corresponding data in the data cache.
406, when receiving a query request from an external application, analyzing a query key value in the query request, querying whether the query key value exists in a key value cache, if not, returning a prompt of illegal query to the external application, if so, further querying whether the query key value and corresponding data exist in a data cache, if so, returning the corresponding data in the data cache to the external application, otherwise, querying in a user behavior database and a network database, and returning the queried data to the external application.
In the above-mentioned method, the first step of the method,
the receiving the data pushed by the first type of peripheral system in real time comprises: receiving data pushed by a first type of peripheral system in real time in a service interface mode;
the second type of peripheral system comprises: the system comprises an application log system, an application system backup library and a web crawler system.
The method further comprises the following steps: traversing the user behavior data tree and/or inquiring the structure attribute of non-user behavior data, inquiring and positioning pre-analysis data according to analysis requirements, and analyzing and calculating the pre-analysis data;
the step of traversing the user behavior data tree, inquiring and positioning pre-analysis data according to analysis requirements, and the step of analyzing and calculating the pre-analysis data comprises the following steps:
traversing the user behavior data tree, inquiring and positioning results corresponding to each behavior, counting the number of results corresponding to each behavior, establishing corresponding relations among behaviors, results and numbers, and storing the corresponding relations among the behaviors, the results and the numbers in a tree structure to obtain a user behavior number tree; the behavior is a root node, the result is a branch node below the root node, and the quantity is a branch node below the result node.
In summary, the data processing system of the present invention comprises: the data acquisition module is used for sending the acquired data to the analysis system; the analysis system analyzes the user behavior data and the non-user behavior data from the received data; the user behavior database stores the user behavior data in a tree structure to obtain a user behavior data tree; the network database stores the non-user behavior data according to the structure attribute; the key value cache module stores the query key values of the data stored in the database; the data cache module stores the queried query key values and corresponding data; the query module analyzes and searches the query key value in the key value cache module, returns an illegal query prompt if the query key value does not exist, and queries in the data cache module and returns the query prompt to an external application or queries in the database if the query key value exists. According to the technical scheme, the computing capacity of the data processing system is improved, and the system is easy to expand, high in automation degree and more beneficial to query.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A data processing system, characterized in that the system comprises: the system comprises a data acquisition module, an analysis system, a user behavior database, a network database, a data cache module, a key value cache module and a query module;
the data acquisition module is used for receiving data pushed by the first type of peripheral system in real time, monitoring the second type of peripheral system, and actively acquiring the data from the second type of peripheral system when the second type of peripheral system is idle; sending the data to an analysis system;
the analysis system is used for analyzing the user behavior data and the non-user behavior data from the received data; analyzing the user behavior data to obtain user, behavior and result data; establishing a corresponding relation among the user, the behavior and the result; wherein the result is a specific behavior content; analyzing the non-user behavior data according to the structure attribute of the network database;
the user behavior database is used for storing the corresponding relation among the user, the behavior and the result in a tree structure to obtain a user behavior data tree; wherein, the user is a root node, the behavior is a branch node under the root node, and the result is a branch node under the behavior node;
the network database is used for storing the non-user behavior data according to the structure attribute;
the key value cache module is used for storing query key values of data stored in the user behavior database and the network database;
the data cache module is used for storing the queried query key values and the corresponding data;
the query module is used for receiving a query request of an external application, analyzing a query key value in the query request, querying whether the query key value exists in the key value cache module, if not, returning a prompt of illegal query to the external application, if so, further querying whether the query key value and corresponding data exist in the data cache module, if so, returning the corresponding data in the data cache to the external application, otherwise, querying in a user behavior database and a network database, and returning the queried data to the external application.
2. The system of claim 1, further comprising: a data routing module;
the data acquisition module is used for receiving the data pushed by the first type of peripheral system in real time through the data routing module and actively acquiring user behavior data from the second type of peripheral system through the data routing module;
and the data routing module is used for converting the data from the first type peripheral system and the second type peripheral system into data which accords with the data processing system and then transmitting the data to the data acquisition module.
3. The system according to claim 1 or 2,
the data acquisition module receives data pushed by the first type of peripheral system in real time in a service interface mode;
and the data acquisition module is used for monitoring the application log system, the application system backup library and the web crawler system, and actively acquiring data from the systems when the systems are idle.
4. The system according to claim 1 or 2, further comprising a data analysis module for traversing the user behavior data tree and/or querying structural attributes of non-user behavior data, querying and locating pre-analysis data according to analysis requirements, and performing analysis calculation on the pre-analysis data.
5. The system of claim 4,
the data analysis module includes: the user behavior quantity counting unit is used for traversing the user behavior data tree, inquiring and positioning results corresponding to all behaviors, counting the quantity of the results corresponding to all behaviors and establishing the corresponding relation among the behaviors, the results and the quantity;
the user behavior database further comprises:
the result storage unit is used for storing the corresponding relation among the behaviors, the results and the quantity in a tree structure to obtain a user behavior quantity tree; the behavior is a root node, the result is a branch node below the root node, and the quantity is a branch node below the result node.
6. The system of claim 4,
the data analysis module further comprises: the query analysis unit is used for traversing the user behavior data tree and/or querying the structural attribute of non-user behavior data according to the query key value obtained by the query module in an analysis manner when the query module does not query the query key value and the corresponding data in the data cache, and positioning pre-analysis data; analyzing and calculating the pre-analysis data according to the query request; storing the analysis calculation result in a user behavior database and/or a network database; so that the query module queries in the user behavior database and/or the network database to obtain a query result and returns the query result to the external application.
7. The system of claim 4,
the data analysis module specifically comprises a plurality of data analysis sub-modules which are arranged on a plurality of devices in a distributed mode, and each data analysis sub-module is used for traversing the user behavior data tree and/or inquiring the structural attribute of non-user behavior data in a distributed computing mode, inquiring and positioning pre-analysis data according to analysis requirements, and analyzing and calculating the pre-analysis data;
and/or the user behavior database stores data in a distributed storage mode.
8. A method of data processing, the method comprising:
receiving data pushed by a first type of peripheral system in real time, monitoring a second type of peripheral system, and actively acquiring the data from the second type of peripheral system when the second type of peripheral system is idle;
analyzing user behavior data and non-user behavior data from the data; analyzing the user behavior data to obtain user, behavior and result data; establishing a corresponding relation among the user, the behavior and the result; wherein the result is a specific behavior content; analyzing the non-user behavior data according to the structure attribute of the network database;
storing the corresponding relation among the user, the behavior and the result in a tree structure to obtain a user behavior data tree, and storing the user behavior data tree in a user behavior database; wherein, the user is a root node, the behavior is a branch node under the root node, and the result is a branch node under the behavior node;
storing the non-user behavior data into a network database according to the structure attribute;
caching query key values of data stored in a user behavior database and a network database in a key value cache, and caching the queried query key values and corresponding data in a data cache;
when a query request of an external application is received, analyzing a query key value in the query request, querying whether the query key value exists in a key value cache, if not, returning a prompt of illegal query to the external application, if so, further querying whether the query key value and corresponding data exist in a data cache, if so, returning the corresponding data in the data cache to the external application, otherwise, querying in a user behavior database and a network database, and returning the queried data to the external application.
9. The system of claim 8,
the receiving the data pushed by the first type of peripheral system in real time comprises: receiving data pushed by a first type of peripheral system in real time in a service interface mode;
the second type of peripheral system comprises: the system comprises an application log system, an application system backup library and a web crawler system.
10. The method according to claim 8 or 9, characterized in that the method further comprises: traversing the user behavior data tree and/or inquiring the structure attribute of non-user behavior data, inquiring and positioning pre-analysis data according to analysis requirements, and analyzing and calculating the pre-analysis data;
the step of traversing the user behavior data tree, inquiring and positioning pre-analysis data according to analysis requirements, and the step of analyzing and calculating the pre-analysis data comprises the following steps:
traversing the user behavior data tree, inquiring and positioning results corresponding to each behavior, counting the number of results corresponding to each behavior, establishing corresponding relations among behaviors, results and numbers, and storing the corresponding relations among the behaviors, the results and the numbers in a tree structure to obtain a user behavior number tree; the behavior is a root node, the result is a branch node below the root node, and the quantity is a branch node below the result node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110300725.9ACN103034650B (en) | 2011-09-29 | 2011-09-29 | A kind of data handling system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110300725.9ACN103034650B (en) | 2011-09-29 | 2011-09-29 | A kind of data handling system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103034650Atrue CN103034650A (en) | 2013-04-10 |
| CN103034650B CN103034650B (en) | 2015-10-28 |
Family
ID=48021552
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110300725.9AActiveCN103034650B (en) | 2011-09-29 | 2011-09-29 | A kind of data handling system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103034650B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514273A (en)* | 2013-09-17 | 2014-01-15 | 宁波东冠科技有限公司 | Data collection and monitoring control system and data processing method of system |
| CN104346345A (en)* | 2013-07-24 | 2015-02-11 | 中兴通讯股份有限公司 | Data storage method and device |
| CN105760548A (en)* | 2016-03-21 | 2016-07-13 | 武汉烽火众智数字技术有限责任公司 | Vehicle first appearance analysis method and system based on big data cross-domain comparison |
| WO2017032240A1 (en)* | 2015-08-24 | 2017-03-02 | 阿里巴巴集团控股有限公司 | Data storage method and apparatus for mobile terminal |
| CN107798037A (en)* | 2017-04-26 | 2018-03-13 | 平安科技（深圳）有限公司 | The acquisition methods and server of user characteristic data |
| CN112182340A (en)* | 2019-07-01 | 2021-01-05 | 中国移动通信集团浙江有限公司 | Internet of Things information query method, subscription method, device and electronic device |
| CN114020850A (en)* | 2022-01-05 | 2022-02-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
| CN114676194A (en)* | 2022-03-15 | 2022-06-28 | 江门职业技术学院 | Distributed database lateral expansion method and device and computer equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170426A (en)* | 2006-10-25 | 2008-04-30 | 马永利 | Personalized content distribution scheme based on user behavior (habit) analysis |
| CN101409690A (en)* | 2008-11-26 | 2009-04-15 | 北京学之途网络科技有限公司 | Method and system for obtaining internet user behaviors |
| US20090313086A1 (en)* | 2008-06-16 | 2009-12-17 | Sungkyunkwan University Foundation For Corporate Collaboration | User recommendation method and recorded medium storing program for implementing the method |
| CN102111453A (en)* | 2011-03-04 | 2011-06-29 | 创博亚太科技（山东）有限公司 | Method and system for extracting Internet user network behaviors |
- 2011
- 2011-09-29CNCN201110300725.9Apatent/CN103034650B/enactiveActive
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170426A (en)* | 2006-10-25 | 2008-04-30 | 马永利 | Personalized content distribution scheme based on user behavior (habit) analysis |
| US20090313086A1 (en)* | 2008-06-16 | 2009-12-17 | Sungkyunkwan University Foundation For Corporate Collaboration | User recommendation method and recorded medium storing program for implementing the method |
| CN101409690A (en)* | 2008-11-26 | 2009-04-15 | 北京学之途网络科技有限公司 | Method and system for obtaining internet user behaviors |
| CN102111453A (en)* | 2011-03-04 | 2011-06-29 | 创博亚太科技（山东）有限公司 | Method and system for extracting Internet user network behaviors |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346345B (en)* | 2013-07-24 | 2019-03-26 | 上海中兴软件有限责任公司 | The storage method and device of data |
| CN104346345A (en)* | 2013-07-24 | 2015-02-11 | 中兴通讯股份有限公司 | Data storage method and device |
| CN103514273B (en)* | 2013-09-17 | 2016-08-17 | 宁波东冠科技有限公司 | DAM control system and the data processing method of this system |
| CN103514273A (en)* | 2013-09-17 | 2014-01-15 | 宁波东冠科技有限公司 | Data collection and monitoring control system and data processing method of system |
| WO2017032240A1 (en)* | 2015-08-24 | 2017-03-02 | 阿里巴巴集团控股有限公司 | Data storage method and apparatus for mobile terminal |
| US10776323B2 (en) | 2015-08-24 | 2020-09-15 | Alibaba Group Holding Limited | Data storage for mobile terminals |
| CN105760548A (en)* | 2016-03-21 | 2016-07-13 | 武汉烽火众智数字技术有限责任公司 | Vehicle first appearance analysis method and system based on big data cross-domain comparison |
| WO2018196650A1 (en)* | 2017-04-26 | 2018-11-01 | 平安科技（深圳）有限公司 | User feature data acquisition method and device, server, and medium |
| CN107798037A (en)* | 2017-04-26 | 2018-03-13 | 平安科技（深圳）有限公司 | The acquisition methods and server of user characteristic data |
| CN112182340A (en)* | 2019-07-01 | 2021-01-05 | 中国移动通信集团浙江有限公司 | Internet of Things information query method, subscription method, device and electronic device |
| CN112182340B (en)* | 2019-07-01 | 2024-06-07 | 中国移动通信集团浙江有限公司 | Internet of things information query method, subscription method, device and electronic equipment |
| CN114020850A (en)* | 2022-01-05 | 2022-02-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
| CN114020850B (en)* | 2022-01-05 | 2022-04-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
| CN114676194A (en)* | 2022-03-15 | 2022-06-28 | 江门职业技术学院 | Distributed database lateral expansion method and device and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103034650B (en) | 2015-10-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103034650B (en) | A kind of data handling system and method | |
| CN111382226B (en) | Database query and retrieval method and device and electronic equipment | |
| JP6617117B2 (en) | Scalable analysis platform for semi-structured data | |
| CN109213756B (en) | Data storage method, data retrieval method, data storage device, data retrieval device, server and storage medium | |
| CN107783985B (en) | Distributed database query method, device and management system | |
| CN109063196B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US9229961B2 (en) | Database management delete efficiency | |
| CN104424258B (en) | Multidimensional data query method, query server, column storage server and system | |
| CN114328601B (en) | Data downsampling and data query method, system and storage medium | |
| CN112231351B (en) | A real-time query method and device for PB-level massive data | |
| US11494395B2 (en) | Creating dashboards for viewing data in a data storage system based on natural language requests | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| JP6996812B2 (en) | How to process data blocks in a distributed database, programs, and devices | |
| CN103440288A (en) | Big data storage method and device | |
| CN104252536A (en) | Hbase-based internet log data inquiring method and device | |
| US10901811B2 (en) | Creating alerts associated with a data storage system based on natural language requests | |
| CN111026709A (en) | Data processing method and device based on cluster access | |
| CN104462161A (en) | Structural data query method based on distributed database | |
| CN111522918A (en) | Data aggregation method and device, electronic equipment and computer readable storage medium | |
| CN115391286A (en) | Link tracking data management method, device, equipment and storage medium | |
| CN118503229B (en) | Hudi data ingestion method and system for multi-source heterogeneous data | |
| CN107291875B (en) | Method and system for metadata organization and management based on metadata graph | |
| CN119336759A (en) | Data label processing method, device, equipment and medium based on dimension conversion | |
| CN108564250A (en) | Forwarding record real time inspection system, method, computer equipment and storage medium | |
| CN118051536A (en) | Multi-source cross-source data query method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | Address after:Room 810, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after:BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. Address before:100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building A block 5 layer Patentee before:BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. | |
| CP02 | Change in the address of a patent holder |