CN103577440A - Data processing method and device in non-relational database - Google Patents
Data processing method and device in non-relational databaseDownload PDFInfo
- Publication number
- CN103577440A CN103577440ACN201210266418.8ACN201210266418ACN103577440ACN 103577440 ACN103577440 ACN 103577440ACN 201210266418 ACN201210266418 ACN 201210266418ACN 103577440 ACN103577440 ACN 103577440A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- memory address
- target memory
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical field
The application relates to a kind of technical field of non-relational database, particularly relate to the data processing method in a kind of non-relational database, data processing equipment in a kind of non-relational database, data enquire method in a kind of non-relational database, and, the data query device in a kind of non-relational database.
Background technology
In routine duties, will face a large amount of data messages sometimes, and these a large amount of information are random, between there is no certain association, be non-relational data.
For example certain financial product has a large amount of user data to need to analyze, and in analyzing these user data processes, need to carry out from a plurality of dimensions, and each dimension has again a lot of key-value pairs, stores these key-value pairs and need to use the storage of non-relational database.But various commercial scheme costs are higher in the market, and the non-relational database scheme of increasing income can not meet the requirement of performance and capacity aspect.The non-relational database technology of increasing income of current popular mainly there will be following two class problems:
1, the data volume of supporting is limited, and has the inconsistent fatal problem of read-write; Or system is extremely unstable, and operation maintenance machine cost is too high.
2, purely with key-value pair storage mode, realize memory requirement that cannot satisfying magnanimity user profile.
Therefore, those skilled in the art's problem in the urgent need to address is: data processing in a kind of non-relational database and method and the device of inquiry are provided, in order to improve the systematicness of non-relational data storage, make to search and extract data more fast accurately, improve the stability of operation; The data volume that expansion is supported, the cost of reduction operation maintenance.
Summary of the invention
The application's technical matters to be solved is to provide data processing in a kind of non-relational database and method and the device of inquiry, in order to improve the systematicness of non-relational data storage, improves the stability of operation; The data volume that expansion is supported, the cost of reduction operation maintenance.
In order to address the above problem, the application discloses the data processing method in a kind of non-relational database, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described method comprises:
Generate the user ID of data to be stored, described data to be stored are the data that store into first in described distributed system;
Described data to be stored are write in the data form in a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, generate target memory address;
Record the mapping relations of described user ID and target memory address.
Preferably, described data to be stored comprise user's characteristic information; The step of the user ID of described generation data to be stored comprises:
Extract user's characteristic information in described data to be stored;
Record the mapping of described user's characteristic information and preset storage sequence number, and described corresponding stored sequence number is designated as to user ID.
Preferably, in the described data form that data to be stored is write to a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, the step that generates target memory address comprises:
Select not reach in a data minute storehouse data form of the preset number of users upper limit, generate target data form;
Described data to be stored are write to described target data form;
Divide the identifier combination in storehouse to generate target memory address the sign of described target data form and corresponding data.
Preferably, the described step that records the mapping relations of described user ID and target memory address comprises:
Set up and select described user ID to shine upon the key-value pair of described target memory address;
Record described key-value pair in map listing.
Preferably, described method also comprises:
Obtain user ID in data to be added; Described data to be added are to store in the corresponding target memory address of the built user ID of described distributed system, are the supplementary datas of built user ID;
According to described user ID inquiry, obtain target memory address;
Described data to be added are write to described target memory address.
Preferably, described method also comprises:
According to user ID, target memory address is obtained in inquiry, the corresponding data minute storehouse of pointing to and target data form, extracts the user data in described target memory address from target memory address.
Preferably, described data to be added comprise user's characteristic information; The described step of obtaining user ID in data to be added comprises:
Extract the user's characteristic information in described data to be added;
According to described user's characteristic information mapping, obtain corresponding stored sequence number, described corresponding stored sequence number is user ID.
Preferably, if a legacy data minute library storage load reaches the upper limit, increase the unit of described distributed system, and in described unit, set up new data minute storehouse.
Preferably, the data to be stored of the same user ID of described correspondence or data to be added are stored among a data form.
The embodiment of the present application also discloses the data processing equipment in a kind of non-relational database, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described device comprises:
User ID generation module, for generating the user ID of data to be stored, described data to be stored are the data that store into first in described distributed system;
Target storing address generating module, for described data to be stored being write to the data form in a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, generate target memory address;
Mapping relations logging modle, for recording the mapping relations of described user ID and target memory address.
Preferably, described data to be stored comprise user's characteristic information; Described user ID generation module comprises:
User's characteristic information extracts submodule, for extracting described data user's characteristic information to be stored;
User ID generates submodule, for recording the mapping of described user's characteristic information and preset storage sequence number, and described corresponding stored sequence number is designated as to user ID.
Preferably, described target storing address generating module comprises:
Target data form generates submodule, for selecting a data minute storehouse not reach the data form of the preset number of users upper limit, generate target data form;
Data to be stored write submodule, described data to be stored are write to described target data form;
Identifier combination submodule, divide the identifier combination in storehouse to generate target memory address the sign of described target data form and corresponding data.
Preferably, described mapping relations logging modle comprises:
Key-value pair is set up submodule, for setting up, is selected described user ID to shine upon the key-value pair of described target memory address;
Key-value pair record sub module, record described key-value pair in map listing.
Preferably, described device also comprises:
User ID acquisition module, for obtaining data user ID to be added; Described data to be added are to store in the corresponding target memory address of the built user ID of described distributed system, are the supplementary datas of built user ID;
Target memory address acquisition module, for according to described user ID inquiry obtain target memory address;
Target memory address writing module, for described data to be added are write to described target memory address.
Preferably, described device also comprises:
According to user ID, target memory address is obtained in inquiry, the corresponding data minute storehouse of pointing to and target data form, extracts the user data in described target memory address from target memory address.
Preferably, described data to be added comprise user's characteristic information; Described user ID acquisition module comprises:
User's characteristic information extracts submodule, for extracting the user's characteristic information of described data to be added;
Storage sequence number obtains submodule, according to described user's characteristic information mapping, obtains corresponding stored sequence number, and described corresponding stored sequence number is user ID.
Preferably, if a legacy data minute library storage load reaches the upper limit, increase the unit of described distributed system, and in described unit, set up new data minute storehouse.
Preferably, the data to be stored of the same user ID of described correspondence or data to be added are stored among a data form.
The embodiment of the present application also discloses the data enquire method in a kind of non-relational database, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described method comprises:
The inquiry request that receives user data, described inquiry request comprises user ID;
According to described user ID inquiry, obtain target memory address, described target memory address is generated by the sign of target data form and the identifier combination in corresponding data minute storehouse;
The corresponding data minute storehouse of pointing to from described target memory address and target data form, extract user data;
Return to described user data.
The embodiment of the present application also discloses the data query device in a kind of non-relational database, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described device comprises:
User ID acquisition module, for receiving the inquiry request of user data, described inquiry request comprises user ID;
Target memory address enquiry module, according to described user ID, target memory address is obtained in inquiry, and described target memory address is generated by the sign of target data form and the identifier combination in corresponding data minute storehouse;
Store data acquisition module, for corresponding data minute storehouse and the target data form pointing to from described target memory address, extract user data;
Storage data return to module, for returning to described user data.
Compared with prior art, the application has the following advantages:
The application stores by a kind of non-relational database the data that application scenarios has a large amount of user data to analyze, adopt data minute storehouse and tables of data to carry out orderly store and management user data simultaneously, and set up the map listing of user ID and target memory address, effectively improve the systematicness of non-relational data storage, make to search and extract data more fast accurately, improve the stability of operation; The data volume that expansion is supported, the cost of reduction operation maintenance.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of the data processing method embodiment 1 in a kind of non-relational database of the application;
Fig. 2 is the process flow diagram of the data processing method embodiment 2 in a kind of non-relational database of the application;
Fig. 3 is the structured flowchart of the data processing equipment embodiment 1 in a kind of non-relational database of the application;
Fig. 4 is the structured flowchart of the data processing equipment embodiment 2 in a kind of non-relational database of the application;
Fig. 5 is the schematic diagram of a kind of target memory address numbering of the application;
Fig. 6 is the process flow diagram of the data enquire method embodiment in a kind of non-relational database of the application;
Fig. 7 is the structured flowchart of the data query device embodiment in a kind of non-relational database of the application.
Embodiment
For the application's above-mentioned purpose, feature and advantage can be become apparent more, below in conjunction with the drawings and specific embodiments, the application is described in further detail.
Relevant database: be the database being based upon on relational model basis, carry out the data in process database by means of the mathematical concepts such as algebra of sets and method.Various entities in real world and the various contacts between entity all represent with relational model.Normal data query language SQL is exactly a kind of language based on relational database, and this language is carried out the retrieval of data in relational database and operation.
Non-relational database: store with key-value pair, its structure is fixing, and each tuple can have different field, and each tuple can increase some oneself key-value pair as required, so just fixing structure can be confined to, the expense in some times and space can be reduced.
The application's application scenarios has a large amount of user data to need to analyze, and the user data of the application's application scenarios magnanimity is opening relationships model one by one, so relevant database can not adapt to the needs of the application's Users'Data Analysis.In the process of the application's Users'Data Analysis, need to carry out from a plurality of dimensions, each dimension has again a lot of k/v key-value pairs, stores these key assignments and adopts non-relational database the most suitable.But various commercial scheme costs are higher in the market, and the non-relational database scheme of increasing income can not meet the requirement of performance and capacity aspect.
The non-relational database technology of increasing income of current popular, has cassandra and HanderSocket.
Cassandra is used local file system storage data, but the data volume of supporting is limited, and has the inconsistent fatal problem of read-write.Although support cluster expansion, system is extremely unstable, and O&M machine cost is too high.
HanderSocket is that pure k/v key-value pair is realized, the replacement scheme that application scenarios is MemoryCache.Memory requirement that cannot satisfying magnanimity user profile.
So the scheme of existing relevant database and non-relational database scheme all cannot be overlapped in the application scenarios of using the application.
The application stores by a kind of non-relational database the data that application scenarios has a large amount of user data to analyze, adopt data minute storehouse and tables of data to carry out orderly store and management user data simultaneously, and set up the map listing of user ID and target memory address, effectively improve the systematicness of non-relational data storage, make to search and extract data more fast accurately, improve the stability of operation; The data volume that expansion is supported, the cost of reduction operation maintenance.
Described in the application, non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, and a described data minute storehouse has data minute storehouse sign, and described data form has data form sign.
With reference to Fig. 1, show the process flow diagram of the data processing method embodiment 1 in a kind of non-relational database of the application, specifically can comprise:
In a preferred embodiment of the present application, described data to be stored are to store first the data in described distributed system into, need newly-built user ID.
In a preferred embodiment of the present application, described data to be stored can comprise user's characteristic information; Describedstep 101 specifically can comprise following sub-step:
Sub-step S11, extract user's characteristic information in described data to be stored;
In specific implementation, user's characteristic information can be user's network ID, the register account number of user in system, or the information of user's relevant proof of identification etc.; So long as can distinguish user and other users' information, can become user's characteristic information.
Sub-step S12, record the mapping of described user's characteristic information and preset storage sequence number, and described corresponding stored sequence number is designated as to user ID.
In specific implementation, preset storage sequence number can directly adopt the serial number of storage, and data to be stored during as the data of the 10021st storage, can be set as its storage sequence number: 001021.
In specific implementation, can also set up map listing, user's characteristic information and preset storage sequence number are recorded in map listing, form mapping relations.
In a preferred embodiment of the present application, describedstep 102 specifically can comprise following sub-step:
Sub-step S21, select not reach in a data minute storehouse data form of the preset number of users upper limit, generate target data form;
In specific implementation, the table quantity in single storehouse too much can be difficult to management, so may be limited in 1000.Can dynamically increase the object that data minute storehouse can reach dilatation according to the demand of oneself like this.
Sub-step S22, described data to be stored are write to described target data form;
Sub-step S23, divide the identifier combination in storehouse to generate target memory address the sign of described target data form and corresponding data.
In specific implementation, described target memory address is equivalent to an exclusive sign, can divide storehouse sign and data form sign to generate according to available data, certain data form sign and its corresponding data minute storehouse sign can be connected to the target memory address that generates this data form.As shown in Figure 5.In Fig. 5, first four can be number information in the sign of data minute storehouse, and latter three can be number information in data form sign.
In specific implementation, can set up the map listing of user ID and target memory address.
In a preferred embodiment of the present application, describedstep 103 specifically can comprise following sub-step:
Sub-step S31, foundation select described user ID to shine upon the key-value pair of described target memory address;
Sub-step S32, record described key-value pair in map listing.
In specific implementation, when extracting certain user's data, can find according to respective user sign the target memory address of all data storages of respective user.Then call the data that preset interface extracts the needs in described target memory address.
With reference to Fig. 2, show the process flow diagram of the data processing method embodiment 2 in a kind of non-relational database of the application, specifically can comprise:
In a kind of preferred embodiment of the application, described data to be added need to store in the corresponding target memory address of the built user ID of described distributed system, are the supplementary datas of built user ID.
In concrete practice, data to be added do not need newly-built user ID.But find a corresponding target memory address of the user ID having existed, data to be added are write wherein.
In a preferred embodiment of the present application, described data to be added comprise user's characteristic information; Describedstep 201 specifically can comprise following sub-step:
Sub-step S41, extract the user's characteristic information in described data to be added;
Sub-step S42, according to described user's characteristic information, corresponding stored sequence number is obtained in mapping, and described corresponding stored sequence number is user ID.
In a preferred embodiment of the present application, if a legacy data minute library storage load reaches the upper limit, increase the unit of described distributed system, and in described unit, set up new data minute storehouse.
In specific implementation, the table quantity in single storehouse too much can be difficult to management, so may be limited in 1000.Can dynamically increase storehouse according to the demand of oneself like this can dilatation.
In a preferred embodiment of the present application, described same storage of subscriber data is among a data form.
In specific implementation, due to a user's whole data model, all data messages of a user, all in a table, have been avoided the query script in multilist or many storehouses, and therefore efficiency and the stability of inquiry also can obtain larger lifting.
It should be noted that, for embodiment of the method, for simple description, therefore it is all expressed as to a series of combination of actions, but those skilled in the art should know, the application is not subject to the restriction of described sequence of movement, because according to the application, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and related action might not be that the application is necessary.
In order to facilitate those skilled in the art to understand better the application, below in conjunction with table 1, by a complete example, illustrate further the application:
Table 1: shown in be a user's model information:
| ID | Nick | Age | Sex |
| 10000 | Lionbule | 30 | man |
Table 1
A1, extraction user ID: 10000; Wherein other data informations of user also comprise: Data{nick:Lionbule; Age:30; Sex:man}.
A2, storage user model information time, first call memory interface: api save (long id, Map < String, Object > data), query aim memory address.If user ID: the mapping address of 10000 correspondences does not exist, applies for new target memory address A, Data{nick:Lionbule; Age:30; Sex:man} is converted to the line item of database table at new target memory address A by key, then store new target memory address A in mapping table.If user ID: the mapping address of 10000 correspondences exists, extracts corresponding target memory address B.Data{nick:Lionbule; Age:30; Sex:man} is converted to the line item of database table at target memory address B by key.
B1, now a user's newly-added information need to be stored in this user's target memory address, know this user ID: 10000, first call query interface: api query (long id), according to user ID: mapping target memory address corresponding to 10000 inquiry.
B2, pass through user ID: 10000 inquire corresponding mapping address exists, and extracts corresponding target memory address C.
B3, by the newly-increased information D ata{nick:Lionbule of user; Age:30; Sex:man} stores in target memory address C.
C1, now need to obtain user ID: 10001 information.First call query interface: api query (long id), according to user ID: mapping target memory address corresponding to 10001 inquiry.
C2, inquiry, without result, are found user ID: the mapping address of 10001 correspondences does not exist, throw exception information exit query script.Result is returned to client.
The structured flowchart that shows the data processing equipment embodiment 1 in a kind of non-relational database of the application with reference to Fig. 3, specifically can comprise:
User ID generation module 301, for generating the user ID of data to be stored;
In a preferred embodiment of the present application, described data to be stored are to store first the data in described distributed system into, need newly-built user ID.
In a preferred embodiment of the present application, described data to be stored comprise user's characteristic information; Described user ID generation module 301 specifically can comprise following submodule:
User's characteristic information extracts submodule, for extracting described data user's characteristic information to be stored;
User ID generates submodule, for recording the mapping of described user's characteristic information and preset storage sequence number, and described corresponding stored sequence number is denoted as to user ID.
Target storing address generating module 302, for described data to be stored being write to the data form in a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, generate target memory address;
In a preferred embodiment of the present application, described target storing address generating module 302 specifically can comprise following submodule:
Target data form generates submodule, for selecting a data minute storehouse not reach the data form of the preset number of users upper limit, generate target data form;
Data to be stored write submodule, for described data to be stored are write to described target data form;
Identifier combination submodule, for the sign of described target data form and corresponding data being divided the identifier combination in storehouse generate target memory address.
Mapping relations logging modle 303, for recording the mapping relations of described user ID and target memory address;
In a preferred embodiment of the present application, described mapping relations logging modle 303 specifically can comprise following submodule:
Key-value pair is set up submodule, for setting up, is selected described user ID to shine upon the key-value pair of described target memory address;
Key-value pair record sub module, for recording described key-value pair at map listing.
User data extraction module 304, for obtaining target memory address according to user ID inquiry, call preset interface and extract the user data in described target memory address.
With reference to Fig. 4, show the structured flowchart of the data processing equipment embodiment 2 in a kind of non-relational database of the application, specifically can comprise:
UserID acquisition module 401, for obtaining data user ID to be added;
In a preferred embodiment of the present application, described data to be added need to store in the corresponding target memory address of the built user ID of described distributed system, are the supplementary datas of built user ID.
In a preferred embodiment of the present application, described data to be added comprise user's characteristic information; Described userID acquisition module 401 specifically can comprise following sub-step:
User's characteristic information extracts submodule, for extracting the user's characteristic information of described data to be added;
Storage sequence number obtain submodule, for according to described user's characteristic information mapping obtain corresponding stored sequence number, described corresponding stored sequence number is user ID.
Target memoryaddress acquisition module 402, for according to described user ID inquiry obtain target memory address;
Userdata extraction module 404, for obtain target memory address according to user ID inquiry, the corresponding data minute storehouse of pointing to from target memory address and target data form, extract the user data in described target memory address.
In a preferred embodiment of the present application, if a legacy data minute library storage load reaches the upper limit, increase the unit of described distributed system, and in described unit, set up new data minute storehouse.
In a preferred embodiment of the present application, described same storage of subscriber data is among a data form.
With reference to Fig. 6, show the flow chart of steps of the data enquire method embodiment in a kind of non-relational database of the application, specifically can comprise the steps:
The inquiry request ofstep 601, reception user data, described inquiry request comprises user ID;
In order to facilitate those skilled in the art to understand better the application, below in conjunction with table 1, by a complete example, illustrate further the embodiment of the present application:
D1, now need to obtain user ID: 10000 information.First call query interface: api query (long id), according to user ID: mapping target memory address corresponding to 10000 inquiry.
If D2 user ID: the target memory address of 10000 correspondences exists, user's data information Data{nick:Lionbule; Age:30; Sex:man} returns to client.
With reference to Fig. 7, show the structured flowchart of the data query device embodiment in a kind of non-relational database of the application, specifically can comprise:
UserID acquisition module 701, for receiving the inquiry request of user data, described inquiry request comprises user ID;
Target memoryaddress enquiry module 702, for according to described user ID inquiry obtain target memory address, described target memory address is generated by the sign of target data form and the identifier combination in corresponding data minute storehouse;
Storedata acquisition module 703, for corresponding data minute storehouse and the target data form pointing to from described target memory address, extract user data;
Storage data return tomodule 704, for returning to described user data.
Because described device embodiment is substantially corresponding to preceding method embodiment, therefore not detailed part in the description of the present embodiment can, referring to the related description in previous embodiment, just not repeat at this.
Those skilled in the art should understand, the application's embodiment can be provided as method, system or computer program.Therefore, the application can adopt complete hardware implementation example, implement software example or in conjunction with the form of the embodiment of software and hardware aspect completely.And the application can adopt the form that wherein includes the upper computer program of implementing of computer-usable storage medium (including but not limited to magnetic disk memory, CD-ROM, optical memory etc.) of computer usable program code one or more.
The application is with reference to describing according to process flow diagram and/or the block scheme of the method for the embodiment of the present application, equipment (system) and computer program.Should understand can be in computer program instructions realization flow figure and/or block scheme each flow process and/or the flow process in square frame and process flow diagram and/or block scheme and/or the combination of square frame.Can provide these computer program instructions to the processor of multi-purpose computer, special purpose computer, Embedded Processor or other programmable data processing device to produce a machine, the instruction of carrying out by the processor of computing machine or other programmable data processing device is produced for realizing the device in the function of flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame appointments.
These computer program instructions also can be stored in energy vectoring computer or the computer-readable memory of other programmable data processing device with ad hoc fashion work, the instruction that makes to be stored in this computer-readable memory produces the manufacture that comprises command device, and this command device is realized the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame.
These computer program instructions also can be loaded in computing machine or other programmable data processing device, make to carry out sequence of operations step to produce computer implemented processing on computing machine or other programmable devices, thereby the instruction of carrying out is provided for realizing the step of the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame on computing machine or other programmable devices.
Although described the application's preferred embodiment, once those skilled in the art obtain the basic creative concept of cicada, can make other change and modification to these embodiment.So claims are intended to all changes and the modification that are interpreted as comprising preferred embodiment and fall into the application's scope.
Data processing method in a kind of non-relational database above the application being provided and the data processing equipment in a kind of non-relational database are described in detail, applied specific case herein the application's principle and embodiment are set forth, the explanation of above embodiment is just for helping to understand the application's method and core concept thereof; Meanwhile, for one of ordinary skill in the art, the thought according to the application, all will change in specific embodiments and applications, and in sum, this description should not be construed as the restriction to the application.
Claims (20)
1. the data processing method in a non-relational database, it is characterized in that, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described method comprises:
Generate the user ID of data to be stored, described data to be stored are the data that store into first in described distributed system;
Described data to be stored are write in the data form in a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, generate target memory address;
Record the mapping relations of described user ID and target memory address.
2. the method for claim 1, is characterized in that, described data to be stored comprise user's characteristic information; The step of the user ID of described generation data to be stored comprises:
Extract user's characteristic information in described data to be stored;
Record the mapping of described user's characteristic information and preset storage sequence number, and described corresponding stored sequence number is designated as to user ID.
3. method as claimed in claim 2, it is characterized in that, in the described data form that data to be stored is write to a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, the step that generates target memory address comprises:
Select not reach in a data minute storehouse data form of the preset number of users upper limit, generate target data form;
Described data to be stored are write to described target data form;
Divide the identifier combination in storehouse to generate target memory address the sign of described target data form and corresponding data.
4. method as claimed in claim 3, is characterized in that, the described step that records the mapping relations of described user ID and target memory address comprises:
Set up and select described user ID to shine upon the key-value pair of described target memory address;
Record described key-value pair in map listing.
5. method as claimed in claim 4, is characterized in that, also comprises:
Obtain user ID in data to be added; Described data to be added are to store in the corresponding target memory address of the built user ID of described distributed system, are the supplementary datas of built user ID;
According to described user ID inquiry, obtain target memory address;
Described data to be added are write to described target memory address.
6. method as claimed in claim 5, is characterized in that, also comprises:
According to user ID, target memory address is obtained in inquiry, the corresponding data minute storehouse of pointing to and target data form, extracts the user data in described target memory address from target memory address.
7. method as claimed in claim 6, is characterized in that, described data to be added comprise user's characteristic information; The described step of obtaining user ID in data to be added comprises:
Extract the user's characteristic information in described data to be added;
According to described user's characteristic information mapping, obtain corresponding stored sequence number, described corresponding stored sequence number is user ID.
8. method as claimed in claim 7, is characterized in that, if a legacy data minute library storage load reaches the upper limit, increases the unit of described distributed system, and in described unit, sets up new data minute storehouse.
9. the method for claim 1, is characterized in that, the data to be stored of the same user ID of described correspondence or data to be added are stored among a data form.
10. the data processing equipment in a non-relational database, it is characterized in that, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described device comprises:
User ID generation module, for generating the user ID of data to be stored, described data to be stored are the data that store into first in described distributed system;
Target storing address generating module, for described data to be stored being write to the data form in a certain data minute storehouse, and divide storehouse sign and data form sign corresponding to described data form according to data corresponding to described data minute storehouse, generate target memory address;
Mapping relations logging modle, for recording the mapping relations of described user ID and target memory address.
11. devices as claimed in claim 10, is characterized in that, described data to be stored comprise user's characteristic information; Described user ID generation module comprises:
User's characteristic information extracts submodule, for extracting described data user's characteristic information to be stored;
User ID generates submodule, for recording the mapping of described user's characteristic information and preset storage sequence number, and described corresponding stored sequence number is designated as to user ID.
12. devices as claimed in claim 11, is characterized in that, described target storing address generating module comprises:
Target data form generates submodule, for selecting a data minute storehouse not reach the data form of the preset number of users upper limit, generate target data form;
Data to be stored write submodule, described data to be stored are write to described target data form;
Identifier combination submodule, divide the identifier combination in storehouse to generate target memory address the sign of described target data form and corresponding data.
13. devices as claimed in claim 12, is characterized in that, described mapping relations logging modle comprises:
Key-value pair is set up submodule, for setting up, is selected described user ID to shine upon the key-value pair of described target memory address;
Key-value pair record sub module, record described key-value pair in map listing.
14. devices as claimed in claim 13, is characterized in that, also comprise:
User ID acquisition module, for obtaining data user ID to be added; Described data to be added are to store in the corresponding target memory address of the built user ID of described distributed system, are the supplementary datas of built user ID;
Target memory address acquisition module, for according to described user ID inquiry obtain target memory address;
Target memory address writing module, for described data to be added are write to described target memory address.
15. devices as claimed in claim 14, is characterized in that, also comprise:
According to user ID, target memory address is obtained in inquiry, the corresponding data minute storehouse of pointing to and target data form, extracts the user data in described target memory address from target memory address.
16. devices as claimed in claim 15, is characterized in that, described data to be added comprise user's characteristic information; Described user ID acquisition module comprises:
User's characteristic information extracts submodule, for extracting the user's characteristic information of described data to be added;
Storage sequence number obtains submodule, according to described user's characteristic information mapping, obtains corresponding stored sequence number, and described corresponding stored sequence number is user ID.
17. devices as claimed in claim 16, is characterized in that, if a legacy data minute library storage load reaches the upper limit, increase the unit of described distributed system, and in described unit, set up new data minute storehouse.
18. devices as claimed in claim 10, is characterized in that, the data to be stored of the same user ID of described correspondence or data to be added are stored among a data form.
Data enquire method in 19. 1 kinds of non-relational databases, it is characterized in that, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described method comprises:
The inquiry request that receives user data, described inquiry request comprises user ID;
According to described user ID inquiry, obtain target memory address, described target memory address is generated by the sign of target data form and the identifier combination in corresponding data minute storehouse;
The corresponding data minute storehouse of pointing to from described target memory address and target data form, extract user data;
Return to described user data.
Data query device in 20. 1 kinds of non-relational databases, it is characterized in that, described non-relational database comprises the distributed system of at least two units, described unit comprises at least two data minute storehouse, a described data minute storehouse comprises some data forms, a described data minute storehouse has data minute storehouse sign, and described data form has data form sign;
Described device comprises:
User ID acquisition module, for receiving the inquiry request of user data, described inquiry request comprises user ID;
Target memory address enquiry module, according to described user ID, target memory address is obtained in inquiry, and described target memory address is generated by the sign of target data form and the identifier combination in corresponding data minute storehouse;
Store data acquisition module, for corresponding data minute storehouse and the target data form pointing to from described target memory address, extract user data;
Storage data return to module, for returning to described user data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210266418.8ACN103577440B (en) | 2012-07-27 | 2012-07-27 | A kind of data processing method and device in non-relational database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210266418.8ACN103577440B (en) | 2012-07-27 | 2012-07-27 | A kind of data processing method and device in non-relational database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103577440Atrue CN103577440A (en) | 2014-02-12 |
| CN103577440B CN103577440B (en) | 2017-08-11 |
Family
ID=50049245
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210266418.8AActiveCN103577440B (en) | 2012-07-27 | 2012-07-27 | A kind of data processing method and device in non-relational database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103577440B (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015154625A1 (en)* | 2014-09-05 | 2015-10-15 | 中兴通讯股份有限公司 | Timing event processing method, storage method, execution method and corresponding device |
| CN105005572A (en)* | 2014-04-24 | 2015-10-28 | 中国移动通信集团云南有限公司 | Database mapping method and device |
| CN105302842A (en)* | 2014-07-31 | 2016-02-03 | 北大方正集团有限公司 | Data processing method and device |
| CN105447151A (en)* | 2015-11-27 | 2016-03-30 | 深圳市金蝶友商电子商务服务有限公司 | Method for accessing distributed database, data source proxy apparatus and application server |
| CN105653692A (en)* | 2015-12-30 | 2016-06-08 | 北京奇艺世纪科技有限公司 | File storage method and apparatus based on non-relational database |
| CN105681477A (en)* | 2016-04-08 | 2016-06-15 | 四川师范大学 | Data access method and server |
| CN105828052A (en)* | 2016-06-02 | 2016-08-03 | 中国联合网络通信集团有限公司 | Video monitoring method and monitoring system based on Storm technology |
| CN106294421A (en)* | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of data write, read method and device |
| WO2017088666A1 (en)* | 2015-11-27 | 2017-06-01 | 华为技术有限公司 | Data storage method and coordinator node |
| WO2018019310A1 (en)* | 2016-07-27 | 2018-02-01 | 中兴通讯股份有限公司 | Big data system data backup and recovery methods and devices, and computer storage medium |
| CN108399175A (en)* | 2017-02-07 | 2018-08-14 | 阿里巴巴集团控股有限公司 | A kind of storage of data, querying method and its device |
| CN105224677B (en)* | 2015-10-16 | 2018-10-30 | 上海晶赞科技发展有限公司 | A kind of database operation method and device |
| CN108733787A (en)* | 2018-05-11 | 2018-11-02 | 中国平安人寿保险股份有限公司 | Database operation method, device, electronic equipment and storage medium |
| CN108809952A (en)* | 2018-05-22 | 2018-11-13 | 隆正信息科技有限公司 | Object authentication method and system based on more mark general unique keepsake chains of sectional |
| CN109144404A (en)* | 2017-06-27 | 2019-01-04 | 北京金山安全软件有限公司 | Data storage and query method and device based on multiple DMPs and electronic equipment |
| CN109284340A (en)* | 2018-09-14 | 2019-01-29 | 清华大学 | Storage and retrieval method of relational data in resource-constrained smart objects |
| CN109597834A (en)* | 2018-10-22 | 2019-04-09 | 平安科技（深圳）有限公司 | Mass data storage means, device, medium and equipment based on redis |
| CN110555703A (en)* | 2019-07-31 | 2019-12-10 | 阿里巴巴集团控股有限公司 | Credit contract multi-view display method and device |
| CN110659258A (en)* | 2019-09-06 | 2020-01-07 | 平安普惠企业管理有限公司 | Document generation method, system, computer device and computer readable storage medium |
| CN110727739A (en)* | 2018-06-28 | 2020-01-24 | 北京京东尚科信息技术有限公司 | Data storage method and device |
| CN110851435A (en)* | 2018-08-03 | 2020-02-28 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| CN111581049A (en)* | 2020-04-08 | 2020-08-25 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for monitoring running state of distributed system |
| CN111651424A (en)* | 2020-06-10 | 2020-09-11 | 中国科学院深圳先进技术研究院 | A data processing method, device, data node and storage medium |
| CN111680313A (en)* | 2020-06-08 | 2020-09-18 | 南京领行科技股份有限公司 | Data processing method, device, equipment and storage medium |
| CN112632117A (en)* | 2020-12-30 | 2021-04-09 | 广州华多网络科技有限公司 | Method and device for processing serial number data, electronic equipment and storage medium |
| CN113111138A (en)* | 2021-05-10 | 2021-07-13 | 网易传媒科技（北京）有限公司 | Data processing method, device, computing equipment and medium |
| CN113420186A (en)* | 2021-06-18 | 2021-09-21 | 自然资源部第三地形测量队 | Data storage method, data storage device, computer readable storage medium and data reading method |
| CN113806405A (en)* | 2021-09-18 | 2021-12-17 | 王剑 | Method for inquiring and storing medical record data and related device |
| CN118779288A (en)* | 2024-08-13 | 2024-10-15 | 企查查科技股份有限公司 | Data interaction method, device, computer equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002063055A (en)* | 2000-08-23 | 2002-02-28 | Ricoh Co Ltd | Write delay database management system and system |
| JP4525074B2 (en)* | 2003-12-24 | 2010-08-18 | 日本電気株式会社 | Product identification information management system |
| US8407250B2 (en)* | 2006-08-07 | 2013-03-26 | Google Inc. | Distribution of content document to varying users with security customization and scalability |
| CN101330431B (en)* | 2007-06-19 | 2011-04-06 | 阿里巴巴集团控股有限公司 | Method and system for storing instant information |
| CN102314480B (en)* | 2011-07-05 | 2013-04-10 | 万达信息股份有限公司 | Distributed data storage method specific to mass data |
- 2012
- 2012-07-27CNCN201210266418.8Apatent/CN103577440B/enactiveActive
Cited By (43)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005572A (en)* | 2014-04-24 | 2015-10-28 | 中国移动通信集团云南有限公司 | Database mapping method and device |
| CN105302842A (en)* | 2014-07-31 | 2016-02-03 | 北大方正集团有限公司 | Data processing method and device |
| CN105468346A (en)* | 2014-09-05 | 2016-04-06 | 中兴通讯股份有限公司 | Timing event management method, storage method, execution method and corresponding apparatus |
| WO2015154625A1 (en)* | 2014-09-05 | 2015-10-15 | 中兴通讯股份有限公司 | Timing event processing method, storage method, execution method and corresponding device |
| CN106294421B (en)* | 2015-05-25 | 2020-02-04 | 阿里巴巴集团控股有限公司 | Data writing and reading method and device |
| CN106294421A (en)* | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of data write, read method and device |
| CN105224677B (en)* | 2015-10-16 | 2018-10-30 | 上海晶赞科技发展有限公司 | A kind of database operation method and device |
| CN105447151A (en)* | 2015-11-27 | 2016-03-30 | 深圳市金蝶友商电子商务服务有限公司 | Method for accessing distributed database, data source proxy apparatus and application server |
| WO2017088666A1 (en)* | 2015-11-27 | 2017-06-01 | 华为技术有限公司 | Data storage method and coordinator node |
| CN105653692A (en)* | 2015-12-30 | 2016-06-08 | 北京奇艺世纪科技有限公司 | File storage method and apparatus based on non-relational database |
| CN105653692B (en)* | 2015-12-30 | 2019-06-21 | 北京奇艺世纪科技有限公司 | A kind of file memory method and device based on non-relational database |
| CN105681477A (en)* | 2016-04-08 | 2016-06-15 | 四川师范大学 | Data access method and server |
| CN105681477B (en)* | 2016-04-08 | 2018-11-16 | 四川师范大学 | A kind of data access method and a kind of server |
| CN105828052A (en)* | 2016-06-02 | 2016-08-03 | 中国联合网络通信集团有限公司 | Video monitoring method and monitoring system based on Storm technology |
| WO2018019310A1 (en)* | 2016-07-27 | 2018-02-01 | 中兴通讯股份有限公司 | Big data system data backup and recovery methods and devices, and computer storage medium |
| CN108399175A (en)* | 2017-02-07 | 2018-08-14 | 阿里巴巴集团控股有限公司 | A kind of storage of data, querying method and its device |
| CN108399175B (en)* | 2017-02-07 | 2022-04-12 | 阿里巴巴集团控股有限公司 | Data storage and query method and device |
| CN109144404A (en)* | 2017-06-27 | 2019-01-04 | 北京金山安全软件有限公司 | Data storage and query method and device based on multiple DMPs and electronic equipment |
| CN108733787A (en)* | 2018-05-11 | 2018-11-02 | 中国平安人寿保险股份有限公司 | Database operation method, device, electronic equipment and storage medium |
| CN108809952A (en)* | 2018-05-22 | 2018-11-13 | 隆正信息科技有限公司 | Object authentication method and system based on more mark general unique keepsake chains of sectional |
| CN110727739A (en)* | 2018-06-28 | 2020-01-24 | 北京京东尚科信息技术有限公司 | Data storage method and device |
| CN110851435B (en)* | 2018-08-03 | 2022-02-11 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| CN110851435A (en)* | 2018-08-03 | 2020-02-28 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| CN109284340A (en)* | 2018-09-14 | 2019-01-29 | 清华大学 | Storage and retrieval method of relational data in resource-constrained smart objects |
| CN109284340B (en)* | 2018-09-14 | 2020-11-20 | 清华大学 | Storage and retrieval method of relational data in resource-constrained smart objects |
| CN109597834A (en)* | 2018-10-22 | 2019-04-09 | 平安科技（深圳）有限公司 | Mass data storage means, device, medium and equipment based on redis |
| CN109597834B (en)* | 2018-10-22 | 2024-05-07 | 平安科技（深圳）有限公司 | Massive data storage method, device, medium and equipment based on redis |
| CN110555703A (en)* | 2019-07-31 | 2019-12-10 | 阿里巴巴集团控股有限公司 | Credit contract multi-view display method and device |
| CN110555703B (en)* | 2019-07-31 | 2023-05-23 | 创新先进技术有限公司 | Credit contract multi-view display method and device |
| CN110659258A (en)* | 2019-09-06 | 2020-01-07 | 平安普惠企业管理有限公司 | Document generation method, system, computer device and computer readable storage medium |
| CN111581049A (en)* | 2020-04-08 | 2020-08-25 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for monitoring running state of distributed system |
| CN111581049B (en)* | 2020-04-08 | 2023-10-10 | 北京百度网讯科技有限公司 | Distributed system running state monitoring method, device, equipment and storage medium |
| US12210507B2 (en) | 2020-04-08 | 2025-01-28 | Beijing Baidu Netcom Science Technology Co., Ltd. | Method for monitoring running state of distributed system, device, and storage medium |
| CN111680313A (en)* | 2020-06-08 | 2020-09-18 | 南京领行科技股份有限公司 | Data processing method, device, equipment and storage medium |
| CN111680313B (en)* | 2020-06-08 | 2022-06-28 | 南京领行科技股份有限公司 | Data processing method, device, equipment and storage medium |
| CN111651424A (en)* | 2020-06-10 | 2020-09-11 | 中国科学院深圳先进技术研究院 | A data processing method, device, data node and storage medium |
| CN111651424B (en)* | 2020-06-10 | 2024-05-03 | 中国科学院深圳先进技术研究院 | Data processing method, device, data node and storage medium |
| CN112632117A (en)* | 2020-12-30 | 2021-04-09 | 广州华多网络科技有限公司 | Method and device for processing serial number data, electronic equipment and storage medium |
| CN113111138A (en)* | 2021-05-10 | 2021-07-13 | 网易传媒科技（北京）有限公司 | Data processing method, device, computing equipment and medium |
| CN113420186A (en)* | 2021-06-18 | 2021-09-21 | 自然资源部第三地形测量队 | Data storage method, data storage device, computer readable storage medium and data reading method |
| CN113420186B (en)* | 2021-06-18 | 2022-10-04 | 自然资源部第三地形测量队 | Data storage method, data storage device, computer readable storage medium and data reading method |
| CN113806405A (en)* | 2021-09-18 | 2021-12-17 | 王剑 | Method for inquiring and storing medical record data and related device |
| CN118779288A (en)* | 2024-08-13 | 2024-10-15 | 企查查科技股份有限公司 | Data interaction method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103577440B (en) | 2017-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103577440A (en) | Data processing method and device in non-relational database | |
| US10860598B2 (en) | Systems and methods for interest-driven business intelligence systems including event-oriented data | |
| CN102270225B (en) | Data change daily record method for supervising and data change daily record supervising device | |
| CN104199986B (en) | Vector data space index method based on hbase and geohash | |
| WO2020192064A1 (en) | Incremental data consistency implementation method and device | |
| WO2006046669A1 (en) | Database management device, method and program | |
| CN105183735A (en) | Data query method and query device | |
| CN105631003A (en) | Intelligent index establishing, inquiring and maintaining method supporting mass data classification and counting | |
| CN102243660A (en) | Data access method and device | |
| CN103399945A (en) | Data structure based on cloud computing database system | |
| CN101963993B (en) | Method for fast searching database sheet table record | |
| US11520763B2 (en) | Automated optimization for in-memory data structures of column store databases | |
| CN102867066A (en) | Data summarization device and data summarization method | |
| CN104636349A (en) | Method and equipment for compression and searching of index data | |
| CN103455335A (en) | Multilevel classification Web implementation method | |
| CN110134683A (en) | Partition optimization research method and system for massive element storage in relational database | |
| CN112667859A (en) | Data processing method and device based on memory | |
| CN113761102B (en) | Data processing method, device, server, system and storage medium | |
| CN114579617A (en) | Data query method and device, computer equipment and storage medium | |
| CN111522820A (en) | Data storage structure, storage retrieval method, system, device and storage medium | |
| CN111666302A (en) | User ranking query method, device, equipment and storage medium | |
| CN115809268A (en) | Self-adaptive query method and device based on fragment index | |
| CN114996270A (en) | Method and device for inquiring paging data | |
| CN103544305A (en) | Two-dimensional dynamic contact inquiry list inquiring method and system based on double-hash structure | |
| CN103617181A (en) | Method and device for establishing universal database of relationships |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | Effective date of registration:20191211 Address after:P.O. Box 31119, grand exhibition hall, hibiscus street, 802 West Bay Road, Grand Cayman, Cayman Islands Patentee after:Innovative advanced technology Co., Ltd Address before:A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Patentee before:Alibaba Group Holding Co., Ltd. | |
| TR01 | Transfer of patent right |