A kind of processing method and processing device of operation requestsTechnical field
The present invention relates to data processing field more particularly to a kind of processing method and processing devices of operation requests.
Background technique
Memcache is a set of distributed high speed key-value pair (Key-Value) caching system, as key-value pair (Key-Value) caching system, by very extensive use in webpage (web) service, in LAMP (Linux-Apache-MySQL-PHP) in framework, even more indispensable part.LAMP framework refers to comprising (SuSE) Linux OS, Apache network server,MySQL database, the frame of Perl, PHP or Python programming language.
But the system as full memory storage, memcache is the case where delay machine or service exception collapse (crash)Under, it may appear that the problem of data cached complete loss.Although memcache also provides the scheme of data fragmentation storage, due toSingle fragment still has the possibility of loss of data, therefore still can not accomplish the reliable memory of data on the whole.
The problem of causing in this way is also obvious:
(1) memcache is relied at one as in the busy system cached, if data occur complete for caching systemIt loses, request penetrates (i.e. directly inquiry database) meeting to the stability structure of current system caused by being declined by cache hit rateAt havoc.
(2) in today exigent for Information Security, data disaster tolerance, requirement of more copy storages as hardness.And memcache is unable to reach the demand of more copies and disaster tolerance due to not having the synchronous function of data.
Summary of the invention
The application provides a kind of processing method and processing device of operation requests, can be serviced by redis to useThe system architecture of memcache provides caching function, and can greatly improve the safety of data.
The application adopts the following technical scheme that.
A kind of processing method of operation requests, comprising:
After redis main thread receives the operation requests using memcache communication protocol, according to memcache communication protocolThe operation requests are parsed, memcache order is obtained；
The redis main thread carries out corresponding operating to internal storage data according to the memcache order；
Periodically disk is written in the internal storage data by the redis main thread.
Wherein, described to may include: according to the memcache communication protocol parsing operation requests
Memcache communication protocol used in determining is text text agreement or binary system binary agreement；
Corresponding protocol analysis call back function is called to parse the operation requests.
Wherein, internal storage data write-in disk periodically be may include: by the redis main thread
The redis main thread periodically generates memory image；
The redis main thread is based on the memory image, and disk is written in internal storage data.
Wherein, the processing method can also include:
The redis main thread generates memory image when receiving from the synchronization request of database instance；
The memory image is sent to from database instance by the redis main thread, same for carrying out from database instanceStep.
Wherein, the redis main thread, which carries out corresponding operating to internal storage data according to the memcache order, can wrapIt includes:
The redis main thread calls the corresponding order call back function of the memcache order, according to describedMemcache order handles internal storage data.
Wherein, internal storage data write-in disk periodically be may include: by the redis main thread
The memcache order is converted into redis instruction by the redis main thread, is recorded in log；
The redis main thread is based on log and disk is written in the internal storage data.
Wherein, can also include: after the memcache order is converted into redis instruction by the redis main thread
The redis being converted to instruction is sent to from database instance by the redis main thread, for from database instanceIt synchronizes.
A kind of processing unit of operation requests, comprising: processor and memory；
The memory is used to save the program for processing operation request；The program for processing operation request existsWhen reading execution by the processor, proceed as follows:
After receiving the operation requests using memcache communication protocol, the behaviour is parsed according to memcache communication protocolIt requests, obtains memcache order；
Corresponding operating is carried out to internal storage data according to the memcache order；
Disk periodically is written into the internal storage data.
Wherein, the program for processing operation request can also carry out such as when reading execution by the processorLower operation:
The redis main thread generates memory image when receiving from the synchronization request of database instance；
The memory image is sent to from database instance, for being synchronized from database instance.
Wherein, internal storage data write-in disk be may include: by the periodicity
The memcache order is converted into redis instruction, is recorded in log；
Disk is written into the internal storage data based on log.
A kind of processing unit of operation requests, comprising:
Parsing module, for being communicated according to memcache after receiving the operation requests using memcache communication protocolOperation requests described in protocol analysis obtain memcache order；
Execution module, for carrying out corresponding operating to internal storage data according to the memcache order；
Persistence module, for disk periodically to be written in the internal storage data.
In at least one embodiment of the application, the system architecture of caching function can be provided using memcache originallyIn, it replaces memcache to provide caching function by redis service, original system architecture can be compatible with, to data base setIt is transparent for system and client；By carrying out persistence to internal storage data, the stability of system can be greatly improved.
In a kind of implementation of the present embodiment, memory image can be generated when meeting trigger condition；On this basis,It is also based on memory image and carries out persistence, snapshot and other database instances can also realize that full dose is synchronous based on memory.Wherein, persistence can guarantee that data are not lost after server delay machine is restarted and in the case where servicing exception crash.
In a kind of implementation of the present embodiment, the operation requests of memcache can be converted into redis instruction, and rememberRecord is in log, so as to carry out persistence based on the log.
In a kind of implementation of the present embodiment, on the basis of data are synchronous, the distribution of access request, Ke Yishi are providedMore copies storage of existing data, thus the problem of making the possibility that is implemented as of read and write abruption, also can solve hot spot key.By sameMore copies storage that function introduces is walked, the safety of data is fundamentally improved.
Certainly, any product for implementing the application does not necessarily require achieving all the advantages described above at the same time.
Detailed description of the invention
Fig. 1 is the flow chart of the processing method of the operation requests of embodiment one；
Fig. 2 is the schematic diagram of redis main thread in the example of embodiment one；
Fig. 3 is the schematic diagram of master-slave synchronisation process in the example of embodiment one；
Fig. 4 is the flow chart of the processing unit of the operation requests of embodiment three.
Specific embodiment
It is described in detail below in conjunction with technical solution of the accompanying drawings and embodiments to the application.
It should be noted that the different characteristic in the embodiment of the present application and implementation can be mutual if do not conflictedIn conjunction within the scope of protection of this application.In addition, though logical order is shown in flow charts, but in certain feelingsIt, can be with the steps shown or described are performed in an order that is different from the one herein under condition.
In one configuration, carry out operation requests processing calculating equipment may include one or more processors (CPU), it is defeatedEnter/output interface, network interface and memory (memory).
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/orThe forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer-readable mediumExample.Memory may include one or more modules.
Computer-readable medium includes permanent and non-permanent, removable and non-movable storage medium, can be by appointingWhat method or technique realizes that information stores.Information can be computer readable instructions, data structure, the module of program or otherData.The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory(SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only memory(ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory techniques, CD-ROM are read-onlyMemory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magneticProperty storage equipment or any other non-transmission medium, can be used for storing and can be accessed by a computing device information.
Embodiment one, a kind of processing method of operation requests, as shown in Figure 1, including step S110~S130:
After S110, redis main thread receive the operation requests using memcache communication protocol, communicated according to memcacheOperation requests described in protocol analysis obtain memcache order；
S120, the redis main thread carry out corresponding operating to internal storage data according to the memcache order；
Periodically disk is written in the internal storage data by S130, the redis main thread.
Redis is the high performance Key-Value cache database system of an open source；In the present embodiment, redis is utilizedThe function of service simulation memcache, provides buffer service；For client, not needing to distinguish is genuine memcacheBuffer service is provided in simulation memcache providing buffer service or redis, redis can be enabled to be compatible with existingThe system architecture of buffer service is provided using memcache, and redis can provide persistence function, it can be ensured that as dataThe internal storage data of banked cache just will not all be lost because of delay machine or system crash, so as to avoid a large amount of inquiry request straightThe problem of being connected to up to database, and influencing system stability generation.
In order to solve the problems, such as that memcache lacks persistence and data synchronizing function, a solution is directly to use itThere is the storage system of persistence and data synchronizing function directly to replace memcache for it, but the solution needs to change instituteThere is the code of client, cost is too big.
Another solution is the data being stored in memcache in caching, is saved in other storage system,Persistence and data synchronizing function are provided by newly-increased storage system；The solution will lead to overhead and greatly increase, and oneAspect, system need additional resource to run the process of newly-increased storage system；On the other hand, memcache process and newly-increasedStorage system process between need to interact, to cooperate, interaction also bring along expense.
And in the present embodiment, by running a redis process (containing at least one redis main thread in process)It supports memcache operation, does not need memcache process of reruning, avoid and handed between the expense and process of newly-increased processMutual expense；And redis process can be compatible with and provide the system architecture of caching function using memcache, not need to clientEnd is modified.In addition, since redis itself can also support redis communication protocol, running a redis process can be withTwo kinds of storage services: redis and memcache are provided for system architecture.
In the present embodiment, persistence, which can refer to, is written disk for the data in memory.It that is: will be in volatile storage carrierData stored with certain format on non-volatile memories carrier；Such as when the present embodiment is applied to LAMP framework, the periodProperty write-in disk internal storage data in also include as MySQL database caching part data, such MySQL databaseIt is data cached just to be lost completely because of delay machine meeting system crash, substantially increase data stability.
It, can be directly according to redis if the communication protocol that operation requests use is redis agreement in the present embodimentPrimary process is handled.
In the present embodiment, memcache order may include completely or partially ordering provided by current memcache systemIt enables.
In the present embodiment, the operation carried out to internal storage data may include one or more of: reading and writing, inquiry, systemMeter, modification etc..
In the present embodiment, it can be regarded as using the operation requests of memcache communication protocol using memcache communication protocolsThe memcache order for discussing encapsulation, according to available memcache order after the parsing of memcache communication protocol.
Wherein, memcache communication protocol is a kind of establish in transmission control protocol (Transmission ControlProtocol, tcp) on data exchange delivery protocol, can be mainly divided into text (text) agreement and binary system (binary)Agreement.
In a kind of implementation, parsing the operation requests according to memcache communication protocol may include:
Memcache communication protocol used in determining is text agreement or binary agreement；
Corresponding protocol analysis call back function is called to parse the operation requests.
In this implementation, for calling text protocol analysis call back function using the operation requests of text agreement, forBinary protocol analysis call back function is called using the operation requests of binary agreement.
In this implementation, protocol analysis call back function can be to be write in advance.
In a kind of implementation, the processing method can also include:
Memory image is generated when meeting scheduled trigger condition.
In this implementation, memory image can be, but not limited to be RDB file.
In this implementation, scheduled trigger condition may include the key arrival predetermined number that timing cycle is reached or increased newlyAmount, that is, periodically generate memory image；Scheduled trigger condition also may include receiving predetermined request, such as primary database exampleRedis main thread receive the synchronization request (SYNC) from the redis main thread of database instance.
In this implementation, redis process can go out a subprocess with bifurcated (fork), then recycle in subprocessAll data in the region of memory of redis system are write data for RDB file as.
In this implementation, the memory image of generation can be used for carrying out persistence, can also be in master-slave database exampleBetween used when full dose data are synchronous.
In a kind of implementation, the method for the present embodiment can also include:
The redis main thread generates memory image when receiving from the synchronization request of database instance；
Memory image is sent to from database instance by the redis main thread, for synchronizing from database instance.
Wherein, above-mentioned redis main thread can refer to the redis main thread of primary database example；It can from database instanceTo realize that full dose is synchronous with primary database example by being loaded into memory image.
In this implementation, can require no knowledge about primary database example from database instance is to perform memcache lifeIt enables, still performs redis instruction, base can be carried out only according to the synchronous process of redis primary full dose and primary database exampleIn the synchronization of memory image.
In a kind of implementation, internal storage data write-in disk periodically be may include: by redis main thread
The redis main thread periodically generates memory image；
The redis main thread is based on the memory image, and disk is written in internal storage data.
It is the equal of that snapshot carries out persistence based on memory in this implementation.
In other implementations, persistence can also be carried out based on other way.
In a kind of implementation, redis main thread carries out corresponding operating to internal storage data according to the memcache orderMay include:
Redis main thread calls the corresponding order call back function of the memcache order, orders according to the memcacheOrder handles internal storage data.
In this implementation, order call back function can be to be write in advance, and that registers in redis process initiation returnsLetter of transfer number.
In this implementation, the corresponding relationship between each memcache order and order call back function can be preset；Redis main thread can determine the corresponding order call back function of a memcache order according to the corresponding relationship.
In this implementation, the treatment effect of order call back function and memcache system to the same memcache orderIt can be equivalent.
In other implementations, redis main thread carries out corresponding operating according to the memcache order and also may include:Memcache order is converted into corresponding redis instruction, then executes redis instruction.But if using this " conversionExecute again " implementation, then need first to judge whether the processing logic of memcache order correct, not so execute and be converted intoRedis instruction may cause maloperation；And if call instruction call back function executes memcache order, can not needJudged directly learn whether processing logic is correct according to implementing result, for example run succeeded, mean handling logic justReally；Processing logic error, which then will lead to, executes failure.
In this implementation, internal storage data write-in disk periodically be may include: by redis main thread
The memcache order is converted into redis instruction by the redis main thread, is recorded in log；
The redis main thread is based on log and disk is written in the internal storage data.
In this implementation, the corresponding relationship between each memcache order and redis instruction can be pre-saved, according toThe corresponding relationship is converted.
In this implementation, held using the mode based on only additional file (Append Only File, AOF)Longization.
In other implementations, persistence can also be carried out based on RDB file or other means.
In a kind of implementation, can also include: after memcache order to be converted into redis instruction
The redis being converted to instruction is sent to from database instance by redis main thread, for carrying out from database instanceIt is synchronous.
In this implementation, carries out step S110, S120 and memcache order is converted into redis instructionRedis main thread can regard the redis main thread of primary database example as.
In this implementation, redis main thread can be sent to after being converted to one or more redis instructions from numberAccording to library example.
It, can be according to redis from the redis main thread of database instance after receiving redis instruction in this implementationAgreement is parsed, and handles the redis instruction after parsing, persistence is carried out, to complete increment synchronization.
In this implementation, can require no knowledge about primary database example from database instance is to perform memcache lifeEnable, still perform redis instruction, can only according to primary database example send redis instruction and primary database example intoRow redis primary increment synchronization.
In this implementation, primary database example can handle memcache order by call instruction call back function,It is obtained data cached to can be dictionary data；The redis instruction that memcache order is converted into is executed from database instance,Obtained is also the data of same dictionary structure.
In a kind of implementation, in the case where master-slave database example is synchronous, more copies operations can be provided；More copiesRefer to that a data save in multiple storages, more copies, and can be real primarily to guarantee the redundant safety of dataExisting read and write abruption evades the problem of hot spot key.
In this implementation, can also include: before the step S110
According to predetermined policy, operation requests are distributed into different database instances and are handled.
In this implementation, be receive the database instance of operation requests redis main thread carry out step S110,S120。
In this implementation, predetermined policy, which can be, is assigned to the operation requests of reading and writing different database instances；In advanceDetermining strategy can also be according to load balancing or other tactful batch operation requests to different database instances.Practical applicationIn, it can designed, designed predetermined policy according to demand.
Illustrate above-described embodiment with an example below.
Fig. 2 is the schematic diagram of redis main thread when realizing memcaceh based on redis in this example, redis main threadIn include:
Redis I/O event processing module 21, data communication protocol judgment module 22, memcache protocol resolution module 23,It is lasting that memcache command processing module 24, memcache order turn redis instruction module 25, redis AOF and memory imageChange module 26, redis master-slave synchronisation module 27.
In addition, module that can also be primary including redis in redis main thread, such as redis protocol resolution module,Redis command process module etc..
In this example, the above-mentioned module in redis main thread can respectively regard one section of execution code as；Above-mentioned module is when neededThe call back function pre-saved can be called.
The workflow of above-mentioned module is as follows:
After process initiation, redis main thread starts redis I/O event processing module 21 and receives I/O event, is receivingAfter complete operation requests, notification data communication protocol judgment module 22；
Data communication protocol judgment module 22 is if it is judged that the received operation requests of institute are memcache requests (that is, makingIt is memcache communication protocol), then notify memcache protocol resolution module 23；
The meeting of memcache protocol resolution module 23 memcache communication protocol according to used by operation requests is text associationView or binary agreement, call corresponding protocol analysis call back function, carry out protocol analysis to operation requests, obtain memcacheOrder.
After protocol analysis, the memcache order that memcache command processing module 24 can be obtained according to parsing,And the corresponding relationship between the order call back function registered when different memcache order and redis process initiation, it callsCorresponding order call back function carries out respective handling according to the memcache order that parsing obtains；
After memcache command process, memcache order, which turns redis instruction module 25, to order memcacheOrder is converted into redis instruction；
After conversion, redis AOF and memory image persistence module 26 and redis master-slave synchronisation module 27 can be intoThe persistence of row data with it is synchronous.
In this example, persistence, which refers to store data from volatile row, stores the mistake stored to non-volatile row with certain formatJourney.Synchronize refer to the process of data from a database instance be transmitted to other database instance (usually principal and subordinate's example itBetween).
This example increases memcache protocol resolution module in redis main thread, memcache request processing module,Memcache instruction turns redis instruction module, to hold to use memcache to increase data as the system architecture of cachingLongization and synchronous function.
In this example, process when realizing the master-slave synchronisation of memcache based on redis is as shown in figure 3, wherein MasterFor main database instance, Slave is from database instance, and Master is from receiving an operation using memcache communication protocolRequest includes the following steps 301~305 to the process for carrying out master-slave synchronisation:
301, the redis main thread of Master carries out memcache protocol analysis, obtains memcache order；
302, the redis main thread of Master carries out memcache command process, carries out corresponding operation；
303, memcache order is converted into redis instruction by the redis main thread of Master, and is recorded in log；
304, snapshot or log carry out persistence to data to the redis main thread of Master based on memory；
305, the redis main thread of Master carries out redis master-slave synchronisation.
The sequencing of above-mentioned steps 304,305 is unlimited；And do not have between 304,305 and 301~303 inevitable successiveSequentially, for example 304 can be and periodically carry out, and 305 can be and carry out after the synchronization request for receiving Slave.
Wherein, memory image can generate or the period when needed (such as after receiving the synchronization request of Slave for the first time)Property generate.
When being synchronized for the first time between Master and Slave, memory image (rdb file) can be sent by MasterTo Slave, it is synchronous to carry out full dose that Slave is loaded into rdb file.After full dose synchronizes, it is subsequent between Master and Slave can be intoFull dose is synchronized the redis instruction that rear received request is parsed and is converted into, is sent to Slave by row increment synchronization, Master.
The operation that the redis main thread of Slave carries out in increment synchronization includes the following steps 3051~3053:
3051, it is instructed using the redis that redis protocol analysis receives；
3052, the redis instruction after parsing is handled；
3053, persistence is carried out to data.
Module included by the redis main thread of Slave and the redis main thread of Master, may be implemented in other wordsFunction is the same.Slave can have one or more.
Embodiment two, a kind of processing unit of operation requests, comprising: processor and memory；
The memory is used to save the program for processing operation request；The program for processing operation request existsWhen reading execution by the processor, proceed as follows:
After receiving the operation requests using memcache communication protocol, the behaviour is parsed according to memcache communication protocolIt requests, obtains memcache order；
Corresponding operating is carried out to internal storage data according to the memcache order；
Disk periodically is written into the internal storage data.
In the present embodiment, when the program for processing operation request is read execution, redis main thread can be run, byRedis main thread completes aforesaid operations.
In a kind of implementation, parsing the operation requests according to memcache communication protocol may include:
Memcache communication protocol used in determining is text text agreement or binary system binary agreement；
Corresponding protocol analysis call back function is called to parse the operation requests.
In a kind of implementation, internal storage data write-in disk periodically be may include: by redis main thread
The redis main thread periodically generates memory image；
The redis main thread is based on the memory image, and disk is written in internal storage data.
In a kind of implementation, for processing operation request program read by the processor execute when, can be withIt proceeds as follows:
The redis main thread generates memory image when receiving from the synchronization request of database instance；
The memory image is sent to from database instance, for being synchronized from database instance.
In a kind of implementation, carrying out corresponding operating to internal storage data according to the memcache order may include:
The corresponding order call back function of the memcache order is called, according to the memcache order to internal storage dataIt is handled.
In a kind of implementation, periodically may include: by internal storage data write-in disk
The memcache order is converted into redis instruction, is recorded in log；
Disk is written into the internal storage data based on log.
In this implementation, can also include: after memcache order is converted into redis instruction
The redis being converted to instruction is sent to from database instance, for being synchronized from database instance.
In the present embodiment, for processing operation request program read by the processor execute when, the behaviour that is carried outStep S110~the S120 that can correspond to embodiment one, the program for processing operation request are being read execution Shi SuojinOther realization details of row operation, reference can be made to embodiment one.
Embodiment three, a kind of processing unit of operation requests, as shown in Figure 4, comprising:
Parsing module 41, for being led to after receiving the operation requests using memcache communication protocol according to memcacheBelieve operation requests described in protocol analysis, obtains memcache order；
Execution module 42, for carrying out corresponding operating to internal storage data according to the memcache order；
Persistence module 43, for disk periodically to be written in the internal storage data.
In the present embodiment, parsing module, execution module, persistence module can be the corresponding journey of redis main thread respectivelyIn sequence code, realize that memcache communication protocol parsing function, memcache order execute the program of function and persistence functionCode.
In a kind of implementation, parsing module parses the operation requests according to memcache communication protocol and may include:
Parsing module determines that used memcache communication protocol is text text agreement or binary system binary agreement；
Parsing module calls corresponding protocol analysis call back function to parse the operation requests.
In a kind of implementation, internal storage data write-in disk periodically be may include: by persistence module
Persistence module periodically generates memory image；
Persistence module is based on the memory image, and disk is written in internal storage data.
In a kind of implementation, the processing unit can also include:
Synchronization module, it is for generating memory image when receiving from the synchronization request of database instance, the memory is fastIt is approved for distribution to give from database instance, for being synchronized from database instance.
In a kind of implementation, execution module carries out corresponding operating to internal storage data according to the memcache order can be withInclude:
Execution module calls the corresponding order call back function of the memcache order, according to the memcache order pairInternal storage data is handled.
In a kind of implementation, internal storage data write-in disk periodically be may include: by persistence module
The memcache order is converted into redis instruction by the persistence module, is recorded in log；
The persistence module is based on log and disk is written in the internal storage data.
In this implementation, the processing unit can also include:
Synchronization module, for being sent to the redis being converted to instruction from database instance, for from database instance intoRow synchronizes.
In the present embodiment, the operation that parsing module, execution module, persistence module are carried out can correspond respectively to implementOther realization details of step S110~S120 of example one, each module can be found in embodiment one.
Those of ordinary skill in the art will appreciate that all or part of the steps in the above method can be instructed by programRelated hardware is completed, and described program can store in computer readable storage medium, such as read-only memory, disk or CDDeng.Optionally, one or more integrated circuits can be used also to realize in all or part of the steps of above-described embodiment.AccordinglyGround, each module/unit in above-described embodiment can take the form of hardware realization, can also use the shape of software function moduleFormula is realized.The application is not limited to the combination of the hardware and software of any particular form.
Certainly, the application can also have other various embodiments, ripe without departing substantially from the application spirit and its essenceVarious corresponding changes and modifications, but these corresponding changes and change ought can be made according to the application by knowing those skilled in the artShape all should belong to the protection scope of claims hereof.