Summary of the invention
In view of this, being designed to provide for the embodiment of the present application is a kind of without lock shared drive processing method, device, electronicsEquipment and readable storage medium storing program for executing, to solve or improve the above problem.
According to the one aspect of the embodiment of the present application, a kind of electronic equipment is provided, may include that one or more storages are situated betweenMatter and one or more processors communicated with storage medium.One or more storage mediums are stored with the executable machine of processorDevice executable instruction.When electronic equipment operation, processor executes the machine-executable instruction, following total without lock to executeEnjoy internal memory processing method.
It according to the another aspect of the embodiment of the present application, provides a kind of without lock shared drive processing method, is set applied to electronicsIt is standby, which comprises
For each data service, the memory configurations data of the data service are configured, the memory configurations data include instituteData block size, data number of blocks and the shared memory file path needed；
Shared memory space needed for calculating the data service according to the data block size and data number of blocks；
Shared memory file is opened according to the shared memory file path, and according to the shared memory space, is passed throughThe shared memory file is mapped in shared drive region by preset memory mapping mode, and obtains reflecting after the completion of mappingPenetrate return value；
The head pointer address of corresponding mapping area in the shared drive region is obtained according to the mapping return valueWith offset record pointer address, and by the head pointer address and offset record pointer address the data service pair is recordedIn the memory for the application program answered；
It is shared drive region distribution index after the atom map for carrying out auxiliary data to the shared drive regionArea, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete the total without lock of the data serviceEnjoy memory processing.
It according to the another aspect of the embodiment of the present application, provides a kind of without lock shared drive processing unit, is set applied to electronicsStandby, described device includes:
Data configuration module, it is described interior for configuring the memory configurations data of the data service for each data serviceDepositing configuration data includes required data block size, data number of blocks and shared memory file path；
Computing module, in being shared needed for calculating the data service according to the data block size and data number of blocksDeposit space；
Memory mapping block, for opening shared memory file according to the shared memory file path, and according to describedThe shared memory file is mapped in shared drive region by shared memory space by preset memory mapping mode, andObtain the mapping return value after the completion of mapping；
Address logging modle, for obtaining corresponding map section in the shared drive region according to the mapping return valueThe head pointer address and offset record pointer address in domain, and the head pointer address and offset record pointer address are recordedInto the memory of the corresponding application program of the data service；
Distribution module is described shared interior after the atom map to shared drive region progress auxiliary dataRegion distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area are deposited, to complete the numberAccording to business without lock shared drive processing.
According to the another aspect of the embodiment of the present application, a kind of readable storage medium storing program for executing is provided, is stored on the readable storage medium storing program for executingThere is machine-executable instruction, can be executed when which is run by processor above-mentioned without lock shared drive processing methodThe step of.
Based on any of the above-described aspect, shared memory file is mapped to altogether by the embodiment of the present application in such a way that memory mapsIt enjoys in region of memory, and according to the mapping return value after the completion of mapping by corresponding mapping area in obtained shared drive regionHead pointer address be recorded in the memory of the corresponding application program of data service with offset record pointer address.It is basic hereinOn, with the atomic data structure of atomic update, and by data be divided into index area, Hash array data area, extract pond withAnd the method for multiple atom integers of storage data area, without setting privately owned retrieval data for each process or thread,Committed memory is smaller when the processing of data high concurrent, so that can provide high concurrent in live broadcast service for high speed development businessReading and writing data service, and the high performance read-write of high concurrent and storage scheme are provided for memory database.
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present applicationIn attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it should be understood that attached drawing in the applicationThe purpose of illustration and description is only played, is not used to limit the protection scope of the application.In addition, it will be appreciated that schematical attachedFigure does not press scale.Process used herein shows real according to some embodiments of the embodiment of the present applicationExisting operation.It should be understood that the operation of flow chart can be realized out of order, the step of context relation of logic can be withReversal order is implemented simultaneously.In addition, those skilled in the art under the guide of teachings herein, can add to flow chartOther one or more operations, can also remove one or more operations from flow chart.
In addition, described embodiments are only a part of embodiments of the present application, instead of all the embodiments.Usually existThe component of the embodiment of the present application described and illustrated in attached drawing can be arranged and be designed with a variety of different configurations herein.CauseThis, is not intended to limit claimed the application's to the detailed description of the embodiments herein provided in the accompanying drawings belowRange, but it is merely representative of the selected embodiment of the application.Based on embodiments herein, those skilled in the art are not being doneEvery other embodiment obtained under the premise of creative work out, shall fall in the protection scope of this application.
Server is a multicore processing equipment mostly, and the most important mode of Performance tuning for server is parallelProgramming, so that server can be with some shared datas of parallel processing.For example, server can using liquidate impact lock by the way of,I.e. multiple Parallel Objects (thread or process) access (read-write) to same part data.Wherein, in order to protect the number of serverAccording to area, when operating there is currently an object to certain part of data block, another object, which needs to block, to be waited.
And then withouted waiting for without lock data structure, it can concurrently carry out, thus by reducing obstruction and waiting, to improveThe concurrency and scalability of server.The deadlock that the data structure of locking may result in server in abnormal cases is askedTopic also results in all threads or process blocking in addition, if the dynamics of locking is too big, leads to money when subsequent unlock and unlockSource consumption and context switching consumption, to Priority Inversion occur and lock the phenomenon that escorting.Data structure without lock, is comparedIn the above-mentioned data structure for having lock, it is possible to reduce resource consumption (namely time loss in some sense), and can eliminateCondition competition, obstruction, deadlock, composability deficiency bring potential problems.However, inventor has found under study for action, current nothingThe data structure of lock needs to set privately owned retrieval data for each process or thread, in the processing of data high concurrent, committed memoryIt is larger.
For this purpose, the discovery based on above-mentioned technical problem, inventor proposes that following technical proposals are above-mentioned to solve or improveProblem.It is inventor by practice and careful it should be noted that defect present in the above scheme in the prior artIt is being obtained after research as a result, therefore, the discovery procedure of the above problem and hereinafter the embodiment of the present application institute regarding to the issue aboveThe solution of proposition all should be the contribution that inventor makes the application during innovation and creation, without should understand thatTechnology contents known to those skilled in the art.
Fig. 1 shows the flow diagram provided by the embodiments of the present application without lock shared drive processing method, it should be understood thatIn other embodiments, the sequence without lock shared drive processing method part step of the present embodiment can not with Fig. 1 andThe sequence of following specific embodiments is limitation, such as can be exchanged with each other according to actual needs or part steps thereinIt can be omitted or delete.The detailed step without lock shared drive processing method is described below.
Step S110 configures the memory configurations data of the data service for each data service.
In the present embodiment, by taking live scene as an example, data service may include live video business, live streaming voice service,The multiple business such as order business are broadcast live, corresponding memory configurations data can be configured for different data services.Wherein, describedMemory configurations data may include data block size, data number of blocks needed for the data service and shared memory file roadDiameter.
Step S120, shared drive needed for calculating the data service according to the data block size and data number of blocks are emptyBetween.
Step S130 opens shared memory file according to the shared memory file path, and according to the shared driveThe shared memory file is mapped in shared drive region by preset memory mapping mode, and obtains mapping by spaceMapping return value after the completion.
Illustratively, which can be mmap mode, and mmap can reflect shared memory fileIt is mapped in shared drive region in the address space of corresponding process, realizes in file disk address and process virtual address spaceThe enantiomorphic relationship one by one of one section of virtual address, after realizing such mapping relations, process can be read by the way of pointerThis section of memory of write operation.After completing the shared memory file mapping, a mmap return value can be returned to, as the mappingReturn value.
Step S140 obtains the starting of corresponding mapping area in the shared drive region according to the mapping return valuePointer address and offset record pointer address, and the number is recorded in the head pointer address and offset record pointer addressIn memory according to the corresponding application program of business.
Step S150 is the shared section key after the atom map of shared drive region progress auxiliary dataDomain distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete the data industryBusiness is handled without lock shared drive.
Based on above-mentioned design, the shared memory file is mapped to shared drive in such a way that memory maps by the applicationIn region, and according to the mapping return value after the completion of mapping by the starting of corresponding mapping area in obtained shared drive regionPointer address is recorded in the memory of the corresponding application program of the data service with offset record pointer address, on this basisWith the atomic data structure of atomic update, and index area, Hash array data area, extraction pond are divided by data and is depositedThe method for storing up multiple atom integers of data field, without setting privately owned retrieval data for each process or thread, in dataCommitted memory is smaller when high concurrent processing, so that can provide the data of high concurrent in live broadcast service for high speed development businessRead-write service, and the high performance read-write of high concurrent and storage scheme are provided for memory database.
In a kind of possible embodiment, for step S150, auxiliary data is carried out to the shared drive regionThe mode of atom map may is that
Generate the remark information area in the shared drive region；
The position configuration flag value of first character section in the remark information area；
The position of second byte in the remark information area configures current data capacity；
The position configuration data block size of third byte in the remark information area；
The position configuration data number of blocks of the 4th byte in the remark information area；
The first character is respectively directed to by the int type pointer of four atoms in the memory of the application program respectivelyThe data word locking nub of section, second byte, the third byte and the 4th byte data, to pass through described fourThe int type pointer type of atom controls the first character section, second byte, the third byte and respectivelyFour bytes are read and write without lock.
In detail, the remark information area in the shared drive region of generation be blank area, first in the blank areaA byte, second byte, the configuration mode of third byte and the 4th byte are as follows:
Data number of blocks: s_blockCount=((au64*) m_mem.data ())+2；
Data block size: s_blockSize=((au64*) m_mem.data ())+1；
Mark value: s_flags=(au32*) m_mem.data ()；
Current data capacity: s_cnt=((au32*) m_mem.data ())+1；
Then, the first character section, second described is controlled by the int type pointer type of four atoms respectivelyThe configuration process without lock read-write of byte, the third byte and the 4th byte is as follows:
au32*s_flags；
au32*s_cnt；
au64*s_blockSize；
au64*s_blockCount；
On this basis, further, Fig. 2 is please referred to, in a kind of possible embodiment, step S150 canTo be realized by following sub-step:
Sub-step S151 is the shared drive region distribution index area, and distribution is used for data in the index areaThe multiple VerIdx arrays and check information list of index are stored with the number pair of each data block in the VerIdx arrayThe cryptographic Hash answered, the check information list include the storage byte obtained according to the quantity of Hash bucket and the storage byteExtended field after adding 16 bytes.
Sub-step S152 distributes Hash array data area for the shared drive region, and to the Hash array dataArea carries out atom map, and the metadata of each data block is recorded by data link table, wherein the metadata passes through atom shapingIt is stored, the metadata includes data block length, data block version number, data block identifier, data block cryptographic Hash, data blockDelete the combination of one or more of label and data block reference count.Wherein, above-mentioned a variety of combinations can beTwo or more combination.
Sub-step S153 extracts pond for shared drive region distribution, and the extraction pond is referred to by 64 atom pointersTo next region of memory that can be written into, and the position of a data block is recorded, the corresponding atom shaping in the extraction pondThe identification information of next data block of global data block information and each data block is had recorded in pointer alignment.
Sub-step S154 distributes storage data area for the shared drive region, and the size of the storage data area is equal toThe quantity of the Hash bucket and the product of size.
In this way, by above-mentioned design, by the atomic data structure with atomic update, and being divided into index by dataArea, Hash array data area, extract pond and storage data area multiple atom integers method, without for each processOr thread sets privately owned retrieval data.
On the basis of foregoing description, below with reference to Fig. 3 to based on it is above-mentioned to the data service without lock shared drive atThe application of data structure after reason carries out exemplary description, referring to Fig. 3, the present embodiment mentions after step S150What is supplied can also include the following steps: without lock shared drive processing method
Step S160, when receiving the concurrent data processing request for data service, at the concurrent dataReason request obtains the identification number of pending data.
In detail, when receiving the concurrent data processing request for data service, for different data services, allIt is corresponding with an identification number FLAG relevant to pending data.
Step S160 calculates the corresponding cryptographic Hash of the identification number, and according to the cryptographic Hash to the pending dataExecute corresponding operation.
In the present embodiment, the mode for calculating the corresponding cryptographic Hash of the identification number can be executed by following code:
U32 hash=CncrHsh::HashBytes (key, klen)；
In a kind of possible example, if concurrent data processing request is data write request, this step is firstThen data number of blocks needed for calculating the pending data according to the cryptographic Hash is extracted from the extraction pond of above-mentioned configurationFreed data blocks corresponding with the data number of blocks, and the information of the freed data blocks is written to the Hash array numberAccording in area.Then, the pending data is written to data corresponding with the freed data blocks in the storage data areaQu Zhong, so as to complete data write-in, the VerIdx being finally written to the information of the freed data blocks in the index areaIn array (data query entrance), in order to which user carries out data query.In this way, completing concurrently writing for the pending dataEnter.
For example, if extracting idle data corresponding with the data number of blocks from the extraction pond of above-mentioned configuration is dataThen the information of data block 0, data block 1 and data block 2 is written to the Hash number by block 0, data block 1 and data block 2In group data area, and the pending data is written in the storage data area and data block 0, data block 1 and dataIn the corresponding data field of block 2, finally the information of data block 0, data block 1 and data block 2 is written in the index areaIn VerIdx array.
In alternatively possible example, if concurrent data processing request is data read request, this step is firstNumber and the version number of corresponding data block are first obtained from the VerIdx array of the index area according to the cryptographic Hash, thenThe corresponding Brief block of information of the data block is obtained from Hash array data area according to the number of the data block, it shouldBrief block of information can refer to the block of information for recording the basic summary info of the data block.Next, it is determined that the cryptographic Hash isWhether the length of the identification number of no pending data identical, described as the cryptographic Hash of the Brief block of information is believed with the BriefCease the identification number of block length is identical and the version number of the data block whether version number's phase with the Brief block of informationTogether.If the length of the identification number of cryptographic Hash pending data identical, described as the cryptographic Hash of the Brief block of information withThe length of the identification number of the Brief block of information is identical, and the version of the version number of the data block and the Brief block of informationIt is number identical, then judge whether the identification number of the data block is identical as the identification number of the Brief block of information.If the dataThe identification number of block is identical as the identification number of the Brief block of information, then it is described to from that needs are obtained from the Brief block of informationReason data are simultaneously read away.
In addition, in above-mentioned deterministic process, if the cryptographic Hash it is different from the cryptographic Hash of the Brief block of information orThe length of the identification number of pending data described in the person number different or described from the length of identification number of the Brief block of informationIt is different from the version number of the Brief block of information according to the version number of block, then it is write from the VerIdx array of the index areaThe number of one data block and version number, and return execution judge the cryptographic Hash whether the Hash with the Brief block of informationBe worth the identification number of identical, the described pending data length it is whether identical as the length of identification number of the Brief block of information withAnd the version number of the data block whether the step identical as the version number of the Brief block of information.
In alternatively possible example, if concurrent data processing request is data removal request, this step canQuasi- mark data block corresponding in the VerIdx array of the index area is carried out deletion label according to the cryptographic Hash, andIt is detecting there is currently when the data read request to the quasi- mark data block, if detecting the quasi- mark data blockIt is marked in the presence of deleting, then returns and deleted prompt information.If detecting that there is currently no to the quasi- mark data blockWhen data read request, then by the corresponding Brief block of information of the quasi- mark data block in Hash array data area intoRow delete label, then by the version number of the Brief block of information be labeled as 0, and to the data in the Brief block of information intoRow recycling.In this way, being carried out when actually deleting by quasi- mark data block corresponding in the VerIdx array by the index areaLabel is deleted, in this way when needing to read the quasi- mark data block there are data read request, if detecting the quasi- reference numeralsExist according to block and delete label, then the information returned in the quasi- mark data block has been deleted, and reading data is no longer performed the next stepStep.
Fig. 4 shows provided by the embodiments of the present application for realizing the above-mentioned electronic equipment without lock shared drive processing method100 schematic diagram, the electronic equipment 100 can be for providing main broadcaster's server of service.In the present embodiment, which is setStandby 100 may include storage medium 110, processor 120 and without lock shared drive processing unit 130.
Wherein, processor 120 can be a general central processing unit (Central Processing Unit,CPU), microprocessor, application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC),Or it is one or more for controlling the integrated of the programs execution without lock shared drive processing method of above method embodiment offerCircuit.
Storage medium 110 can be ROM or can store the other kinds of static storage device of static information and instruction,RAM or the other kinds of dynamic memory that can store information and instruction, are also possible to the read-only storage of electric erazable programmableDevice (Electrically Erasable Programmabler-Only Memory, EEPROM), CD-ROM(Compactdisc Read-Only Memory, CD-ROM) or other optical disc storages, optical disc storage (including compression optical disc, swashOptical disc, optical disc, Digital Versatile Disc, Blu-ray Disc etc.), magnetic disk storage medium or other magnetic storage apparatus or can useIn carry or storage have instruction or data structure form desired program code and can by computer access it is any itsHis medium, but not limited to this.Storage medium 110, which can be, to be individually present, and is connected by communication bus with processor 120.It depositsStorage media 110 can also be integrated with processor.Wherein, storage medium 110, which is used to store, executes answering for application schemeWith program code, such as without lock shared drive processing unit 130 shown in Fig. 4, and execution is controlled by processor 120.PlaceReason device 120 is for executing the application code stored in storage medium 110, such as without lock shared drive processing unit 130, withExecute above method embodiment without lock shared drive processing method.
The application can be drawn according to above method embodiment to without the lock progress of shared drive processing unit 130 functional modulePoint, for example, each functional module of each function division can be corresponded to, two or more functions can also be integrated in oneIn a processing module.Above-mentioned integrated module both can take the form of hardware realization, can also be using software function moduleForm is realized.It should be noted that be schematical, only a kind of logical function partition to the division of module in the application,There may be another division manner in actual implementation.For example, the functional module each using corresponding each function division the case whereUnder, shown in Fig. 4 is a kind of schematic device without lock shared drive processing unit 130.In detail, no lock shared drive processingDevice 130 may include data configuration module 131, computing module 132, memory mapping block 133, address logging modle 134 andDistribution module 135 separately below explains the function of each functional module without lock shared drive processing unit 130 in detailIt states.
Data configuration module 131 configures the memory configurations data of the data service, institute for being directed to each data serviceStating memory configurations data includes required data block size, data number of blocks and shared memory file path.It is appreciated that shouldData configuration module 131 can be used for executing above-mentioned steps S110, and the detailed implementation about the data configuration module 131 canWith referring to above-mentioned to the related content of step S110.
Computing module 132, for being total to needed for calculating the data service according to the data block size and data number of blocksEnjoy memory headroom.It is appreciated that the computing module 132 can be used for executing above-mentioned steps S120, about the computing module 132Detailed implementation is referred to above-mentioned to the related content of step S120.
Memory mapping block 133, for opening shared memory file according to the shared memory file path, and according to instituteShared memory space is stated, the shared memory file is mapped in shared drive region by preset memory mapping mode,And obtain the mapping return value after the completion of mapping.It is appreciated that the memory mapping block 133 can be used for executing above-mentioned stepsS130, the detailed implementation about the memory mapping block 133 are referred to above-mentioned to the related content of step S130.
Address logging modle 134, for obtaining corresponding in the shared drive region reflect according to the mapping return valueThe head pointer address and offset record pointer address in region are penetrated, and by the head pointer address and deviates record pointer addressIt is recorded in the memory of the corresponding application program of the data service.It is appreciated that the address logging modle 134 can be used for holdingRow above-mentioned steps S140, the detailed implementation about the address logging modle 134 are referred to above-mentioned related to step S140Content.
Distribution module 135 is described shared after the atom map to shared drive region progress auxiliary dataRegion of memory distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete to be somebody's turn to doData service is handled without lock shared drive.It is appreciated that the distribution module 135 can be used for executing above-mentioned steps S150, closeIt is referred in the detailed implementation of the distribution module 135 above-mentioned to the related content of step S150.
Due to it is provided by the embodiments of the present application without lock shared drive processing unit 130 be it is above-mentioned without lock shared drive processing sideAnother way of realization of method, and can be used for executing provided by above-described embodiment without lock without lock shared drive processing unit 130Shared drive processing method, therefore it can be obtained technical effect can refer to above method embodiment, details are not described herein.
Further, based on the same inventive concept, the embodiment of the present application also provides a kind of computer readable storage medium,It is stored with computer program on the computer readable storage medium, which executes above-mentioned no lock when being run by processorThe step of shared drive processing method.
Specifically, which can be general storage medium, such as mobile disk, hard disk, on the storage mediumComputer program when being run, be able to carry out above-mentioned without lock shared drive processing method.
The embodiment of the present application be referring to according to the method for the embodiment of the present application, equipment (electronic equipment 100 of such as Fig. 4) andThe flowchart and/or the block diagram of computer program product describes.It should be understood that flow chart can be realized by computer program instructionsAnd/or the knot of the process and/or box in each flow and/or block and flowchart and/or the block diagram in block diagramIt closes.These computer program instructions be can provide to general purpose computer, special purpose computer, Embedded Processor or other programmable numbersAccording to the processor of processing equipment to generate a machine, so that passing through the processing of computer or other programmable data processing devicesThe instruction that device executes generates for realizing in one box of one or more flows of the flowchart and/or block diagram or multiple sidesThe device for the function of being specified in frame.
Although the application is described in conjunction with each embodiment herein, however, implementing the application claimedIn the process, those skilled in the art are by checking the attached drawing, disclosure and the appended claims, it will be appreciated that and it is realOther variations of the existing open embodiment.In the claims, one word of " comprising " is not excluded for other components or step,"a" or "an" is not excluded for multiple situations.Single processor or other units may be implemented to enumerate in claim severalItem function.Mutually different has been recited in mutually different dependent certain measures, it is not intended that these measures cannot group closeTo generate good effect.
The above, the only various embodiments of the application, but the protection scope of the application is not limited thereto, it is anyThose familiar with the art within the technical scope of the present application, can easily think of the change or the replacement, and should all containLid is within the scope of protection of this application.Therefore, the protection scope of the application shall be subject to the protection scope of the claim.