Specific embodiment
Retouching in detail to one or more embodiment of the invention is hereafter provided together with the attached drawing for illustrating the principle of the inventionIt states.The present invention is described with reference to such embodiment, but the present invention is not limited to any embodiments.The scope of the present invention is only by rightClaim limits, and the present invention covers many replacements, modification and equivalent.Illustrate in the following description many details withJust it provides a thorough understanding of the present invention.These details are provided for exemplary purposes, and without in these detailsSome or all details can also realize the present invention according to claims.
An aspect of of the present present invention provides a kind of cloud disk video coding and storage method.Fig. 1 is according to embodiments of the present inventionCloud disk video coding and storage method flow diagram.
Video stream data file is divided into three ranks by the present invention according to volume.When the volume of video stream data is less than firstDuring threshold value, submitted it as frequency file is neglected to monitoring system cloud disk database purchase；When the volume of video stream data is more thanDuring second threshold, then file server cluster is stored in as super large video file；When the volume of video stream data is in firstIt when between second threshold, is then split according to predefined strategy, monitoring system cloud disk number is stored in a manner of video blockAccording in library.The following specifically describes the step of method：
Video storage request is received, the video storage request carries monitoring video flow to be stored；To ensure that data are depositedThe safety of storage carries out Authority Verification after video storage request is received, and is judged whether according to the source of video storage requestAllow to store data in corresponding storage region.When permission is verified, then continue subsequent step；Work as Authority VerificationIt is obstructed out-of-date, then terminate data Stored Procedure.
Authority Verification after the entrained monitoring video flow to be stored of video storage request is added to caching, is pressed by rearStorage according to monitoring video flow to be stored in the sequential processes caching of first in first out is asked.
Obtain the volume of the monitoring video flow to be stored；When the volume of the monitoring video flow to be stored is less than the first thresholdDuring value, the monitoring video flow to be stored is stored into monitoring system cloud disk database the area for memory block data directoryDomain.
When the volume of the monitoring video flow to be stored is more than second threshold, the monitoring video flow to be stored is storedTo file server cluster, and the first configuration information is stored to the monitoring system cloud disk database, described first and matches confidenceBreath includes the first URL, and the first URL is URL of the monitoring video flow to be stored in the file server cluster；
When super large video file need to be read, it is obtained from the block data directory of monitoring system cloud disk database firstURL further according to the URL, directly reads the video file from the file directory of file server cluster.In this way, subsequently graspingIn work, without being split combination to super large video file.
When the volume of the monitoring video flow to be stored is more than the first threshold and the monitoring video flow to be storedWhen volume is less than the second threshold, according to predefined strategy, the monitoring video flow to be stored is divided into multiple video blocksAfterwards, each video block is stored to monitoring system cloud disk database respectively, and according to each video block described to be storedPutting in order in monitoring video flow stores the second configuration information of each video block to the monitoring system cloud disk numberAccording to library, second configuration information includes the 2nd URL, and the 2nd URL is the video block in the monitoring system cloud disk dataURL in library.
When big video file need to be read, it is every that it is obtained from the block data directory of monitoring system cloud disk database firstThe URL of a video block obtains each video block from the video block table of monitoring system cloud disk database in order further according to URL,It is combined into big video file.The configuration information of big video file includes its file name, video format, file description, video blockSequence and file URL.
Since the issued transaction of storage operation cannot simply reliance server in itself, so the mechanism of caching need to be designed, be usedIn the video file catalogue and data information of storage different scenes.I.e. this method further includes：
Interim URL and first-class address caching are generated, and the interim URL paths are delayed added to the first-class addressIt deposits；When the volume of the monitoring video flow to be stored is greater than or equal to the first threshold, by the monitor video to be storedThe stream write-in interim URL storages；When the interim URL is written not successfully in the monitoring video flow to be stored, arbitration is performedMechanism is discussed in detail below.After execution during continuous specific storing step, the prison is directly obtained from first-class address cachingThe URL of video flowing is controlled, and reads the monitoring video flow from interim URL on this basis.
When the monitoring video flow to be stored is stored not successfully to the monitoring system cloud disk database, second is generatedAddress caching；Judge whether the monitoring video flow to be stored is successfully stored in the file server cluster；If it is, it obtainsThe first URL is taken, and the first URL is added to second address caching；If it is not, then perform arbitration mechanism；
When monitoring video flow to be stored is stored not successfully in file server cluster, then follow-up storage step is no longer performedSuddenly, after the monitoring video flow to be stored in performing arbitration mechanism and removing caching, terminate Stored Procedure.At this point, server setGroup is restored to the state before the storing step for performing the monitoring video flow to be stored.
Judge whether first configuration information is successfully stored to the monitoring system cloud disk database；If it is, it deletesExcept the monitoring video flow to be stored stored in the interim URL；If it is not, then perform arbitration mechanism；When the first configuration informationWhen success stores, which is successfully stored in server cluster.At this point, it can delete corresponding interimURL。
When the first configuration information is stored not successfully in monitoring system cloud disk database, perform arbitration mechanism and remove cachingIn monitoring video flow to be stored after, terminate Stored Procedure.
This method further includes, and generates third stream address caching；It is described to judge whether each video block is successfully stored inMonitoring system cloud disk database；If it is, the 2nd URL of each video block is obtained, and according to each video block2nd URL of each video block is added to third stream address by putting in order in the monitoring video flow to be storedCaching；If it is not, then perform arbitration mechanism；
It, can be directly from third stream address caching after each video block is successfully stored in monitoring system cloud disk databaseThe 2nd URL of each video block is obtained, then is stored in sequence in the block data directory of monitoring system cloud disk database.When certainOne video block or certain several video block are stored not successfully in monitoring system cloud disk database, then no longer perform follow-up storage stepSuddenly, after the monitoring video flow to be stored in performing arbitration mechanism and removing caching, terminate Stored Procedure.
Judge whether the second configuration information of each video block is successfully stored to the monitoring system cloud disk database；IfIt is then to delete corresponding interim URL；If it is not, then perform arbitration mechanism.
When the second configuration information also successfully stores, which is successfully stored in server clusterIn.At this point, the monitoring video flow to be stored in interim URL can be deleted.In addition, when the second configuration information is stored in not successfullyDuring monitoring system cloud disk database, then after performing arbitration mechanism and removing the monitoring video flow to be stored in caching, terminate storageFlow.
For above-mentioned arbitration mechanism, its specific steps is described more detail below：
1：Check whether current video file is to neglect frequency file less than first threshold；If it is, perform step 2；If it is not, then perform step 3；
2：The rollback of the monitoring system cloud disk database is performed, terminates arbitration mechanism；It is stored when neglecting frequency fileDuring failure, storing step is cancelled, deletes the partial data neglected and stored in frequency file in the database.
3：It checks in first-class address caching with the presence or absence of the interim URL for needing to clear up；If it is, according to described firstThe interim URL in address caching is flowed, deletes the interim URL；
When interim URL failures are written in big video file, the interim URL for storing big video file is deleted, to preventInterim URL occupies system memory space.
Wherein, when affairs towards object be not to neglect frequency file when, the arbitration mechanism further includes：
4：It checks in second address caching with the presence or absence of the first URL for needing to clear up；If it is, according to described secondThe first URL in address caching is flowed, deletes the monitoring video flow to be stored stored in the file server cluster；
5：It checks in third stream address caching with the presence or absence of the 2nd URL for needing to clear up；If it is, according to the thirdThe 2nd URL in address caching is flowed, deletes the video block stored in the monitoring system cloud disk database.
It is right in the URL deletion respective memory regions in first to third stream address caching respectively by above-mentioned stepsAfter the data answered, you can the state that server is made to be restored to before execution method so that ensure video stream data storage operationIntegrality.
It, respectively will be every after the monitoring video flow to be stored is divided at least one video block according to predefined strategyA video block is stored to monitoring system cloud disk database, and the present invention further comprises：
The video file of acquisition is addressed to obtain the corresponding storage cell of video block.The acquisition video block corresponds toStorage cell, including：
The hashed value of video block ID is calculated by hash function, to obtain the corresponding pseudorandom values of video block ID；It will obtainThe corresponding pseudorandom values of video block ID and mask step-by-step phase and operation, identified with obtaining the corresponding storage cell of video block.
The corresponding storage cell of video block that addressing obtains finally ensures the sum of the object stored in each storage cellAccording to amount approaches uniformity, real-time storage efficiency and system performance are improved.
The corresponding storage cell of the video block of acquisition is addressed to obtain the corresponding monitoring cloud disk node of storage cell.Multiple monitoring cloud disk nodes are collectively responsible for block all in the corresponding storage cell of storage and maintenance.
In the present embodiment, three corresponding monitoring cloud disk nodes are configured in each storage cell, realize user's storage fileTriple backups, while the monitoring cloud disk node configuration solid state disk realizes the fast reading and writing operation of data.
After the corresponding monitoring cloud disk node of storage cell is got, by the storage list of the video file write-in acquisition of acquisitionThe corresponding monitoring cloud disk node in position.The video file write-in monitoring cloud disk node, including：
Step 1：The corresponding first monitoring cloud disk node of storage cell that the video file write-in of acquisition is obtained.
Step 2：By the video file object of the first monitoring cloud disk node write-in be respectively written into the second monitoring cloud disk node andThird monitors cloud disk node.
Step 3：Video file, which is sent, to video acquisition terminal successfully stores information.
After three monitoring cloud disk nodes all complete write operation, control and successfully deposited to video acquisition terminal transmission storage fileInformation is stored up, finally realizes three of video acquisition terminal storage file in three monitoring cloud disk nodes of cloud storage system serverWeight secure storage.
Wherein after video storage request is received, it will be regarded according to the monitor video stream information carried in video storage requestThe storage of frequency flow data is further included to monitoring cloud disk node：Regard most recently used data as dsc data, it will nearest not used numberAccording to cold data is regarded as, dsc data is stored in high-speed hardware equipment, and cold data is stored in inactive memory node.OftenA new write-in data are regarded as dsc data, can be directly stored in monitoring cloud disk node, can directly to monitor next timeTarget data is found in cloud disk node to prepare.It can be unified by that will be based on from the video storage request of different storage applicationsTransformation standard, be converted into standard request so that the video storage request for coming from different storage applications can be appliedThe system identification that the method for the present invention is cached, and then improve the versatility of this caching method.
Preferably embodiment, the invention also includes：Update cache management information.Calculate monitoring cloud disk nodeDeposit data amount and the ratio of the capacity of monitoring cloud disk node.Judge whether ratio is greater than or equal to threshold value, if it is, utilizingLru algorithm removes the cold data monitored in cloud disk node according to cache management information, successively until ratio is less than threshold value.ForData are not lost, can backed up to this compared with cold data before it will be deleted from monitoring cloud disk node successively compared with cold dataInactive memory node.As preferred embodiment, according to cache management information by the cold data monitored in cloud disk node according toSecondary removal specifically includes：Cold data is stored successively to inactive memory node, and delete and stored to inactive memory nodeAnd be stored in monitoring cloud disk node in cold data.
When the storing process of monitoring system cloud disk database encounters mistake and terminates, the present invention is further by with lower sectionMethod confirms abort situation and then correctly retries write operation and perform data recovery.If monitoring system cloud disk database includes at least mA memory node and q error correction node.Storing process then based on BMSR codings is as follows：
Step 1, memory node determines that the first storage is asked, and monitor video to be stored is carried in the first storage requestStream and the first update mark, wherein first update mark be used for show video acquisition terminal storage by the prison to be storedThe update mark of video block that control video flowing is updated, the data volume of the monitoring video flow to be stored are less than a BMSR and compileThe data volume of code character；
Step 2, memory node response the first storage request, obtains the second update mark, second update markFor showing the update mark of the video video block of itself storage.
Step 3, according to corresponding as a result, to the prison to be stored between first update mark and the second update markControl video flowing is handled, and further comprise：
Step 31, memory node judges whether the first update mark is identical with the second update mark, if identical, firstUpdate mark is corresponding with the second update mark, if it is not the same, then the first update mark and the second update mark be not corresponding.SecondUpdate mark is newest update mark, if the second update mark illustrates earlier than the first update mark in certain primary storageNode do not update completion and video acquisition terminal to be completed for update, thus video acquisition terminal has updated the update of itselfLabel, either the second update mark is later or earlier than the first update mark, as long as the two does not correspond to, memory node all can be to depositingStorage system or video acquisition terminal send out error information.
Step 32, memory node is handled the monitoring video flow to be stored according to first judgment value.At thisIn the further embodiment of invention, step 32 further includes：
Step 321, if first judgment value shows that first update mark and second update mark be not rightShould, the first update response is sent, update failure result is carried in the first update response；
Step 322, the monitoring video flow to be stored is abandoned.
Step 323, if first judgment value shows that first update mark is corresponding with second update mark,Update video piece of collection { the Δ P of error correction increment is determined according to monitoring video flow to be stored and the video block1, Δ P2..., ΔPg..., Δ Pq, wherein, Δ PgFor the error correction increment of g-th of error correction node, 1≤g≤q, q are opposite with a BMSR codingThe quantity of verification video block answered；
Step 324, by the error correction increment Delta P of g-th of error correction nodegIt is carried in the second storage request, by described secondStorage request is sent to g-th of error correction node, and second storage asks to trigger g-th of error correction node according toError correction increment Delta PgThe verification data of itself storage is updated, and updates the version of g-th of error correction node itself storageThis vector.
It is described that the monitoring video flow to be stored is handled according to the first judgment value, it further includes：If described firstJudgment value shows that first update mark is corresponding with second update mark, using the monitoring video flow to be stored to instituteVideo block is stated to be updated；After the completion of being updated to the video block, second update mark is updated.
After the completion of being updated to the video block, the first update response is sent, update is carried in the first update responseSuccessful result, it is described to be updated successfully result for triggering video acquisition terminal update first update mark.Furthermore, it is possible to it setsTimer is put, before the deadline if be not updated successfully, video acquisition terminal can re-initiate storage request.
Memory node safeguards a update mark to each initial video block, and when there is write operation, memory node first canJudge whether the update mark of data received and the update mark of itself storage correspond to, to it is corresponding when memory node just can be rightData in storage request are updated, so as to avoid terminating when storing process encounters mistake.
In BMSR codings, it is assumed that the volume of video file is M, which is divided into multiple video blocks,A volume is combined into per k video block, the encoding operation of BMSR codings is mutual indepedent in each volume.In order to maximize dataReliability, n block after coding are stored in n different monitoring cloud disk nodes (including k video block and n-k encoding block)In.
In monitoring system cloud disk database operational process, when r node failure, need to find other r replacement sectionPut the data to restore failure.The restoration model that the present invention is represented using (n, k, r).Source node completes data segmentation, codingWith distributed tasks.X1, X2..., Xn-rThe memory node of expression activity, Y1, Y2..., YrRepresent r substitute node.Destination nodeArbitrary k node is read to complete the reconstruct of primary data.
Specifically, operational process includes 3 steps, initialization, data recovery and download.During initialization, at the beginning of volume is MBeginning video file is divided into multiple video blocks, and a volume is formed per k video block.Assuming that video file M is divided into k videoBlock.K video block generates n-k encoding block by coding.Assuming that k video block is expressed as (d1, d2..., dk), each blockVolume is M/k, and k video block generates n-k encoding block (p by coding1, p2..., pn-k)。
Wherein, each encoding block pi(1≤i≤n-k) is all by k video block coding generation, αI, jIt is piFrom j-thVideo block djCode coefficient.Finally, after the completion of coding, n block (k video block and n-k encoding block) is respectively stored into nMemory node (X1, X2..., Xn), the data volume of each node storage is M/k.That is encoding block piRestored data PiFor
Wherein 1≤i≤n-r
As r memory node Xn-r+1, Xn-r..., XnAfter failure, in order to ensure the original redundancy of data, r is reselectedA substitute node completes the recovery Y of data1, Y2..., Yr.In recuperation, in r substitute node, a main section can be selectedPoint Ymain.The node connects owner node and remaining substitute node simultaneously.Specifically, including：
First, k owner node is selected from the memory node of activity, and r are selected from other enabled nodesSubstitute node；A host node Y is elected from r substitute node simultaneouslymain.Secondly, host node YmainFrom k supplierIt reads k video block and passes through the block that the k video block restores r failure.Finally, YmainCorresponding recovery video block is storedIt is separately sent in corresponding r-1 substitute node in local, while by remaining r-1 recovery video block.Remaining substitute nodeThe data received are stored in local, complete recuperation.
It restores after completing, initial n-r active node X1, X2..., Xn-rWith r substitute node Y1, Y2..., YrCompositionNew n memory node combination provides video storage service, and the access request of data can be by wherein arbitrary k node come weightBuild primary data.
For substitute node after selected, it is host node to select with the substitute node of other substitute node topology distance minimums.It is falseIf active node XiWith supplier YjBetween topology distance be expressed as distanceI, j, then k is selected from n-r active nodeTotal shortest node of topology distance is as supplier in a to r substitute node.Active node XiWith supplier YjBetween total open upFlutterring distance is：
In the selection process, n-r total topology distances are arranged by the way of ascending order, k have minimum before finally takingThe node of total topology distance is as supplier X1, X2..., Xk。
The selection of host node needs simultaneously to consider and topology distance between the two.Assuming that substitute node YiTo YjTopologyDistance is distance 'I, j, then substitute node YiTopology distance summation to other r-1 substitute node is
The node with the minimum total topology distance of other substitute nodes and supplier is selected as host node Ymain。
distancemain=min1≤i≤rdistance’i)
After selected substitute node and supplier, between supplier and host node, real between host node and other suppliersExisting data transmission.Between supplier and host node, k all supplier X1, X2..., XkWith host node YmainConnection, andData are transmitted to it.Each supplier transmits the video block that a volume is M/k to host node.Host node is receiving supplierData after pass through coding with decoding operate complete r trouble block recovery operation.
Between host node and other suppliers, host node, first will be corresponding after the construction work for completing r trouble blockBlock be stored in local, while remaining r-1 block is separately sent in corresponding substitute node.
Other substitute nodes are stored in local after the data for receiving substitute node.Data are completed in all substitute nodes to connectAfter receiving and storing operation, recuperation terminates.
Host node YmainIn specific calculating process, the data according to failure are the specific feelings of video block or encoding blockCondition, correspondence do different calculating operations.Assuming that r invalid video video block number in the block and encoding block number are respectively r1Withr2, r=r1+r2。r1The video block of a failure is expressed as d1, d2..., dr1, r2The encoding block of a failure is expressed as p1, p2..., pr2,The data that substitute node is read from k supplier are b=(b1, b2..., bk), then video block diWith encoding block pjCalculating such asUnder：
1) the k video block vector b=(b read1, b2..., bk) directly it can restore this in vain by decoding operateVideo block.Assuming that encoder matrix is P, then cataloged procedure can be expressed as P*D=C, wherein, D is video block；C for video block withThe corresponding matrix of encoding block.If the k video block extracted out from C is b=(b1, b2..., bk), corresponding matrix is matrixP ' then can obtain matrix D=P ' by matrix operation-1*b.And all video blocks, therefore, d are included in matrix DiRecoveryOperation can be expressed as：
di=(λ1, λ2..., λk)T*(b1, b2..., bk)
Wherein, λiIt is P '-1Middle diCorresponding vector.
2) recovery operation of encoding block is the encoding operation that a corresponding position is remake on the basis of decoded.It is specific andSpeech, if the k video block extracted out from C is vector b=(b1, b2..., bk), corresponding matrix be matrix P ', decoding operateIt can be expressed as D=P '-1*b.And pjCalculating process can be expressed as pj=βj* D, wherein vector βjIt is pjCorresponding vector,Then pjCalculating process can be expressed as：
pj=(β1, β2... βk)*P’-1*(b1, b2..., bk)
Complete all r block (r1A video block and r2A encoding block) recovery after, video that host node restores theseBlock is sent in corresponding substitute node.It reads a data and r video block is built by different parameters simultaneously, meanwhile, it replacesFor the calculating in node using multithreading, the degree of parallelism of calculating process is increased.
In conclusion the present invention proposes a kind of cloud disk video coding and storage method, can meet to magnanimity small video textThe storage demand of part, moreover it is possible to ensure the treatment effeciency to super large video file, improve the performance of intelligent transportation data storage.
It obviously, can be with general it should be appreciated by those skilled in the art each module or each step of, the above-mentioned present inventionComputing system realize that they can concentrate in single computing system or be distributed in multiple computing systems and be formedNetwork on, optionally, they can be realized with the program code that computing system can perform, it is thus possible to which they are storedIt is performed within the storage system by computing system.It to be combined in this way, the present invention is not limited to any specific hardware and softwares.
It should be understood that the above-mentioned specific embodiment of the present invention is used only for exemplary illustration or explains the present invention'sPrinciple, without being construed as limiting the invention.Therefore, that is done without departing from the spirit and scope of the present invention is anyModification, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.In addition, appended claims purport of the present inventionCovering the whole variations fallen into scope and boundary or this range and the equivalent form on boundary and repairingChange example.