Detailed description of the invention
Hereafter provide retouching in detail one or more embodiment of the present invention together with the accompanying drawing of the diagram principle of the inventionState.Describe the present invention in conjunction with such embodiment, but the invention is not restricted to any embodiment.The scope of the present invention is only by rightClaim limits, and the present invention contains many replacements, amendment and equivalent.Illustrate in the following description many details withThorough understanding of the present invention is just provided.These details are provided for exemplary purposes, and without in these detailsSome or all details can also realize the present invention according to claims.
An aspect of of the present present invention provides the method for a kind of Bluetooth control display content synchronization.Fig. 1 is real according to the present inventionExecute the method flow diagram of a kind of Bluetooth control display content synchronization of example.
The display frame of input, first during picture compression, is divided into block with fixed size, based on frame sequence by the present inventionThe in-frame correlation of row and frame-to-frame correlation compressed encoding；Coded method based on content-adaptive, i.e. utilizes bitmap region specialLevy and picture is categorized as bitmap region and character area；Bitmap region and character area are respectively placed in bitmap layer and character layer,Bitmap region uses picture coding to process, and character area carries out color description and quantization and utilizes light intensity and saturatedThe passage dependency coding of degree.In picture synchronization transmitting procedure, it is divided into multiple block according to image content and carries out independenceCoding and transmission；According to the characteristic of above compression method, utilize the delay improved to reduce method and transmit picture.
The present invention utilizes many AR equipment collaboration picture synchronization display platform, same in AR terminal disposition based on remote AgentStep display platform.Agent maintain a virtual synchronous display platform for resolving, typesetting and drafting display picture, draw outThe picture image come is transferred to AR terminal in encoded.AR terminal receives and decoded bit stream, is then demonstrated by picture imageCome.On the basis of AR terminal simultaneous display platform based on Agent, utilize display picture playing progress rate synchronization mechanism, it is achievedThe seamless connection of playing progress rate.
Input picture image first with 64 × 64 size piecemeal.Then by carrying out difference inspection between consecutive frameSurveying, find out the most constant block of content and position, the present invention is referred to as time shift block.Encoded by combination entropy and compress each time shiftThe type of block, but in remaining block, perform motion detection based on block boundary alignment find out that content is constant to be subjected to displacementRegion.For each moving mass, its type and motion vector are encoded.In the constant content detected between consecutive frameAfterwards, it is bitmap region and character area by remaining block sort, is then respectively adopted different methods and is compressed.
The present invention is when classification, and for a block i, making XP (i) and YP (i) represent respectively will be current under current gradient factorTerritorial classification is bitmap region distortion rate when encoding and bit rate, XT (i) and YT (i) represent current gradient system respectivelySeveral lower current region is categorized as distortion rate when character area encodes and bit rate.If meeting following condition:
XT (i)≤XP (i), and YT (i) ＜ YP (i)
Then current region is classified as character area.
If meeting following condition:
XP (i) ＜ XT (i), and YP (i)≤YT (i)
Then current region is classified as bitmap region.
Otherwise, current region will be classified with Lagrange rate-distortion optimization algorithm.
After having classified, first carry out the quantization of Pixel-level, four kinds of the highest for the frequency of occurrences in block colors are determinedBased on color.The quantization function used is expressed as follows:
Wherein x is intended to the color quantified, BCk, it is 4 basic colors, k=0 ..., 3, Δ1And Δ2It is by two predetermined amountQuantizer input quantization increment.
If | x-BCk|≥Δ1, then this color x is referred to as non-basic color.Three Color Channels of word content are eachIndependently quantify.After quantifying, each block S (i, j) (0≤i≤w-l, 0≤j≤i-1, the w of the picture image of inputWidth and the height of block respectively with h) each Color Channel can with a gray matrix I (i, j) and basic color, non-Basis color describes.I (i, j) is defined as follows,
If S (i, j)-BCk ＜ Δ 1, then I (i, j)=k, otherwise I (i, j)=4
When compressing word content, if text pixel accounts for maximum ratio in certain block, then method is preferentially classified asCharacter area is compressed.Basis color compression text pixel is utilized to utilize non-basic color compress bitmap pixel can keep compressionHigh efficiency during text pixel.
Then (i, (i j) represents word j) to be converted into two dimension associated diagram P by gray matrix I to use following methodThe Two-Dimensional Correlativity of content.
So two dimension associated diagram is done one-dimensional scanning and just can utilize the Two-Dimensional Correlativity of word content.When P (i, j)=0Time, further code index value I (i, j).
For the every a line in block to be encoded, first attempt to full line pattern its coupling coding.Without full linePattern can be mated, then this row is decomposed into several pixel groups, and each pixel groups is carried out pattern match.Pixel two dimensionAssociated diagram numerical value represents the pattern of pixel domain.By every L pixel two dimension associated diagram numerical value one pixel groups two-dimensional symensional symbol of composition,Then according to the probability distribution of pixel groups pattern, it is carried out combination entropy coding.Pixel groups two-dimensional symensional symbol can be further converted intoSimple symbol 0 and 1, as follows:
Wherein k=0 ..., w/L-1, j=0 ..., h-1；M takes [0, L-1]
If (i, pixel groups two-dimensional symensional symbol is encoded G by j)=0 item further.
For bitmap layer, generate a most independent jpeg stream the most respectively, corresponding in bitmap region positionHold and fill with the meansigma methods of neighbouring encoding block color.The distribution of color of all bitmap region on statistics piece image, then willAll colours is quantified as 512 colors.It is mapped as quantifying increment by the mass parameter that bitmap region quantifies.Make QS1..., QSnRepresentEach quantization increment in n basis colo(u)r specification.When the bitmap region in image being quantified with quantization increment, will make mean squareThe QS that difference and bit rate sum minimizekIt is defined as optimum quantization increment size.
For reducing the delay of picture transmission, first the picture image grabbed is divided into multiple block by the present invention, eachBlock independently encodes.Send into transmission queue after the bit result packing that coding produces, be then transferred to by UDP channelAR terminal.AR terminal receives these bags, decodes each block and is shown on screen.When transmitting queue full, compileCode control module stops screen coding process, until having enough space next one coded frame in transmission queue.Then profitReduce unnecessary re-transmission with the transmission method improved, thus reduce transmission delay.
The most still with fixed size, picture image is divided into block to encode, and display screen enters in units of frameRow decoding.(x, y) (0 < x < W/w, 0 < y < H/h) wherein x, y represents the position of a block in image to construct a frame renewal table UPut.W and H is width and the height of picture image respectively, w and h is width and the height of a block in image respectively.For n-thFrame, with S, (x, y n) represent and are wherein positioned at that (x, the content of block y) is the most identical with the block of former frame correspondence position (instantMove block).If S (x, y, n)=1, then content is identical.When receiving n-th frame data, if (x, y, n)=0 and n is more than S(x, y), then by being positioned in n-th frame decoded result, (x, block y) copies to the relief area of present frame to U, and (x y) is then to update Un。
For declining low transmission delay at picture synchronization transmission environment, the present invention uses following methods, if in a certain frameThe data of one non-time shift block have been received and state updates, then before in frame the Data Identification of the block of same position bePhase, no longer retransmit these data.Specifically, construct the state table of each data block in host side, for each packet, obtainData block set in bag, is set to i by the index of each data block in current collection, if the type of current block is not time shiftThe numbering of block and place frame is more than i state A in state tablei, then using the numbering of place frame as state Ai.Whenever receptionTo the more new information of AR terminal, state table all can be updated.
Receive terminal and all can comprise packet in this time period to sending one state bag of terminal feedback every predetermined periodThe relevant information received.If sending terminal do not receive state bag in Preset Time, then judge that current network is as overload；If present packet loss is less than fixed threshold, then judge that current network is as idle condition；Other situations are judged to normal condition.
If it is determined that network is overload, attenuation quotient λ is used to reduce current bit rate parameter Bx:
Bx=max (λ × Bx, Bmin}(0<λ<1)
If it is determined that network is underload, growth factor θ is used to increase current bit rate parameter Bx:
Bx=min{Bx+ θ, Bmax}(θ<<Bx)
When transmitting terminal is if it is determined that network is to approach overload under normal condition, use attenuation quotient δ adjust by a small margin currently thanSpecial rate parameter Bx:
Bx=max (Bx-δ, Bmin}(δ<<Bx)
Wherein BmaxAnd BminRepresent minima and the maximum of bit-rate parameters respectively.
If encoder bit rate output reaches the minima set, network is still in overload, then below employingMethod reduces the data volume being transported to network further: work as BxEqual to BminAnd occurrence number is when reaching p time, by complete to I frame and P framePortion abandons, and only sends B frame, controls less data volume output.Still transshipping when continuous q (q ＞ p) individual periodic network, sendingEnd continues to abandon part B frame；Work as BxMore than BminTime, send terminal and send all types of frame of video.Occurring that packet loss reducesDuring to normal condition, first transmitting terminal will I frame and P frame all abandon, and continues to judge BxEqual to BminWhether the number of times occurred reachesTo p time, if less than p time, just sending all frames, it is ensured that video Real Time Streaming.
In terms of playing progress rate linking based on Agent, when an AR equipment of a certification user is connected to AgentTime, the TERMINATION MANAGER of Agent identifies this user and creates a new virtual display example for this equipment, performing pictureThe logic of face engine, and safeguard the state in the broadcasting of all of display picture；After equipment is successfully connected, Agent notifies phaseWith user every other AR equipment can and this equipment between play out progress and keep consistent.
The linking of picture playing progress rate is reality by the corresponding relation between the virtual display example of change and AR terminal unitExisting.When AR equipment B send a synchronization request with target AR device A to Agent time, Agent by its playing progress rate withTarget device is mapped, and the picture image drawn out in equipment B is sent to device A, and user's input of device A simultaneously is also sent outGive equipment B.All of display picture broadcast state is all by virtual display Example maintenance, and this mechanism can keep all of aobviousShow picture broadcast state；
Described Agent carries out picture playing progress rate linking, under migration model, equipment B's according to pattern set in advancePlaying progress rate is connected device A, and the playing progress rate of device A will be released.In a shared mode, the playing progress rate of equipment BDevice A can be connected, meanwhile, create a new broadcasting example having equal state, the broadcasting that equipment B is original for equipment BState in progress, is used for building this new example.Under exchange model, the playing progress rate of equipment B and the broadcasting of device A are enteredDegree is exchanged with each other.
In an additional preferred embodiment, for reducing the impact of the pressure of device A and B direct link, the AR of transmitting terminal setsStandby A carries out Bluetooth transmission with the AR equipment B of receiving terminal by forwarding unit C.Forwarding unit C can include multiple equipment C1…Cn.FirstFirst device A sends handshake and receives data to forwarding unit C, forwarding unit C and calculate number between device A and forwarding unit CPacket loss and the transmission performance parameter of transfer rate composition according to transmission link；Equipment C sends handshake to equipment B, equipment BReturn data calculates packet loss and the biography of data transmission link between forwarding unit C and equipment B to forwarding unit C, forwarding unit CThe transmission performance parameter of defeated speed composition.Forwarding unit C by between himself calculating speed and device A and with equipment B itBetween the transmission performance parameter device A postbacking sending end sent along.
Forwarding unit C similar for this parameter, according to the transmission performance parameter of alternative forwarding unit C, is divided into one group by device A,As forwarding unit group.For same forwarding unit C, calculate between device A and forwarding unit C, forwarding unit C and equipment BTransfer rate, and with the least transmission rate value for reference to being grouped.Basic transmission when device A is transmitted according to itThe forwarding that the transmission performance parameter of data rate and forwarding unit group chooses one group of forwarding performance to meet predefined transmission requirement setsStandby group, as selected forwarding unit group, i.e. selects transfer rate to pass more than in the alternative forwarding unit group of basic transmitted data ratesMaximum one group of defeated speed is as selected forwarding unit group, and sets up repeated link.
Self image information is captured by device A, by the time when image information captured and crawl and equipmentThe parameters such as identification number are encapsulated in transmitting data frame in the lump, and to equipment B and have built up link chain by Bluetooth transmission networkThe selected forwarding unit group connect sends same transmission frame number evidence；The selected forwarding unit group established the link with device A and equipment B willThe transmission data received are forwarded to equipment B by Bluetooth transmission network.When equipment B receives device A or selected forwarding unitGroup starts after the data of any one device transmission timing, and waits the data that other equipment transmit.Timing time exceedes pre-If maximum delay time, equipment B notice not yet transmission data is transferred to the device A corresponding to the link of equipment B or forwarding setsStandby C abandons the transmission data of time-out；Then the transmission data having been received by are carried out selection combining by equipment B.Equipment B will processAfter transmission data carry out screen and show.
When a certain moment is carried out synchronicity detection from the transmission data that device A and equipment C receive, calculate previous momentThe asynchronous number of times of terminal feedback of each road picture transmission information, by itself and the acceptable threshold of the asynchronous number of times of terminal feedback arrangedThe not acceptance threshold of the asynchronous number of times of value, terminal feedback compares, and in Ruo Gelu transmission data, minimum terminal feedback is asynchronousThe number of times acceptable thresholds less than the asynchronous number of times of terminal feedback or the not acceptance threshold more than the asynchronous number of times of terminal feedback,Then select the video data that road transmission data are equipment B that the asynchronous number of times of terminal feedback is minimum.In Ruo Gelu transmission dataMinimum asynchronous number of times is not between the acceptable thresholds of the asynchronous number of times of terminal feedback and accepting of the asynchronous number of times of terminal feedbackBetween threshold value, then the display number that transmission data are equipment B that in the transmission data that selected equipment B receives, occurrence number is mostAccording to, if there are the transmission data that occurrence number is identical, then choose transmission data that wherein the asynchronous number of times of terminal feedback is low for settingThe video data of standby B.
In sum, the present invention proposes the method for a kind of Bluetooth control display content synchronization, reduces the direct chain of AR equipmentThe pressure on road to video smoothness influential effect.
Obviously, it should be appreciated by those skilled in the art, each module of the above-mentioned present invention or each step can be with generalCalculating system realize, they can concentrate in single calculating system, or be distributed in multiple calculating system and formedNetwork on, alternatively, they can realize with the executable program code of calculating system, it is thus possible to by they storePerformed by calculating system within the storage system.So, the present invention is not restricted to the combination of any specific hardware and software.
It should be appreciated that the above-mentioned detailed description of the invention of the present invention is used only for exemplary illustration or explains the present invention'sPrinciple, and be not construed as limiting the invention.Therefore, that is done in the case of without departing from the spirit and scope of the present invention is anyAmendment, equivalent, improvement etc., should be included within the scope of the present invention.Additionally, claims purport of the present inventionWhole within containing the equivalents falling into scope and border or this scope and border change and repairChange example.