Summary of the invention
The purpose of this invention is to provide a kind ofly in the multi-mode estimation, can be easy determine the method for inter prediction macroblock partition pattern fast.
This method may further comprise the steps:
With the various possible pattern of cutting apart of one 16 * 16 block of pixels (being called " macro block " down), be divided into from 0 to 10 totally 11 grades by the particulate degree of cutting apart (granularity).
The pattern of cutting apart of 11 grades being divided into is formed a tree by certain division rule, in this structure, because many patterns of cutting apart have towards characteristic, promptly because the difference of each pixel brightness value spatial distribution makes the sad value difference of different size difference towards the block of pixels estimation, be divided into level towards with vertical towards two classes, therefore this structure forms towards tree, and the pattern of cutting apart of different stage constitutes " tree root " respectively, has towards subtree and do not have towards subtree.
Utilization is cut apart pattern and is determined inter prediction macroblock partition pattern apace towards map is easy.
Utilization is cut apart pattern and is determined that fast the method for inter prediction macroblock partition pattern comprises towards map is easy:
Select the originate mode collection: select one to comprise set of patterns that minority cuts apart pattern, see 601 among Fig. 6,602,603 and 604 for details as the originate mode collection.
Select originate mode.
The result who selects according to originate mode determines the optimum segmentation pattern of each macro block in first P frame, forms initial optimum segmentation pattern and towards map.
With the optimum segmentation pattern of determining in the previous step with towards map next predictive frame (B or P) is handled.
Wherein select the method for originate mode to comprise:
Whole algorithm is that unit carries out with GOP, and GOP is in the digital video signal sequence, from an I frame, all the P frames before and the summation of B frame occur to next I frame.The starting point of determining entire method is first P frame of digital video signal sequence units GOP.
Adopt " cost function mcost " as weighing a criterion of cutting apart the pattern quality.Cost function mcost is a numerical value that combines residual sum SAD and other coding expense (for example motion vector data etc.).If cut apart the cost function that pattern causes and cut apart the cost function that pattern causes less than another for one, then this is cut apart pattern and is better than another and cuts apart pattern.
Determine initial optimum segmentation pattern and comprise towards the method for map:
Select a macro block, it is a maximum MAX (701) that cost function mcost is set.
Concentrate one of taking-up arbitrarily to cut apart pattern from originate mode.
Adopt this pattern that selected macroblock is carried out exploratory estimation, and write down the cost function mcost of this pattern correspondence: if this cost function of cutting apart pattern then keeps the current pattern of cutting apart greater than current cost function value; If this cost function of cutting apart pattern less than current cost function value, is then cut apart pattern with this and is replaced the current pattern of cutting apart.If this cost function of cutting apart pattern equates with current cost function value, be chosen in that of in tree rank numbering less (being that fine granularity is thicker); If fine granularity is also identical, then select one arbitrarily.
Using such method respectively each macro block in first P frame of each GOP is adopted that originate mode concentrates other cut apart pattern and carry out estimation, all relatively after, that pattern of cost function mcost minimum is the optimum segmentation pattern.
After determining the initial optimum segmentation pattern of each macro block of first P frame of each GOP, promptly obtain initial optimum segmentation pattern and towards map.What this map will instruct each macro block in the later predictive frame of this GOP cuts apart determining of pattern.
Comprise with optimum segmentation pattern of determining in the previous step and the method for next predictive frame (B or P) being handled towards map:
Pass through principle of inertia, promptly utilize the correlation between present frame and the reference frame, the default pattern MODE0 (901 of cutting apart during with the cutting apart pattern (be documented in cut apart pattern and in map) and cut apart the model selection process and begin of reference frame correspondence position macro block as current macro, 902), carry out other estimation of integer pixel level.Draw the default SAD (residual sum) under the pattern, total residual sum SAD0 and the cost function mcost0 cut apart of selected macroblock.
Adopt trim step (alligatoring and refinement):
Check earlier total residual sum SAD0 whether " satisfaction ";
Because for same macro block, the sad value that causes than the coarse segmentation pattern cuts the SAD of pattern less than segmenting scarcely, therefore, this method quantitatively defines " satisfaction " with " the SAD upper limit " and " SAD lower limit "." the SAD upper limit " and " SAD lower limit " is to calculate two empirical values that obtain according to the neighbor information of selected macroblock.The way of a specific implementation example is: for current macro, according to adjacent macroblocks correlation spatially, by the SAD of its left side macro blockL, the top macro block SADU, the upper right side macro block SADURAnd the SAD of upper left side macro blockULDetermine respectively " the SAD upper limit " and " SAD lower limit ":
The SAD upper limit=(SADL+ SADU+ SADUR+ SADUL)/4+C1
SAD lower limit=(SADL+ SADU+ SADUR+ SADUL)/4-C2
Wherein the quantity of the sad value of addition is denominator, C1And C2It is the empirical of determining by experiment.
SAD0 is compared with " the SAD upper limit " and " SAD lower limit ",, accept to cut apart the motion estimation result (905) that pattern draws with this if SAD0, just thinks that this default mode meets coding requirement between between the two.If SAD0 is greater than " the SAD upper limit ", this method thinks that this default pattern of cutting apart gets too slightly, needs " refinement ".If SAD0 is less than " SAD lower limit ", this method is thought needs this default pattern split hairs of cutting apart " alligatoring ".As the total residual sum SAD1 that the default pattern of cutting apart carried out alligatoring/refinement, then can get making new advances and new integrate-cost function m cost1.
Compare step: be about to new integrate-cost function m cost1 and default integrate-cost function m cost0 compares, keep less one.
Modification is cut apart pattern and towards map: determine whether to revise according to the result in the comparison step and cut apart pattern and towards map, promptly keep and confirm the pairing mode of integrate-cost function m cost that keeps in the above-mentioned steps.In view of the above to cut apart pattern and in map the entry of this macro block make amendment or prolong usefulness, and in the next predictive frame the default pattern of cutting apart of the macro block of correspondence position.
Along in the process of tree alligatoring and refinement, exist to allow and the pattern of cutting apart that is not allowed to.As current cut apart pattern and in topographic map corresponding selected macroblock towards be masked as level towards, then refinement/alligatoring rule allows this to cut apart pattern to be cut apart pattern by refinement/alligatoring for another level, be an isotropism pattern (will keep level simultaneously) perhaps, but can not refinement/alligatoring be that a nothing is towards pattern towards sign by refinement/alligatoring.(in like manner be applicable to vertically towards.)
As current cut apart pattern and in topographic map corresponding selected macroblock towards be masked as level towards, to cut apart pattern be an isotropism pattern (all having some isotropism patterns towards subtree with vertical in subtree in level) and sound out, and this pattern need be by refinement/alligatoring, it is that a level is cut apart pattern by refinement/alligatoring that then refinement/alligatoring rule allows this to cut apart pattern, but can not refinement/alligatoring be that a nothing is towards pattern.(in like manner be applicable to vertically towards.)
Situation when the method that the present invention proposes extends to more than a reference frame.In the time of need using a plurality of reference frame as the estimation of present frame, cutting apart pattern and all preserved each reference frame towards map.These reference frames are all from same GOP, but because the existence of refinement and alligatoring mechanism, therefore with regard to each macro block, pairing is consistent towards sign in each reference frame corresponding map; But in each map, provide to cut apart pattern then not necessarily consistent.
The present invention is directed to such as H.264 waiting and realize the rather time-consuming situation of many sized blocks estimation that high-performance code adopted in the standard, proposed a kind ofly in multiple different inter prediction macroblock partition pattern, to select a kind of pattern of better cutting apart rapidly, effectively reduced the method for above-mentioned selection course institute spended time.The present invention is applicable to the two-way reference (B-Frame) in the standard such as MPEG-2 simultaneously, H.264 waits multiframe in the standard with reference to (Multi-frame Reference).
Embodiment
Below in conjunction with accompanying drawing embodiments of the invention are described in further detail.
Fig. 3 shows, we can be with the various possible pattern of cutting apart to one 16 * 16 block of pixels (hereinafter referred to as " macro block "), is divided into from 0 to 10 totally 11 grades by the particulate degree of cutting apart (granularity).Wherein, the 0th grade is had only a pattern, promptly this macro block is not done any pattern of cutting apart.The 1st grade has two patterns, is about to the pattern that this macro block is divided into the pattern of two 16 * 8 block of pixels and is divided into two 8 * 16 block of pixels.The 2nd grade has a pattern, is about to the pattern that this macro block is divided into four 8 * 8 block of pixels.Or the like.
Fig. 4 shows that the present invention forms a tree with the pattern of cutting apart of 11 grades shown in Figure 3 by certain division rule.This tree " tree root " has only one, is in the 0th grade, promptly this macro block do not done the pattern of any cutting apart (401).This tree root pattern has two direct " children " to cut apart pattern (the 1st grade of pattern), promptly is divided into the pattern (402) of two 16 * 8 block of pixels and is divided into the pattern (403) of two 8 * 16 block of pixels.Wherein, notice that pattern (402) has the advantages that level is cut apart, and pattern (403) have the characteristics of vertical segmentation.In general, concerning same 16 * 16 block of pixels, take exercises with two 16 * 8 block of pixels and to estimate the sad value that obtains, the sad value that obtains with the estimation of taking exercises with two 8 * 16 block of pixels is different.Its result is often relevant with the spatial distribution of each pixel brightness value in this macro block, we this specific character call block of pixels in cutting apart towards characteristic, total level towards with vertical towards two classes.
The 2nd grade of tree has two patterns, i.e. pattern (404) and pattern (405), and they all are the block of pixels that former 16 * 16 macro blocks is divided into 48 * 8.We notice that it is duplicate cutting apart pattern for these two, and segmentation obtains but they pass through further by different " father and mother " patterns (402) and (403) respectively, so list it respectively in tree.
From the 2nd grade down, whole tree is divided into to be had towards subtree (comprise level towards subtree with vertical towards subtree) and not to have towards subtree.There is " growth " rule to be: children's pattern of each pattern towards subtree, be on the basis of this pattern, for any one square pixel piece (8 * 8) of remaining in the horizontal direction on (descendants) or the vertical direction for 402 (descendants) to 403 once divide equally just and form; Be on the basis of this pattern, with a pair of have towards the rectangular pixels piece be divided into four 4 * 4 block of pixels and form.This process is performed until this macro block and all is divided into till 16 4 * 4 block of pixels (410).According to this rule as seen, from the 2nd grade of growth, only used article one rule to 3rd level; And from 3rd level growth course down, two rules have all been used; Only used the second rule from the 9th grade to the 10th grade.
What form like this has following three characteristics towards subtree:
1, the closer to the pattern of cutting apart of root, fine granularity spatially is thick more; And more away from the pattern of cutting apart of root, the fine granularity on the space is thin more.
2, by in 402 patterns level down each pattern in subtree, only exist divided in horizontal direction or isotropism to cut apart (i.e. four 4 * 4 block of pixels or four 8 * 8 block of pixels), do not exist vertical direction to cut apart.And, only exist vertical direction to cut apart or isotropism is cut apart by in 403 patterns vertical each pattern in subtree down, there is not divided in horizontal direction.
3, with vertical some is cut apart pattern and is equal in branch, for example pattern (404) is identical with (405) towards branch for level, and pattern (410) is the leaf of all branches, etc.
From 9 grades of 3rd levels to the, every level that promptly do not belong to or not vertically mixing towards subtree again towards cutting apart pattern towards subtree, and we are included into nothing towards subtree with them., in cutting apart pattern, or cut apart by isotropism entirely and form a nothing, or both having contained level cuts apart and also contain vertical segmentation.Nothing also is divided into some levels by the particulate degree of cutting apart towards subtree, and wherein the pattern in each grade is one 8 * 8 block of pixels just once to be segmented and obtain (for example pattern 408 and 409 relation) on upper level pattern basis.In the method, Tong Ji pattern particulate degree is considered to equally.
Fig. 5 shows that the bare bones of cutting apart mode selecting method proposed by the invention comprises:
● the whole algorithm process is that unit carries out with GOP.GOP is a unit of digital video signal sequence.General digital video signal sequence is made of some frames, comprising I frame (promptly can not rely on the information of other frame to the encoding and decoding of this frame and independently carry out), P frame (promptly need rely on the frame that early occurs in the sequence to the encoding and decoding of this frame on time shaft or the information of some frames just can be carried out) and B frame (promptly need rely in the sequence on time shaft the encoding and decoding of this frame and just can carry out morning and two frames of later appearance or the information of some frames) etc.From an I frame, all P frames and B frame before occur to next I frame and be called as a GOP.Initial first P frame of this method since a GOP.
● select originate mode collection and originate mode.H.264 waiting in the video encoding standard, need sound out the more than one pattern of cutting apart usually, cutting apart pattern and can cause forced coding this macro block so which kind of to be determined to the inter prediction formula of each 16 * 16 macro block in the P frame (or B frame) coding.So-called " the best " can have many different criterions, and for the sake of simplicity and be without loss of generality, we use " cost function (mcost) " to be used as weighing a criterion of cutting apart the pattern quality in the following description.Through after the exploratory calculating, cost function can be quantified as an integer.Cost function mcost is a numerical value that combines residual sum SAD and other coding expense (for example motion vector data etc.).In our method,, just think that this is cut apart pattern and is better than another and cuts apart pattern if one cuts apart the cost function that pattern causes and cuts apart the cost function that pattern causes less than another.All are cut apart in the pattern in that this method is soundd out, and the best pattern of cutting apart is that of cost function minimum.
When this method is carried out predictive coding to a GOP, we do not wish Fig. 3 is listed all cut apart pattern and all sound out one time, the required like that time is too many.For this reason, we at first select one to comprise minority and cut apart the originate mode collection of pattern and sound out (501).In the method, the pattern of cutting apart of selected originate mode collection can be arbitrarily, and Fig. 6 shows, in a specific implementation example, we are with 2 16 * 8 (seeing 601), 28 * 16 (seeing 602), and 88 * 4 (seeing 603), 84 * 8 (seeing 604) are cut apart pattern as the originate mode collection for these 4 kinds.Cutting apart pattern for these 4 all has towards property, and wherein,pattern 601 and 602 is arranged in Fig. 4 the 1st grade towards tree, andpattern 603 and 604 is arranged in Fig. 4 the 6th grade towards tree.
To first P frame among each GOP cut apart pattern determine it is the basis of subsequent process.Therefore, when allow computing time, concentrate at originate mode also to comprise more kinds of patterns of cutting apart.
● determine initial optimum segmentation pattern and towards map.To each macro block of first P frame of each GOP, each pattern that this method adopts originate mode to concentrate is carried out exploratory estimation (502), and detailed process is seen Fig. 7.At first cost function is initialized as a maximum MAX (701).Cut apart cost function that pattern causes greater than current cost function value if one new, then keep the current pattern of cutting apart; Cut apart cost function that pattern causes less than current cost function value if one new, then replace the current pattern of cutting apart with this new pattern of cutting apart.If new cost function equates with existing cost function value, then be chosen in that of in tree rank numbering less (being that fine granularity is thicker); If fine granularity is also identical, then select one arbitrarily.
After the optimum segmentation pattern of each macro block of first P frame of each GOP all determines, we just obtain as shown in Figure 8 initial optimum segmentation pattern and towards map.What this figure will instruct each macro block in the later predictive frame of this GOP cuts apart determining of pattern.
Fig. 9 shows, utilize cut apart pattern and towards map to a GOP in the method handled of all the other predictive frames (B or P) except that first P frame.Its details was divided into for four steps:
● principle of inertia: because the motion relevance between the motion image sequence consecutive frame, the optimum segmentation pattern of consecutive frame correspondence position macro block is close often, or even identical.Therefore, utilize the correlation between present frame and the reference frame, the default pattern (901 of cutting apart during with the cutting apart pattern (be documented in cut apart pattern and in map) and cut apart the model selection process and begin of reference frame correspondence position macro block as current macro, 902), and carry out other estimation of integer pixel level.
Above-mentionedly cut apart the result that pattern is put in order the pixel scale estimation, comprise that this cuts apart the SAD of each piecemeal in the pattern (residual sum), total residual sum SAD0 of this macro block, and a cost function mcost0 with default.
● trim step (alligatoring and refinement): next this method enters trim step.Whether the total residual sum SAD0 that checks earlier this macro block " satisfaction "." the SAD upper limit " and " SAD lower limit " with SAD0 and this macro block compares for this reason.Here, " the SAD upper limit " and " SAD lower limit " is to extrapolate according to the neighbor information of this macro block.The way of a specific implementation example is: for current macro, according to adjacent macroblocks correlation spatially, by the SAD of its left side macro blockL, the top macro block SADU, the upper right side macro block SADURAnd the SAD of upper left side macro blockULDetermine:
The SAD upper limit=(SADL+ SADU+ SADUR+ SADUL)/4+C1
SAD lower limit=(SADL+ SADU+ SADUR+ SADUL)/4-C2
C wherein1And C2It is the empirical of determining by experiment.
Because for same macro block, the sad value that causes than the coarse segmentation pattern cuts the SAD of pattern less than segmenting scarcely, therefore, this method quantitatively defines " satisfaction " with " the SAD upper limit " and " SAD lower limit ".If SAD0, just thinks that this default mode meets coding requirement between between the two, accept to cut apart the motion estimation result (905) that pattern draws with this.If SAD0 is greater than " the SAD upper limit ", this method thinks that this default pattern of cutting apart gets too slightly, needs " refinement ".If SAD0 is less than " SAD lower limit ", this method is thought needs this default pattern split hairs of cutting apart " alligatoring ".
In the method, alligatoring and refinement have two common features:
1, alligatoring and refinement are all being carried out under the sign constraint, that is to say, for each macro block, if to the coding of first P frame of a GOP time, be registered as alignment target, then among this GOP in all subsequent prediction frames the macro block of this position all will carry out estimation with the pattern of cutting apart of level in subtree; If to the coding of first P frame of a GOP time, be registered as vertical sign, then among this GOP in all subsequent prediction frames the macro block of this position all will carry out estimation with the vertical pattern of cutting apart in subtree.
2, alligatoring and refinement are all only carried out at the adjacent level in tree.That is to say, if the current default pattern of cutting apart is positioned at towards the K level of setting, then the result of alligatoring selects one in same K-1 level towards tree to cut apart pattern, and the result of refinement selects one in same K+1 level towards tree to cut apart pattern (0<K<10).
● comparison step.The result of alligatoring or refinement still needs further to compare with former default mode, could determine whether be used.Criterion relatively is integrate-cost function m cost.If the mcost of new model is less, then adopts the new pattern of cutting apart, otherwise keep former default mode.
● revise and to cut apart pattern and towards map.If in above-mentioned fine setting and comparison step, accepted the default mode m ode0 of cutting apart, then needn't to cut apart pattern and in map the entry of this macro block make amendment, this is default cuts apart the macro block that pattern will be in use to correspondence position in the next predictive frame.If alligatoring or refinement have taken place in above-mentioned trim step and have adopted the new mode m ode1 of cutting apart to substitute the default mode m ode0 of cutting apart, then cut apart pattern and in map the entry of this macro block be modified to mode1, as the default pattern of cutting apart of the macro block of correspondence position in the next predictive frame.
Figure 10 shows, along in tree alligatoring and thinning process, permission cut apart model selection with unallowed.Wherein, Figure 10 (A) expression, if current cut apart pattern and towards corresponding this macro block of topographic map kind towards be masked as level towards, then refinement rule allows this to cut apart pattern to be refined as another level and to cut apart pattern, perhaps be refined as an isotropism pattern (will keep level simultaneously), but can not be refined as a nothing towards pattern towards sign.(in like manner be applicable to vertically towards.) Figure 10 (B) demonstration, if current cut apart pattern and towards corresponding this macro block of topographic map kind towards be masked as level towards, to cut apart pattern be an isotropism pattern (all having some isotropism patterns towards subtree with vertical in subtree in level) and sound out, and this pattern need be by refinement, then refinement rule allows this to cut apart pattern to be refined as a level and to cut apart pattern, but can not be refined as a nothing towards pattern.(in like manner be applicable to vertically towards.)
Figure 10 (C) shows, if current cut apart pattern and towards corresponding this macro block of topographic map kind towards be masked as level towards, then alligatoring rule allows this to cut apart pattern to be cut apart pattern by alligatoring for another level, be an isotropism pattern (will keep level simultaneously) perhaps, but can not alligatoring be that a nothing is towards pattern towards sign by alligatoring.(in like manner be applicable to vertically towards.) Figure 10 (D) demonstration, if current cut apart pattern and towards corresponding this macro block of topographic map kind towards be masked as level towards, to cut apart pattern be an isotropism pattern and sound out, and this pattern need be by alligatoring, then to allow this to cut apart pattern be that a level is cut apart pattern by alligatoring for alligatoring rule, but can not alligatoring be that a nothing is towards pattern.(in like manner be applicable to vertically towards.)
Figure 11 shows, the situation the when method that the present invention is proposed is generalized to more than a reference frame.The two-way reference (B-Frame) that this is applicable in the standard such as MPEG-2 is applicable to that also the multiframe that H.264 waits in the standard is with reference to (Multi-frame Reference).As shown, if the estimation of present frame need be used reference frame A, B, C, and D, then this method requires reference frame A, B, C, D's cuts apart pattern and is all preserved towards map.Because these reference frames are all from same GOP, therefore with regard to each macro block, pairing towards sign at map A, map B, map C and is consistent among the map D; But because the existence of alligatoring and refinement mechanism, usually, with regard to each macro block, the pattern of cutting apart that it provides in each map is not necessarily consistent.In a specific implementation example, our regulation: with regard to a macro block, it finally cuts apart determining and need each map all being compared by the flow process of Fig. 9 of pattern, and cuts apart pattern with that of cost function mcost minimum as it.When certain two map occurring and provide same cost function just, this method is selected thicker that of particulate degree.