US20090067728A1 - Image matching method and image interpolation method using the same - Google Patents

Abstract

An image matching method includes setting first and second lattices to first and second images respectively, computing potential force to each second lattice point of the second lattice by a gradient of an image correlation potential energy based on a position of each first lattice point and pixel thereof and a position of the second lattice point and pixel thereof, computing elasticity force to the second lattice point from elasticity energy between the second and adjacent lattice points, computing frictional force occurring at the second lattice point, performing a numerical analysis of an equation of motion regarding the second lattice point and based on the potential force, elasticity force and frictional force to obtain a convergence state of the second lattice points, and adding a new lattice point between an adjacent lattice point pair of second lattice points according to a distance between the adjacent lattice point pair.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This is a division of application Ser. No. 11/355,144, filed Feb. 16, 2006, which is incorporated herein by reference in its entirety.
• This application is based upon and claims the benefit of priority from prior Japanese Patent Applications No. 2005-039623, filed Feb. 16, 2005; and No. 2005-336291, filed Nov. 21, 2005, the entire contents of both of which are incorporated herein by reference.
BACKGROUND OF THE INVENTION
• 1. Field of the Invention
• The present invention relates to an image matching method for matching images by detecting corresponding points from two images, and an image interpolation method for interpolating the image using the same.
• 2. Description of the Related Art
• A technique of image matching to obtain association between one image and the other image is a basic technique in many technical fields such as motion detection, stereo matching, image morphing, image recognition, and video encoding.
• According to Murat Tekalp, “Digital Video Processing”, Prentice Hall, 1995, a technique of image matching can be sorted to four methods. In other words, there are an optical flow method, a block data base method, a gradient method, and a Bayesian method.
• The optical flow method is a method for deriving an optical flow type equation defining that “a change of luminance is constant” and obtaining a flow using the optical flow equation as condition of constraint. The block data base method is a method for obtaining a motion by block-by-block template matching. The gradient method is a method for performing matching in a direction in which luminance gradient of an image decreases. The Bayesian method is a method for obtaining plausible matching stochastically.
• Japanese patent No. 2927350 discloses an image matching method using a multiple resolution filter. This method is an image matching method for generating a pyramid of a plurality of multi-resolution images by means of a plurality of multiple resolution filters, and permitting an image matching from a high speed motion to a low speed motion with a high robustness by subjecting the image pyramid to a matching process sequentially from the top thereof.
• A method for obtaining a smooth mapping relation is mentioned in Japanese patent No. 2927350. However, there is a problem that constraint condition to make the mapping smooth acts on a discontinuous part of motion, resulting in that the discontinuous mapping cannot be expressed.
• This is a trade off to occur always in a conventional image matching method. There is a problem that if the mapping is smoothed, it comes to be difficult to express discontinuity, and if the discontinuity is expressed, the smoothness becomes impaired.
• It is an object of the present invention to provide an image matching method that even if the mapping is smoothed, discontinuity is not impaired, and even if discontinuity is expressed, smoothness is obtained, and an image interpolation method using the same.
BRIEF SUMMARY OF THE INVENTION
• An aspect of the present invention provides an image matching method for detecting a mapping from a first image to a second image, comprising: setting a first lattice having a plurality of first lattice points to a first image; setting a second lattice having a plurality of second lattice points, each second point of the second lattice points of corresponding to each first lattice point of the first lattice points, to a second image; computing potential force applied to the each second lattice point by a gradient of an image correlation potential energy based on a position of the each first lattice point, pixel data of the each first lattice points, a position of the each second lattice point and pixel data of the each second lattice point; computing elasticity force applied to the each second lattice point from an elasticity energy between the each second lattice point and adjacent lattice points on the second lattice which are adjacent to the each second lattice point; computing frictional force occurring at the each second lattice point; performing a numerical analysis of an equation of motion regarding the second lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points; and adding a new lattice point between an adjacent lattice point pair of the second lattice points according to a distance between the adjacent lattice point pair while the numerical analysis of the equation of motion is performed.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
• FIG. 1 is a block diagram of an image matching apparatus executing an image matching method according to a first embodiment of the present invention.
• FIG. 2 is a diagram for explaining dynamics concerning an image matching problem.
• FIG. 3 is a diagram showing an adjacent lattice relation of the other object.
• FIG. 4 is a conceptual diagram for explaining addition of lattice points.
• FIG. 5 is a diagram showing a connection between spaces.
• FIGS. 6A and 6B are diagrams representing a force due to an image correlation potential energy.
• FIG. 7 is a diagram showing a state which phases of an object on a source image and an object of on a reference image are maintained.
• FIG. 8 is a diagram showing a heterogeneous lattice point space.
• FIG. 9 is a diagram for explaining addition of lattice points.
• FIG. 10 is a diagram for explaining analysis of a lattice point addition model.
• FIG. 11 is a diagram of a mapping when a lattice point is not added.
• FIG. 12 is a diagram showing a mapping obtained by a simplified lattice point addition.
• FIG. 13 is a block diagram of an image matching apparatus executing an image matching method according to a second embodiment.
• FIG. 14 is a flow chart of the image matching method of the second embodiment.
• FIG. 15 is a diagram showing an occlusion region.
• FIG. 16 is a diagram for explaining a state solved by occlusion matching.
• FIG. 17 is a diagram for explaining occlusion reliability.
• FIG. 18 is a diagram representing a sigmoid function.
• FIG. 19 is a diagram representing a sigmoid function.
• FIG. 20 is a diagram for explaining an interpolation method.
• FIG. 21 is a diagram for explaining an occlusion matching interpolation method.
• FIG. 22 is a block diagram of an image matching apparatus executing an image matching method according to a third embodiment.
• FIG. 23 is a flow chart showing the image matching method of the third embodiment.
• FIG. 24 is a diagram showing an occlusion region obtained by backward matching.
• FIG. 25 is a block diagram of an image matching apparatus executing an image matching method according to a fourth embodiment of the present invention.
• FIG. 26 is a diagram showing a connection between spaces.
• FIG. 27 is a diagram showing an occlusion region obtained by backward matching.
• FIG. 28 is a diagram for explaining that forward and backward occlusions can be detected by the expansion and contraction of a spring.
• FIG. 29 is a diagram for explaining that the detection of occlusion is not satisfied only by detection of floating out.
• FIG. 30 is a diagram for explaining combination of images by a plurality of reliabilities.
• FIG. 31 is a diagram showing to provide a robust even if a right map is not provided.
• FIG. 32 is a block diagram of a matching apparatus executing a matching step.
• FIG. 33 is a diagram for explaining optimization due to a static energy function.
• FIG. 34 is a diagram for explaining optimization due to a static energy function.
• FIG. 35 is a flow chart for explaining an image matching method according to a fourth embodiment.
• FIG. 36 is a flow chart showing a procedure of a discrete variable method.
• FIG. 37 is a flow chart showing a procedure for computing a power generated by a potential energy in an image correlation.
• FIG. 38 is a flow chart showing a procedure for computing a power due to elasticity energy.
• FIG. 39 is a flow chart showing a procedure for computing a frictional force.
• FIG. 40 is a flow chart showing a procedure of interpolation computation.
• FIG. 41 is a block diagram of an image matching apparatus executing an image matching method according to a fifth embodiment.
• FIG. 42 is a diagram showing a state that a matching error appears as elongation of the spring.
DETAILED DESCRIPTION OF THE INVENTIONFirst Embodiment
• There will be described the present embodiment according to a block diagram ofFIG. 1.
• According toFIG. 1, animage matching unit11 comprising a processor including a memory and so on comprises amodel creation module12 and anumerical analysis module13. When a source image and a reference image are input to theimage matching unit11, the same parts of the object and reference images stored in a frame memory, for example, the same parts of 3D image are matched. In other words, when two images of the object and reference images are input, a dynamic model is created using dynamic concept. Because this dynamic model is output from themodel creation module12 in form of an ordinary differential equation, thenumerical analysis module13 solves the output repeatedly by a general numerical solution. The image of the result provided in the last of repeated computation becomes a final state of matching. In the present embodiment, a latticepoint addition module14 comprising, for example, a processor is provided. The analysis of the dynamic model is performed repeatedly while adding a lattice point sequentially repeatedly as discussed below.
• Explaining the above referring toFIG. 2, a lattice space having, for example, nine lattice points is provided temporarily on each of the source image and reference image. In this case, the lattice points correspond to the image parts (pixels) of the source image and reference image. The lattice spaces of the source image and reference image have the same size, and the corresponding points are set beforehand. For example, the middle point of the lattice space of the source image corresponds to the middle point of the lattice space of the reference image. Matching of the object image with the reference image is done in a state that corresponding points are decided. An equation of motion is made in this matching. The equation of motion makes the movements of the lattice points uniquely correspond to each other between the object and reference images with respect to a certain time. It is possible to make a model transforming the lattice of the image in a suitable form by solving the equation of motion during a certain time. Assuming that there is an image wherein the object moves, the lattice space deformed by movement of the lattice point as shown inFIG. 2 is generated. When the computation is stopped at a certain time T after elapse of a time τ, a converged deformation lattice space is generated. This is a basic dynamic model.
• A lattice point addition will be explained. InFIG. 3, an object is adjacent to a lattice point Xn in the object image, but in reference image, the object separates from a lattice point Yn. In other words, in the reference image, since a distance between the lattice point Yn+(1,0) on the object and the lattice point Yn increases, a spring between two lattice points expands. In this case, the object moves, so that a space appears in a region on the reference image which corresponds to a region of the object on the source image. A lattice point is added to this space as shown inFIG. 4. In this case, a lattice point is added, whenever the distance between the lattice points exceeds 2.
• The image matching and lattice point addition are explained in detail hereinafter. In a fundamental image matching, the following model is thought as a source image. A continuous image model is expressed as shown inequation 1.
• $\begin{array}{cc}{s}_c\left(x,t\right),\left[\begin{array}{c}x\\ t\end{array}\right]\in {\mathrm{<figure-callout\; label="R"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">R</figure-callout>}}^3& 1\end{array}$
• This is a continuous image model of a real number basis. Because a digital image is regarded as a source image, the following sampling image model obtained by sampling the above model is used. This sampling image model is expressed by thefollowing equation 2.
• $\begin{array}{cc}{s}_p\left(x,t\right),\left[\begin{array}{c}x\\ t\end{array}\right]=\left[\begin{array}{cc}\mathrm{<figure-callout\; label="V"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">V</figure-callout>}& 0\\ 0& \Delta t\end{array}\right]\left[\begin{array}{c}n\\ k\end{array}\right]\in {\mathrm{<figure-callout\; label="\Lambda "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\Lambda </figure-callout>}}^3& 2\end{array}$
• where V is a sampling matrix, coupling the lattice space nεΛ²and real number space xεR²as shown inFIG. 5. In this time, the image matching matter is formulated as a matter searching for a motion vector d:R³→R²satisfying the following condition, that is,equation 3.
• 
  s_p(x,t)=s_c(x+d(x,t;lΔt),t+lΔt)  3
• However, in the case of this formulation, the image value of the same object is assumed not to vary according to a time. Because the motion vector d is a real number, it should be noted that the right-hand side uses a designator of a continuous image model (equation 1). Because matching between two images, i.e., the source image and reference image is conceived, the followingequivalent equation 4 is established.
• 
  s_k1,p(x)=s_k2,c(x+d(x))  4
• k₁,k₂εΛ are variables representing the source image and the reference image, respectively, and these times are t₁=k₁Δt,t₂=k₂Δt.
• The motion vector is simplified to be d:R²→R². As the motion vector d has only to be decided just one with respect to the point x, the motion vector d is assumed to be a one-to-one mapping. Since x+d(x) can be considered to be a mapping with respect to x, g(x)=x+d(x) is defined. g indicates an unique mapping of g:R²→R². The above matching problem results in a problem searching for a mapping g satisfying thefollowing equation 5.
• 
  s_k1,p(x)=s_k2,c(g(x))  5
• The point decided by the mapping g is defined as y=g(x). yεR2 is defined. Because x=Vn, the lattice point x corresponds to a point n on the lattice space uniquely. Because the mapping g is a unique mapping, y is unique with respect to the lattice point x, too. Accordingly, the lattice point y corresponds to n uniquely. This is shown inFIG. 5. In other words, the space to be processed is a deformed lattice space corresponding one-to-one therewith by the point n on the lattice space.
• As described above, because y=g(x)=g(Vn), this is redefined to yn=g(x) to make one-to-one correspondence understandable. Theequation 5 of the image matching problem results in a problem searching for ynopt satisfying thefollowing equation 6.
• 
  s_k1,p(x)=s_k2,c(y_n)  6
• Dynamics is introduced to the point yn to solve theequation 6 of the image matching problem. In other words, the image matching problem results in a problem solving a dynamical system concerning the point yn. The point yn moves in a state satisfying theequation 6 as considering a relation with surrounding points to converge in an equilibrium state. It is assumed that the image matching is completed by the equilibrium state. This state is expressed as shown inFIG. 2.
• A new time axis τεR is introduced to the point y, and the function yn (T) is defined. Assuming that an initial value is same as a rectangular lattice x as thefollowing equation 7.
• 
  y_n(0)=x  7
• When the dynamics converge
• $\mathrm{in}\tau {->}^{\infty },\underset{\tau ->\infty }{\lim }y_n\left(\tau \right)=y_{n\mathrm{opt}}$
• is defined. Because a new time axis is introduced, the derivation concerning a time can be defined by thefollowing equation 8.
• $\begin{array}{cc}\frac{}{\tau }y_n\left(T\right)={\stackrel{.}{y}}_n\left(T\right),\frac{{}^2}{{\mathrm{<figure-callout\; label="\tau "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\tau </figure-callout>}}^2}y_n\left(T\right)={\ddot{y}}_n\left(T\right)& 8\end{array}$
• The ordinal dynamics is described by the following ordinarydifferential equation 9.
• 
  ÿ_n(T)=F,y_n(0)=x,{dot over (y)}_n(0)=0  9
• Fε R²is a sum total of forces. This is referred to as a equation of motion.
• A force applied to the point yn(τ) is conceived. At first a force due to a potential energy becoming a force making the dynamics drive is considered. This is a force for moving the point yn(τ) to a state satisfying theequation 6. Theequation 5 is transformed to the followingequation 10.
• 
  s_k2,c(y_n)−s_k1,p(x)=0  10
• It is difficult due to noise components included in the image to search for the point which satisfies theequation 10 closely. Consequently, the energy function as expressed by the followingequation 11 is conceived.
• 
  E_u(n)=(s_k2,c(y_n(T))−s_k1,p(x)²  11
• The point which this energy function Eu becomes smallest is searched for. If the principle of steepest descent method is used, the point can arrives at the local minimum by coming down in a direction of the steepest descent of the energy function Eu around the point yn(T). Accordingly, the gradient in the direction of this steepest descent is defined as a force to the point yn(τ). Since the energy function Eu is conceivable as correlation between images, this force is assumed to be a force Fu due to an image correlation potential energy.
• Although various methods of computing a gradient in a direction of the steepest descent are conceivable, the following method is adopted in the present embodiment. A differential value between the source image and the reference image is shown inFIG. 6A, and the gradient in the steepest descent direction is obtained directly by local optimization as shown inFIG. 6B. The image model Sc is continuous image model, but actually, only the sampled image model Sp can be used. Consequently, the local optimization is done based on the sampled image model. Because the sampling point most near the point yn(τ) should be a local space center yc as shown inFIG. 6A, it is obtained by the followingequation 12.
• $\begin{array}{cc}{y}_c=\lfloor y_n\left(T\right)+\left[\begin{array}{c}0.5\\ 0.5\end{array}\right]\rfloor .& 12\end{array}$
• ^└•┘ indicates an operator to be converted to an integer (omission of fractions). When the adjacency space is defined by anequation 13, a local search space Q can be defined by the followingequation 14.
• $\begin{array}{cc}{N}_s=\left\{\begin{array}{c}\left[\begin{array}{c}-1\\ -1\end{array}\right],\left[\begin{array}{c}0\\ -1\end{array}\right],\left[\begin{array}{c}1\\ -1\end{array}\right],\left[\begin{array}{c}-1\\ 0\end{array}\right],\\ \left[\begin{array}{c}0\\ 0\end{array}\right],\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}-1\\ 1\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right],\left[\begin{array}{c}1\\ 1\end{array}\right]\end{array}\right\}& 13\\ \Omega =\left\{y_n\left(T\right),y_c+1|1\in N_s\right\}& 14\end{array}$
• If a vector in the steepest descent direction is obtained by local optimization, normalized, and multiplied by a magnitude of gradient, the followingequation 15 is provided. This equation shows a force due to an image correlation potential energy (square error energy).
• $\begin{array}{cc}{F}_u\left(x,y_n\left(T\right)\right)=\frac{{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_{\min }\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_n\left(T\right)\right)\right)}^2}{u_d}u_du_d=y_{\min }-y_n\left(T\right)y_{\min }=\arg \underset{y\in \Omega }{\min }{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,p}\left(y\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(x\right)\right)}^2& 15\end{array}$
• It is possible to use a force due to the image correlation potential energy (absolute value difference error energy) of the followingequation 17 wherein the energy function is defined by the following equation 16 in view of easy handling in mounting.
• $\begin{array}{cc}{E}_u\left(n\right)=s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_n\left(T\right)\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(x\right)F_u\left(x,y_n\left(T\right)\right)=\frac{s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_{\min }\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_n\left(T\right)\right)}{u_d}u_d& 16\\ u_d=y_{\min }-y_n\left(T\right)y_{\min }=\arg \underset{y\in \Omega }{\min }s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,p}\left(y\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(x\right)& 17\end{array}$
• Subsequently, a force describing a relation of the point with the peripheral points is conceived. The image to be subjected to matching is assumed to be an image obtained by reflecting a three-dimensional space to a two-dimensional plane. When an object on the three-dimensional space is assumed to be a rigid body, the surface of the rigid body is observed as an object on the two-dimensional image. If the object on three-dimensional space is observed on the source image and reference image, in this time, the phase of the object observed on each image is maintained at a high probability. As shown inFIG. 7, the positional relation of the point x on the object of the source image should be maintained on the point yn(τ) on the reference image object, too. This behavior can be simulated by connecting between the point yn(τ) and adjacent points by a spring. The relation between the object and the periphery is described by the force Fk of the spring. At first a lattice point space Nn surround the object point is defined as follows. In the case of four points surrounding the object point, the followingequation 18 is established.
• $\begin{array}{cc}{N}_n=\left\{\left[\begin{array}{c}0\\ -1\end{array}\right],\left[\begin{array}{c}-1\\ 0\end{array}\right],\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\right\}& 18\end{array}$
• If a spring constant (elasticity constant) is supposed to be k, the resilience of the spring applied to the point yn(τ) is a spring force expressed by anequation 19. The elasticity constant is a balancer between an image correlation energy and an elasticity energy. If the elastic constant has a large value, the lattice space is hard to be deformed, and thus a result is stable. However, the adequateness to the image becomes bad. If the elastic constant has a small value, the lattice space is easy to be deformed, and thus adequateness to the image is improved. However, consequence becomes too flexible. Consequently, this parameter is given from experience for the present. Because the behavior is not sensitive to the value of this parameter, a certain constant value is given fixedly basically.
• $\begin{array}{cc}{F}_k\left(y_n\left(T\right),\left\{y_{n+1}\left(T\right)|\forall 1,1\in N_n\right\}\right)=\sum _{1\in N_n}k\left(y_{n+1}\left(T\right)-y_n\left(T\right)\right)& 19\end{array}$
• When four points surrounding the lattice point are connected, a four-point connection spring model as expressed by the followingequation 20 is obtained.
• $\begin{array}{cc}{F}_k\left(y_n\left(T\right),\left\{y_{n+1}\left(T\right)|\forall 1,1\in N_n\right\}\right)=k\left(y_{n+\left[\begin{array}{c}0\\ -1\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)+k\left(y_{n+\left[\begin{array}{c}-1\\ 0\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)+k\left(y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)+k\left(y_{n+\left[\begin{array}{c}0\\ 1\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)& 20\end{array}$
• At last, a force making a stored energy dissipate is conceived.
• Energy is saved if a force applied to the point yn (τ) is only Fu, Fk, so that a system becomes an oscillating stationary state. Consequently, a force making the saved energy dissipate is introduced. This force can use a frictional force. When a speed can be near to a constant value, the frictional force can be expressed by the followingequation 21.
• 
  F_μ({dot over (y)}_n(T))=−μ{dot over (y)}_n(T)  21
• If the above forces are collected, the equation of motion is expressed by the followingequation 22.
• Equation of Motion
• 
  ÿ_n(τ)=F_u(x,y_n(T))+F_k(y_n(T),{y_n+1(T)|∀l,lεN_n})+F_μ({dot over (y)}_n(T)),y_n(0)=x,{dot over (y)}_n(0)=0  22
• Because the force Fu due to the image correlation potential energy is not solved for in analysis, the ordinarydifferential equation 22 is not solved for in analysis. Accordingly, it is difficult to take the limit inτ→∞ of a system. Consequently, a convergence state of the system is estimated by computing an interval of t=(0,T) by a numerical analysis, in view of a sufficiently longer time T required for convergence of the system.
• The ordinary differential equation provides a solution uniquely by a numerical analysis if an initial value is determined. Generally, this is called an initial value problem of the ordinary differential equation. There are many numerical solutions for solving this problem, for example, a Euler method, a Runge Kutta method, a Bulirsch-Soer method, a predictor-corrector method, an implicit Runge Kutta method. The Runge Kutta method is most famous and has a high use frequent. However, theequation 22 is hard to fit a complicated numerical solution because it has a dimension for an image size. Consequently, the Euler method that implementation is the simplest is adopted in this embodiment. Because the Euler method is a numerical solution for the ordinary differential equation of first-order, at first theformula 22 is converted into a one-time ordinary differential equation. Then, the variable is transformed as expressed by the followingequation 23.
• $\begin{array}{cc}\{\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}\left(T\right)=y_n\left(T\right)\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(T\right)={\stackrel{.}{y}}_n\left(T\right)\end{array}& 23\end{array}$
• This transformation of variable is substituted for theequation 22 to make the followingequation 24.
• Converted equation of motion
• $\begin{array}{cc}\{\begin{array}{c}{\stackrel{.}{\psi }}_{1,n}\left(T\right)={\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(T\right)\\ {\stackrel{.}{\psi }}_{2,n}\left(T\right)=F_u\left(\mathrm{Vn},{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}\left(T\right)\right)+F_k\left(\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}\left(T\right),\\ \left\{\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n+1}\left(T\right)|\forall 1,\\ 1\in N_n\end{array}\right\}\end{array}\right)+F_{\mu }\left({\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(T\right)\right),\end{array}\hspace{1em}[\begin{array}{c}{\psi }_{1,n}\left(0\right)\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(0\right)\end{array}]=\left[\begin{array}{c}\mathrm{Vn}\\ 0\end{array}\right]& 24\end{array}$
• where ψ_1,n(T),ψ_2,n(T) correspond to the position and velocity of yn(T), respectively.
• The scheme of the Euler method for the ordinarydifferential equation 25 is expressed by the followingequation 26.
• 
  {dot over (x)}(t)=F(x(t),t)  25
• 
  x⁽ⁿ⁺¹⁾=x⁽ⁿ⁾+hF(x⁽ⁿ⁾,t)  26
• This makes a solution progress from t(n) to t⁽ⁿ⁺¹⁾=−t⁽ⁿ⁾+h, where x(n) indicates an n-th step and h indicates a step size. When the scheme of Euler method is applied to theequation 24, there is provided an updating equation due to Euler method of the followingequation 27.
• $\begin{array}{cc}\{\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n+1\right)}={\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n\right)}+h{\psi }_{2,n}^{\left(n\right)}\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(n+1\right)}={\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(n\right)}+h\left[\begin{array}{c}{F}_u\left(\mathrm{Vn},{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n\right)}\right)+\\ F_k\left({\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n\right)},\left\{\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n+1}^{\left(n\right)}|\forall 1,\\ 1\in N_n\end{array}\right\}\right)+F_{\mu }\left({\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(n\right)}\right)\end{array}\right],\end{array}\left[\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(0\right)}\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(0\right)}\end{array}\right]=\left[\begin{array}{c}\mathrm{Vn}\\ 0\end{array}\right]& 27\end{array}$
• The above image matching is collected as algorithm as follows.
• Image Matching Algorithm
• 1. Initial value is set to
• $\left[\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(0\right)}\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(0\right)}\end{array}\right]=\left[\begin{array}{c}\mathrm{Vn}\\ 0\end{array}\right]$
• 2. A lattice point addition step (described below).
• 3. The value of time T⁽ⁿ⁺¹⁾is computed by an updating equation
• $\hspace{1em}\{\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n+1\right)}={\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n\right)}+h{\psi }_{2,n}^{\left(n\right)}\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(n+1\right)}={\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(n\right)}+h\left[F_u\left(\mathrm{Vn},{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n\right)}\right)+F_k\left({\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}^{\left(n\right)},\left\{{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n+1}^{\left(n\right)}|\forall 1,1\in N_n\right\}\right)+F_{\mu }\left({\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}^{\left(n\right)}\right)\right]\end{array}$
• in the time τ⁽ⁿ⁾.
• 4. T⁽ⁿ⁺¹⁾=T⁽ⁿ⁾+h
• 5. If the relation is T⁽ⁿ⁺¹⁾>T, the process is ended, otherwise the lattice point addition step of theabove item 2 is done.
• Accordingly, the solution y_noptof theequation 6 of the image matching is obtained by y_nopt=ψ_1,n^(T).
• The lattice point addition step is explained hereinafter. This is theabove process2 of the image matching algorithm.
• If there are adjacent deformation lattice points y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right),$
• to open both points greatly is to generate a large spring energy therebetween and occurs a large force. If y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right)$
• belong to different objects respectively, nothing should exist between y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right)$
• originally. This conflicts with that large spring energy is accumulated between y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right)$
• This is shown inFIG. 3. A large space broaden between y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right),$
• so that it is not correct to connect with a spring between y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right).$
• It is preferable for solving this contradiction to add a new lattice point between the lattice points y_n(T) and
• $y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right)$
• as shown inFIG. 4, for example. As a result, it is expected that a spring is provided between the adjacent lattice points y_n+(3,0)Tand y_n+(4,0)Ton an object boundary, so that no large spring energy occurs.
• A heterogeneous lattice point space shown inFIG. 8 is introduced for the purpose of formulating concept to add a lattice point. If the heterogeneous lattice point space is used, the lattice point addition can be expressed by conformation as shown inFIG. 9. The spring force applied between the lattice points is decreased to ½ by adding one lattice point as shown inFIG. 9. Therefore, if a lattice point is added between the lattice points when a distance between the lattice points exceeds 2, the spring force becomes equivalent to that when the spring does not expand. Therefore, the lattice point addition algorithm is as follows:
• 1. When N is assumed to be a neighbor lattice space, l_asatisfying ∥y_n+lεN(T)−y_n(T)∥>2.0 is searched for.
• 2. Lattice points
• $x=V\left(n+\frac{1}{2}1_a\right)\mathrm{and}y_{n+\frac{1}{2}1_a}\left(T\right)=\frac{1}{2}\left(y_{n+1_a}\left(T\right)+y_n\left(T\right)\right)$
• are added.
• The processing returns to the image matching step after having added the lattice point.
• When adding the lattice point as described above, it is to introduce the heterogeneous lattice point space, whereby computation becomes complicated. Consequently, the lattice point addition model was analyzed. As a result, it is found that the lattice points connected to each other by the spring I of a spring constant k are subjected to a force of kI in the normal spring model as shown inFIG. 10. When a lattice point is added between the lattice points, the spring decreases to I/2 in length, resulting in that the spring force applied to each of the lattice points is decreased to kI/2. This corresponds to the spring I of a spring constant k/2. In other words, it is equivalent to set the spring constant to ½ to add one lattice point between the springs. However, because the potential of the added lattice point is not considered, the spring is only equivalent with respect to a spring model consistently, but not equivalent with respect to the whole dynamic matching model including the potential.
• The above explanation describes balance between the forces of the springs in the equilibrium state thereof and does not describe on the dynamic behavior of the spring. In other words, it simply describes that the spring forces are equivalent in static.
• It is equivalent to making a spring constant 1/(n+1) times to add n lattice points to the spring. It is the lattice point addition algorithm to add one lattice point if the length of the spring doubles, two lattice points if it is three times, three lattice points if it is four times, . . . .
• This can be simplified by setting the spring constant to ½ if the length of the spring doubles, setting the spring constant to ⅓ if it is three times, setting the spring constant to ¼ if it is four times, . . . . A spring model of such a variable type can be expressed as follows.
• A spring model of a simplified model lattice point addition
• $\begin{array}{cc}{F}_k\left(y_n\left(T\right),\left\{y_{n+1}\left(T\right)|\forall 1,1\in N_n\right\}\right)=\sum _{1\in N_n}k\frac{1}{\lfloor ||y_{n+1}\left(T\right)-y_n\left(T\right)||\rfloor }\left(y_{n+1}\left(T\right)-y_n\left(T\right)\right)& 28\end{array}$
• where^└•┘ is an operator to be converted to an integer.
• The following spring model can be used elsewhere.
• A linear function spring model varying in linear.
• $\begin{array}{cc}{F}_k\left(y_n\left(T\right),\left\{y_{n+1}\left(T\right)|\forall 1,1\in N_n\right\}\right)=\sum _{1\in N_n}k\frac{1}{||y_{n+1}\left(T\right)-y_n\left(T\right)||}\left(y_{n+1}\left(T\right)-y_n\left(T\right)\right)& 29\end{array}$
• A logistic function spring model
• $\begin{array}{cc}{F}_k\left(y_n\left(T\right),\left\{y_{n+1}\left(T\right)|\forall 1,1\in N_n\right\}\right)=\sum _{1\in N_n}k\sigma \left(||y_{n+1}\left(T\right)-y_n\left(T\right)||\right)\left(y_{n+1}\left(T\right)-y_n\left(T\right)\right)\sigma \left(r\right)=\frac{1}{1+\exp \left(\alpha \left(r-\beta \right)\right)}& 30\end{array}$
• where α>0, ⊖>0 are constants.
• In other words, the simplified lattice point addition algorithm uses a spring model of the equation 28. More specifically, the spring model of the equation 28 may be employed in the image matching algorithm.
• An effect was confirmed by a test pattern which a round object moves in parallel. As a result, when the lattice point addition is not done, a result shown inFIG. 11 is provided. According to this, it is found that a distortion occurs on a backward boundary of the object. In contrast, a result obtained by carrying out the simplified model lattice point addition is shown inFIG. 12. According to this, it is found that no distortion occurs on a backward boundary of an object.
• As described above, the present embodiment can provide a highly precise image matching method that can express discontinuity of movement by an algorithm adding a lattice point.
Second Embodiment
• (Single Direction Occlusion Model)
• The present embodiment is explained according to the block diagram ofFIG. 13 and the flow chartFIG. 14. The present embodiment considers a problem how an interpolation is done for the purpose of making an occlusion region correspond to a motion image well, and is directed to generating an interpolation frame containing an occlusion area between frames of an image based on an image signal of a motion image.
• When an interpolation frame is generated, there is an occlusion problem making it difficult to generate an interpolation frame of high quality. The present embodiment provides a method for generating an interpolation frame of high quality by interpolating an occlusion region adequately.
• Schematically, this method performs an ordinal image matching (surface matching) in a time interval t1-t2, and an occlusion matching in a time interval t2-t3. A reliability of an occlusion region is obtained from a result of the ordinal matching in the time interval t1-t2 based on a certain standard. The interpolation frame is generated from this reliability, result of the image matching and result of occlusion matching. The present embodiment is explained in detail hereinafter.
• An image signal of a motion picture is assumed to be divided into a plurality of frames, and subjected to processing. In this time, each frame is stored in aframe memory21 for each division, and some frames can be accessed at the same time. A frame in a time t is referred to as a reference frame t, a frame in a time t+lΔt is referred to as a reference frame t+lΔt, and a frame of time t+2lΔt is referred to as a reference frame t+2lΔt. An interpolation frame is interpolated between the reference frame t and reference frame t+lΔt.
• An occlusion region must be detected adequately for the occlusion region to be interpolated adequately. The ground that can be detected by the present embodiment adequately are explained referring toFIG. 15.FIG. 15 shows an example in which a round object moves to the right from the left. On the reference frame t+lΔt, a uncovered occlusion uncovered occlusion region occurs at a position before the round object moves.
• The lattice points on the rear part of the round object, that is, lattice points xn, xn+(1,0)T and yn, yn+(1,0)T are noted. Since the round object moves, if it is subjected to matching, an interval between the lattice points yn, yn+(1,0)T should expand greatly. In other words, a spring between the lattice points yn, yn+(1,0)T is expanded. When the uncovered occlusion region appears by movement of the object as described above, it is found that the spring always expands. In other words, the part that the spring expands greatly is an occlusion region. The superscript “T” expresses transposition.
• FIG. 16 show one-dimensionally a movement of the round object shown in two-dashed line, that is, shows a state that the round object moves downward. The uncovered occlusion appears on the rear of the round object between the time t and the time t+lΔt. It is found that the spring expands then. When the round object just moves downward, it is found that a part corresponding to the uncovered occlusion region appears at the time t+2lΔt. In other words, if matching is performed between the reference frame t+lΔt and the reference frame t+2lΔt, a region corresponding to the occlusion region can be found. That is to say, when the object moves, the region hid by the moving object (hidden region or occlusion region) appears. In this time, when watching a state of the lattice point, the spring on the occlusion region expands. Therefore, it can be determined whether it is occlusion by the length of the expanding spring. This is referred to as occlusion matching.
• In the above example, the distance between the lattice points is used as an evaluation value, and the occlusion is determined when the distance is greater than a reference value. The occlusion can be detected based on distribution of lattice points. Density of distribution of the lattice points decreases on the occlusion region because the spring expands thereon. Therefore, the density may be used as an evaluation value for determining occlusion. In this case, the density lower than the reference is detected as occlusion. For example, the reference frame t+2lΔt is divided into plural equal rectangular regions, and the lattice points in each region are counted. The number of lattice points is a quantity corresponding to the density of distribution of the lattice points.
• Otherwise, the occlusion can be detected using an area of a region surrounded by the lattice points instead of the density. The area of the region surrounded by the lattice points increases on the occlusion region because the spring expands thereon and thus the distance between the lattice points increases. Therefore, the area can be used as an evaluation value for determining occlusion. In this case, the region having an area greater than the reference is detected as occlusion. For example, an area of a triangular region having the lattice points as apices and including no other lattice points, or an area of a rectangular region having the lattice points as apices and including no other lattice points is used.
• It can be understood from the above that the occlusion region is detected and the region corresponding to the occlusion region is subjected to matching. Thereafter, these results may be combined for generating an interpolation frame. The present embodiment will be described referring to the block circuit ofFIG. 13 and the flow chart ofFIG. 14 concretely hereinafter.
• In an image matching step S21, a matching image g(;t;lΔt) is acquired using the frame t as an source image and the frame t+lΔt as a reference image, with animage matching module22. The image matching step S21 performs the same processing as the first embodiment.
• In an image matching step S22, the image matching for finding the occlusion region is done using the frame t+lΔt as an source image and the reference frame t+2lΔt as a reference image, with anocclusion matching module23. This processing is the same as theimage matching step11 of the first embodiment. Therefore, the same processing as the image matching of the first embodiment is done. Occlusion matching is done for acquiring an occlusion image g(;t+lΔt;lΔt).
• A conventional image matching method can be used in the occlusion matching module. In other words, a conventional motion estimation method mentioned in Japanese patent No. 2927350, for example, a block matching algorithm, an optical flow estimating method, a gradient method, a bays method can be used.
• The block matching algorithm is described as an example. The block matching algorithm is a method for assigning the same motion vector to the lattice points in the block with assumption that movements in a block are the same.
• The whole screen is divided into plural (ordinarily, rectangular) blocks each including lattice points. In this example, a rectangular block is adopted.
• A point on the upper left of the block is assumed to be a base point XεΛ². Each block is assumed to be B(X). A motion vector to each block is obtained as follows.
• $\begin{array}{cc}\tilde{d}\left(X\right)=\arg \underset{d\in \Omega }{\min }\sum _{x\in B\left(X\right)}{\left(s_p\left(x,t+l\Delta t\right)-s_p\left(x+d,t+2l\Delta t\right)\right)}^2& 31\end{array}$
• where ΩεΛ²is a searching space. Since a motion vector {tilde over (d)}(X) is obtained for the block, this is assigned to each lattice point as expressed by the followingequation 32.
• 
  d(x,t+lΔt;lΔt)={tilde over (d)}(X),∀xεB(X)  32
• Therefore, the mapping is obtained by the followingequation 33.
• 
  g(x,t+lΔt;lΔt)=x+d(x,t+lΔt;lΔt)=x+{tilde over (d)}(X),∀xεB(X)  33
• The occlusion reliability computation step S23 carried out by the occlusionreliability computation module24 is described hereinafter. The occlusion reliability is the probability distribution representing whether the pixels of the region are in the occlusion region. The probability that the region is an occlusion region increases as the occlusion reliability nears 1. When the occlusion region is detected by the length of the spring, the reliability in the region must be computed actually as described previously.
• In the present embodiment, in view of a problem how an interpolation which makes the occlusion region correspond to the pixels of the motion image in units of pixel well is done, there is used, for example, a triangular area surrounded by the lattice points yn, yn+(0,−1),s□Ayn+(−1,−1),s as shown inFIG. 17. The area of this triangular region is obtained from an absolute value of outer product of a vector by the followingequation 34. The superscript “T” expresses a transpose.
• $\begin{array}{cc}{\mathrm{<figure-callout\; label="S"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">S</figure-callout>}}_1\left(n\right)=\frac{1}{2}\left|\left(y_{n+{\left(0,-1\right)}^T}-y_n\right)\times \left(y_{n+{\left(-1,-1\right)}^T}-y_n\right)\right|& 34\end{array}$
• Similarly, the area of the lower triangular region is computed by the followingequation 35.
• $\begin{array}{cc}{\mathrm{<figure-callout\; label="S"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">S</figure-callout>}}_2\left(n\right)=\frac{1}{2}\left|\left(y_{n+{\left(-1,-1\right)}^T}-y_n\right)\times \left(y_{n+{\left(-1,0\right)}^T}-y_n\right)\right|& 35\end{array}$
• Because the area of each of the triangular region xn, xn+(0,−1)T, xn+(−1,−1)T and the triangular region xn, xn+(−1,0)T, xn+(−1,−1)T is ½, if the ratio between the triangular regions is computed, the ratio between the areas of the triangular regions is expressed by the followingequations 36 and 37. The superscript “T” expresses a transpose.
• 
  R₁(n)=|(y_n+(0,−1)T−y_n)×(y_{n+(−1,−1)T}−y_n)|  36
• 
  R₂(n)=|(y_{n+(−1,−1)T}−y_n)×(y_n+(−1,0)T−y_n)|  37
• A probability distribution is acquired from these triangular ratios. If the length of the spring exceeds a certain constant value, it is assumed to be occlusion. In this case, if the triangular ratio exceeds a certain constant value, it is determined to be occlusion. A sigmoid function according to the following equation 38 as shown inFIG. 18 can be used as such a transform function.
• $\begin{array}{cc}\sigma \left(r\right)=\frac{1}{1+\exp \left(-\alpha \left(r-\beta \right)\right)}& 38\end{array}$
• where α indicates a constant representing inclination and β indicates coordinates of the center of the inclination.
• The triangle in the lattice point n is assumed to be R1(n), R2(n). The following processing is done on all lattice points n, and an occlusion reliability map is generated according to the following equation 39.
• 
  P(y|g(•,t;lΔt))=σ(R₁(n)),∀yεR₁(n)
• 
  P(y|g(•,t;lΔt))=σ(R₂(n)),∀yεR₂(n)  39
• A motion compensation step is done by amotion compensation unit25, and includes an interpolation frame generation step carried out by an interpolationframe generation module26 and an image correlation reliability computation step S25 done by an image correlationreliability computation module27. The image correlation reliability computation step is called in the interpolation frame generation step.
• The interpolation frame generation step S24 will be described. The method similar to the Digital Image Warping is adopted.
• In the Digital Image Warping, a motion vector is obtained for each point of a rectangle. The rectangle is divided into two triangles, and an affine transformation is computed on the basis of three vertexes for each triangle (as described in J. Nieweglowski, T. George Campbell, P. Haavisto, □gA
• Novel Video Coding Sheme Based on Temporal Prediction Using Digital Image Warping□h, IEEE Transactions on, Volume: 39 Issue: 3, Aug. 1993, the entire contents of which are incorporated herein by reference). A triangular image to be interpolated by the affine transformation is obtained (as described in J. Nieweglowski, P. Haavisto, □gMotion Compensated Video Sequence Interpolation Using Digital Image Warping□h, IEEE Acoustics, Speech, and Signal Processing, vol. 5, p 205-208, 1994, the entire contents of which are incorporated herein by reference).
• As shown in Motion Compensated Video Sequence Interpolation Using Digital Image Warping, the affine transformation from a triangle having vertexes V={(u_i,v_i)^T, (i=0, 1, 2)} to a triangle having vertexes Y={(x_i,y_i)^T, (i=0, 1, 2)} can be expressed by the followingequation 40.
• $\begin{array}{cc}T\left(V,Y\right)=\frac{1}{\det \left(U\right)}B\left[\begin{array}{ccc}{x}_0& {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">y</figure-callout>}}_0& 1\\ x_1& {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ x_2& {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\end{array}\right]B=\left[\begin{array}{ccc}{v}_1-v_2& v_2-v_0& v_0-v_1\\ u_2-u_1& u_0-u_2& u_1-u_0\\ {\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_1v_2-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_2v_1& {\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_2v_0-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">u</figure-callout>}}_0v_2& {\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">u</figure-callout>}}_0v_1-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_1v_0\end{array}\right]\det \left(U\right)={\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">u</figure-callout>}}_0\left(v_1-v_2\right)+{\mathrm{<figure-callout\; label="v"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">v</figure-callout>}}_0\left(u_2-u_1\right)+\left({\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_1v_2-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_2v_1\right)& 40\end{array}$
• It can be computed using the affine transformation that a point (u,v)^Tin the triangle V corresponds to a point (x,y)^Tin a triangle expressed by [x,y,1]=[u,v,1]T(V,Y). The superscript “T” expresses a transpose.
• An interpolation frame is obtained by the affine transformation. A time position of the reference frame t is set to 0, and a time position of the reference frame t+lΔtk is set to 1. A time position of the interpolation frame is assumed to beΔt_i(0≦Δt_i␣1). When a mapping g(;t;lΔt) is subjected to linear transformation according to Δti, it becomes an image represented by gi(x,t,lΔt)=x+Δtid (x,t,lΔt). The interpolation frame is generated as shown inFIG. 20.
• A triangle having vertexes {g(x1.t; lΔt), g(x2.t;lΔt), {g(x3.t;lΔt)} is assumed to be W, and a triangle having vertexes {gi(x1.t;lΔt), gi(x2.t;lΔt), {gi(x3.t;lΔt)} is assumed to be Y. A pixel value of a point in the triangle W on the interpolation frame shown by the followingequation 41 will be obtained.
• 
  wεR²|w=VnεΛ²  41
• Affine transformation from the triangle W to the triangle X and affine transformation from the triangle W to the triangle are obtained by theequation 40, and a pixel value corresponding to the point w is obtained by the former and latter affine transformations. In other words, the point in the triangle which corresponds to the point w can be computed by the following equation 42:
• 
  [x^T1]=[w^T1](W,X)  42
• Similarly, the point in the triangle which corresponds to the point w can be computed by the followingequation 43.
• 
  [y^T1]=[w^T1]T(W,Y)  43
• If a weight average of both frames is obtained according to the time position, a pixel value of the interpolation frame can be computed by the interpolation frame obtained by the followingequation 44 and the mapping g(;t;lΔt).
• 
  s_p(w,t+lΔt_i)=Δt_is_c(x,t)+(1−Δt_i)s_c(y,t+lΔt)  44
• An interpolation frame based on the mapping g(;t;lΔt) is generated by doing this computation for all triangles in a screen. An occlusion reliability map on the interpolation frame is generated by the occlusion reliability map as follows. The occlusion reliability map on the interpolation frame according to the mapping g(;t;lΔt) is expressed by the followingequation 45.
• 
  P_o(w,t+lΔt_i|g(•,t;lΔt))=P(y|g(•,t;lΔt))  45
• Similarly, an image correlation reliability map due to the mapping g(;t;lΔt) is generated based on theequation 46 in the image correlation reliability computation step described below.
• 
  P(w,t+lΔt_i|g(•,t;lΔt),s(•,t),s(•,t+lΔt))=p(s_c(x,t),s_c(y,t+lΔt))  46
• Because the occlusion mapping g(;t+lΔt;lΔt) produced by occlusion matching is due to matching from the reference frame t+lΔt to the reference frame t+2lΔt, the occlusion mapping is subjected to linear transformation based on the followingequation 47 to generate a matching image between the reference frame t+lΔt and the reference frame t.
• 
  g_o(x,t+lΔt;−lΔt)=x−d(x,t+lΔt;lΔt)  47
• The frame is assumed to be −lΔt because a direction of the image is inversed. When the occlusion mapping g(;t+lΔt;lΔt) is subjected to linear transformation according to Δti, it becomes an image based on the followingequation 48.
• 
  g_i,o(x,t+lΔt;−lΔt)=x−(1−Δt_i)d(x,t+lΔt;lΔt)  48
• Correspondence between the points is shown inFIG. 21. Assuming that a triangle having vertexes {gi,o (x1.t+lΔt; −lΔt), gi,o (x2.t+lΔt; −lΔt), gi,o (x3.t+lΔ; −lΔt)} is indicated by Wo, a triangle having vertexes {x1,x2,x3} is indicated by Xo, and a triangle having vertexes {go(x1.t+lΔt; −lΔt), go(x2.t+lΔt; −lΔt), go(x3.t+lΔt; −lΔt)} is indicated by Yo. A point in the triangle Xo corresponding to the point w is computed according to the following equation 49.
• 
  [x_o^T1]=[w^T1]T(W_o,X_o)  49
• Similarly, a point in the triangle Yo corresponding to the point w is computed according to the followingequation 50.
• 
  [y_o^T1]=[w^T1]T(W_o,Y_o)  50
• If a weight average of both frames is computed according to a time position, a pixel value of an interpolation frame due to the occlusion mapping g(;t+lΔt;lΔt) can be computed by the followingequation 51.
• 
  s_o,p(w,t+lΔt_i)=(1−Δt_i)s_c(y_o,t)+Δt_is_c(x_o,t+lΔt)  51
• If this processing is executed for all triangles in a screen, the interpolation frame due to the occlusion mapping g(;t+lΔt;lΔt) is generated. An image correlation reliability map due to the occlusion mapping g(;t+lΔt;lΔt) is generated based on the followingequation 52 in the image correlation reliability computation step S25.
• 
  P(w,t+lΔt_i|g(•,t+lΔt;lΔt),s(•,t),s(•,t+lΔt))=p(s_c(y_o,t),s_c(x_o,t+lΔt))  52
• Because the interpolation frame due to the mapping g(;t;lΔt) and the interpolation frame due to the occlusion mapping g(;t+lΔt;lΔt) can be generated by the above processing, these are combined by a reliability map. The probability of reliability given by the image correlation and the probability of reliability given by the occlusion are independent from each other.
• Therefore, the probability that the occlusion matching is adopted is expressed by the followingequation 52 according to the multiplication theorem of probability.
• 
  P₁=P_o(w,t+lΔt_i|g(•,t;lΔt))P(w,t+lΔt_i|g(•,t+lΔt;lΔt),s(•,t),s(•,t+lΔt))  53
• The probability that the ordinary matching is adopted is obtained by the followingequation 54.
• 
  P₀={1−P_o(w,t+lΔt_i|g(•,t;lΔt))}P(w,t+lΔt_i|g(•,t;lΔt),s(•,t),s(•,t+lΔt))  54
• The transparency to be used for combination is computed from these probabilities by the followingequation 55.
• $\begin{array}{cc}{\alpha }_j=\frac{{\mathrm{<figure-callout\; label="P\; j\; P"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">P</figure-callout>}}_j}{P_{}},\left(j=0,1\right)& 55\end{array}$
• The interpolation frame due to the mapping and the interpolation frame due to the occlusion mapping are combined using this transparency and based on the following equation 56.
• 
  s_i,p(w,t+lΔt_i)=α₀s_p(w,t+lΔt_i)+α₁s_o,p(w,t+lΔt_i)  56
• The interpolation frame Si, p(w,t+lΔt_i) can be generated by doing the above-mentioned processing for the whole screen.
• The image correlation reliability computation step S25 is described hereinafter. This is to obtain a reliability of a corresponding point due to the mapping relation of each point. The differential value between the pixel values at each point can be used as such a reliability. In other words, the image correlation reliability is computed according to the following equation 57.
• $\begin{array}{cc}p\left({\mathrm{<figure-callout\; label="s"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">s</figure-callout>}}_1,s_2\right)=\frac{1}{1+\left|s_1-{\mathrm{<figure-callout\; label="s"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">s</figure-callout>}}_2\right|}& 57\end{array}$
• This is a model that the reliability increases with a decrease of the differential value of the image. However, correlation between the images and a differential value between the pixels have no linear correlation generally. In other words, if the differential value between the pixels is small, the correlation between the images may be high. However, if the differential value between the pixels exceeds a certain constant value, the correlation between the images descends irrespective of the differential value between the pixels. The sigmoid function as shown inFIG. 19 may be used for introducing such a non-linearity. This is assumed to be α<0 in the equation 38. If this is used, the image correlation reliability can be defined by the following equation 58.
• $\begin{array}{cc}p\left({\mathrm{<figure-callout\; label="s"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">s</figure-callout>}}_1,s_2\right)=\sigma \left(s_1-{\mathrm{<figure-callout\; label="s"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">s</figure-callout>}}_2\right)\sigma \left(r\right)=\frac{1}{1+\exp \left(\alpha \left(r-\beta \right)\right)}& 58\end{array}$
• where α>0, β>0 are constants.
• As described above, the present embodiment permits generation of the interpolation frame of high quality, which corresponds to a uncovered occlusion region.
Third Embodiment
• (Bidirectional Occlusion Model)
• The present embodiment is explained according to the block diagram ofFIG. 22 and the flow chart ofFIG. 23. The present embodiment is directed to generating an interpolation frame between the frames of an image signal of a motion image.
• When the interpolation frame is generated, it is difficult owing to occlusion problem to generate an interpolation frame of high quality. The second embodiment provides a method dealing with uncovered occlusion. The present embodiment can deal with covered occlusion as well as uncovered occlusion, by matching in bi-direction of forward and backward with respect to a time axis.
• The image signal of the motion image is divided into frames, and subjected to processing. In this time, each frame is stored in aframe memory31, and some frames can be accessed at the same time. A frame in the time t is referred to as a reference frame t, a frame in the time t+lΔt is referred to as a reference frame t+lΔt, a frame in the time t+2lΔt is referred to as reference frame t+2lΔt, and a frame in the time t−lΔt to as a reference frame t−lΔt. An interpolation frame is interpolated between the reference frame t and the reference frame t+lΔt.
• As explained in the second embodiment, a expansion of a spring corresponds to a uncovered occlusion region as shown inFIG. 15. In this case, because matching is done in the forward direction with respect to the time axis, the forward matching is hit. However, the covered occlusion region on the forward part of an object is not considered. When the covered occlusion region is viewed by backward matching as shown inFIG. 24, it appears as a uncovered occlusion region. Accordingly, it is found that it is possible to devise a countermeasure for both occlusions by combining the forward matching with the backward matching.
• In an image matching step S31 executed by animage matching module32, matching is done in forward and backward directions with respect to the time axis to produce two mappings. The internal processing is the same as the image matching step of the first embodiment. In other words, the matching of the first embodiment is done using the reference frame t as an source image and the reference frame t+lΔt as a reference image to output a mapping g(;t;lΔt) of the forward matching. The matching of the first embodiment is done using the reference frame t+lΔt as an source image and the reference frame t as a reference image to output a mapping g(;t+lΔt; −lΔt) of the forward matching.
• In an occlusion matching step S32 done by anocclusion matching module33, the matching is done in the forward and backward with respect to the time axis to produce two occlusion mappings. The internal processing is the same as the image matching of the first embodiment. In other words, the matching of the first embodiment is done using the reference frame t+lΔt as an source image and the reference frame t+2lΔt as a reference image to output a mapping g(;t+lΔt;lΔt) of the forward occlusion matching. The matching of the first embodiment is done using the reference frame t+2lΔt as an source image and the reference frame t+lΔt as a reference image to output a mapping g(;t; −lΔt) of the forward matching.
• An occlusion reliability computation step S33 executed by an occlusionreliability computation module34 is similar to the second embodiment. A forward occlusion reliability map P(ylg(;t;lΔt)) is computed based on the forward matching mapping g(;t; −lΔt). A backward occlusion reliability map P(ylg(;t+lΔt;−lΔt) is computed based on the forward matching mapping g(;t+lΔt; −lΔt).
• The motion compensation step comprises an interpolation frame generation step S34 done by an interpolationframe generation module36 and an image correlation reliability computation step S35 done by an image correlationreliability computation module37. The image correlation reliability computation step S35 is similar to the second embodiment. The image correlation reliability computation step S35 is accessed in the interpolation frame generation step S34.
• The interpolation frame obtained by the forward matching is similar to the second embodiment. In other words, the interpolation frame and reliability map are obtained by theequations 26, 27, 28, 29, 30. The interpolation frames are changed in subscript for identification as follows:
• 
  s_f,p(w,t+lΔt_i),s_f,o,p(w,t+lΔt_i)
• A time position of the interpolation frame to be interpolated is assumed to be Δt_i(0≦Δt_i≦1), where a time position of the reference frame t is set to 0 and a time position of the reference frame t+lΔt is set to 1. When the backward matching mapping g(;t+lΔt; −lΔt) is subjected to linear transformation according to Δti, the following equation 59 is established.
• 
  g_i(x,t+lΔt;−lΔt)=x+(1−Δt_i)d(x,t+lΔt;−lΔt)  59
• Assuming that a triangle having vertexes {gi(x1.t+lΔt; −lΔt), gi(x2.t+lΔt; −lΔt), gi(x3.t+lΔt; −lΔt)} is indicated by Wo, a triangle having vertexes {x1,x2,x3} by X, and a triangle having vertexes {g(x1.t+lΔt; −lΔt), g(x2.t+lΔt; −lΔt), g(x3.t+lΔt; −lΔt)} by Y.
• A pixel value of a point wεR²|w=VnεΛ²in the triangle W will be obtained. Affine transformation from the triangle W to the triangle X and affine transformation to the triangle Y are obtained by the equation 39, and a pixel value corresponding to the point w is obtained by each of the affine transformations. In other words, a point in the triangle X corresponding to the point w can be computed by the followingequation 60.
• 
  [x^T1]=[w^T1]T(W,X)  60
• Similarly, a point in the triangle corresponding to the point w can be computed according to the following equation 61.
• 
  [y^T1]=[w^T1]T(W,Y)  61
• When a weight average of both frames is computed according to the time position, a pixel value of an interpolation frame due to the backward matching mapping g(;t+lΔt; −lΔt) can be computed by the following equation 62.
• 
  s_b,p(w,t+lΔt_i)=(1−Δt_i)s_c(y,t)+Δt_is_c(x,t+lΔt)  62
• The interpolation frame due to the backward matching mapping g(;t+lΔt; −lΔt) is generated by doing this processing on all triangles in a screen.
• A backward occlusion reliability map on an interpolation frame is generated by a backward occlusion reliability map P(y|g(;t+lΔt;−lΔt) as described hereinafter. In other words, a backward occlusion reliability map on an interpolation frame due to the backward matching mapping g(;t+lΔt; −lΔt) is obtained by the following equation.
• 
  P_o(w,t+lΔt_i|g(•,t+lΔt;−lΔt))=P(y|g(•,t+lΔt;−lΔt))  63
• Similarly, a backward image correlation reliability map is generated by an image correlation reliability computation step. In other words, the backward image correlation reliability map due to the backward matching mapping g(;t+lΔt;−lΔt) is obtained by the following equation 64.
• 
  P(w,t+lΔt_i|g(•,t+lΔt;−lΔt),s(•,t),s(•,t+lΔt))=p(s_c(y,t),s_c(x,t+lΔt))  64
• Because the backward occlusion mapping g(;t; −lΔt) due to the backward occlusion matching corresponds to matching from the reference frame t to the reference frame t−lΔt, linear transformation defined by go(x,t;lΔt)=x−d(x,t;−lΔt) is subjected to the backward occlusion mapping to generate a mapping between the reference frame t+lΔt and the reference frame t. Since the direction of the mapping is reversed, it is written into +lΔt. When the backward occlusion mapping g (;t;−lΔt) is subjected to linear transform according to Δti, the backward occlusion mapping is converted into an image expressed by the equation 65.
• 
  g_i,o(x,t;lΔt)=x−Δt_id(x,t;−lΔt)  65
• where a triangle having vertexes {gi,o (x1.t;lΔt), gi,o(x2.t;lΔt), gi,o (x3.t; lΔt)} is indicated by Wo, a triangle having vertexes {x1,x2,x3} Xo, and a triangle having vertexes {go(x1.t;lΔt), go(x2.t;lΔt), go(x3.t;lΔt)} by Yo.
• A point in the triangle Xo which corresponds to the point w is computed by the following equation 66.
• 
  [x_o^T1]=[w^T1]T(W_o,X_o)  66
• Similarly, a point in the triangle Yo which corresponds to the point w is computed by the following equation 67.
• 
  [y_o^T1]=[w^T1]T(W_o,Y_o)  67
• If a weight average of both frames is computed according to the time position, a pixel value of an interpolation frame due to the backward occlusion mapping g (;t;lΔt) is computed by the following equation 68.
• 
  s_b,o,p(w,t+lΔt_i)=Δt_is_c(x_o,t)+(1−Δt_i)s_c(y_o,t+lΔt)  68
• An interpolation frame due to the backward occlusion mapping g(;t;lΔt) is generated by being done on all triangles in a screen. A backward image correlation reliability map is generated by an image correlation reliability computation step. In other words, a backward image correlation reliability map due to the backward occlusion mapping g (;t;lΔt) is generated based on the following equation 69.
• 
  P(w,t+lΔt_i|g(•,t;−lΔt),s(•,t),s(•,t+lΔt))=p(s_c(x_o,t),s_c(y_ot+lΔt))  69
• Because an interpolation frame due to the forward matching mapping q(;t;lΔt) and an interpolation frame due to the forward occlusion mapping g(;t+lΔt;lΔt), and an interpolation frame due to the backward matching mapping g(;+lΔt; −lΔt) and an interpolation frame due to the backward occlusion mapping g(;t; −lΔt) can be generated, these interpolation frames are combined by a reliability map.
• probability of reliability given by image correlation and probability of the reliability given by occlusion are independent from each other. Therefore, the probability that the forward occlusion matching is adopted can be expressed by the followingequation 70 according to multiplication theorem of probability.
• 
  P₁=P_o(w,t+lΔt_i|g(•,t;lΔt))P(w,t+lΔt_i|g(•,t+lΔt;lΔt),s(•,t),s(•,t+lΔt))  70
• The probability that the forward matching is adopted can be expressed by the following equation 71:
• 
  P₀={1−P_o(w,t+lΔt_i|g(•,t;lΔt))}P(w,t+lΔt_i|g(•,t;lΔt),s(•,t),s(•,t+lΔt))  71
• The probability that the backward occlusion matching is adopted can be expressed by the following equation 72.
• 
  P₃=P_o(w,t+lΔt_i|g(•,t+lΔt;−lΔt))P(w,t+lΔt_i|g(•,t;−lΔt),s(•,t),s(•,t+lΔt))  72
• The probability that the backward matching is adopted can be expressed by the following equation 73:
• 
  P₂={1−P_o(w,t+lΔt_i|g(•,t+lΔt;−lΔt))}P(w,t+lΔt_i|g(•,t+lΔt;−lΔt),s(•,t),s(•,t+lΔt))  73
• The transparency used for combination of the interpolation frames is computed from these probability based on the following equation 74:
• $\begin{array}{cc}{\alpha }_j=\frac{P_j}{\sum _{n=0}^3P_n},\left(j=0,1,2,3\right)& 74\end{array}$
• Using this transparency the interpolation frames are combined based on the reliability according to the following equation 75.
• 
  s_i,p(w,t+lΔt_i)=α₀s_f,p(w,t+lΔt_i)+α₁s_f,o,p(w,t+lΔt_i)+α₃s_b,p(w,t+lΔt_i)+α₃s_b,o,p(w,t+lΔt_i)  75
• The interpolation frame Si, p (w,t+lΔti) can be generated by doing on the whole screen the above-mentioned processing.
Fourth Embodiment
• An image matching apparatus of the present embodiment will be explained referring to the block diagram ofFIG. 25. The present embodiment provides an image mapping apparatus supplied with an image signal of a motion image and used for generating an interpolation frame between the frames of the image signal like the previous embodiments. The present embodiment provides an image matching apparatus using a method for interpolating an occlusion region adequately to generate an interpolation frame of high quality.
• According to the constitution shown inFIG. 25, the image signal is input to aframe memory41. Theframe memory41 is connected to aforward matching unit42 and abackward matching unit43. The outputs of the forward and backward matchingunits42 and43 are connected to animage combination unit44.
• Theforward matching unit42 comprises an image matching module42-1 and an occlusion matching module42-2 to each of which the input image signal and the stored image signal of theframe memory41 are input. Theforward matching unit42 further includes surface reliability calculators42-3,42-4,42-5 cascade-connected to the image matching module42-1 and an interpolation frame generator42-6.
• Thebackward matching unit43 comprises an image matching module43-1 and an occlusion matching module43-2 to each of which the input image signal and the stored image signal of theframe memory41 are input. Thebackward matching unit43 further includes surface reliability calculators43-3,43-4,43-5 cascade-connected to the image matching module43-1 and an interpolation frame generator42-6.
• In the configuration, the image signal of motion image is divided into a plurality of frames and then processed. In this time, each frame is stored in theframe memory41 and some frames can be accessed at the same time. In the present embodiment, a frame in the time t is referred to as a reference frame t, a frame in the time t+lΔt is referred to as a reference frame t+lΔt, a frame in the time t+2lΔt is referred to as reference frame t+2lΔt, and a frame in the time t+lΔt to as reference frame t+lΔt. An interpolation frame is interpolated between the reference frame t and the reference frame t+lΔt.
• An occlusion region must be detected adequately for the occlusion region to be interpolated adequately. The ground that can detect the occlusion region by the present embodiment adequately is explained referring toFIG. 26.FIG. 26 shows an example wherein a round object moves to the left to the right. On the reference frame t+lΔt, a uncovered occlusion region appears at a position before the round object moves as indicated in a circle ofFIG. 26. A lattice point of the rear part of the round object is noted, that is, the lattice points xn, xn+(1,0)T and yn, yn+(1,0)T are noted. Since the round object moves, if matching is subjected thereto, the interval between the lattice points yn, yn+(1,0)T should be expanded greatly. In other words, a spring between the lattice points yn, yn+(1,0)T stretches. As thus described when an object moves and thus a uncovered occlusion region appears, it is found that a spring thereon always expands. In other words, the part that the spring stretches greatly can be expected to be the uncovered occlusion region.
• The uncovered occlusion appears on the rear of the round object between the times t and t+lΔt as explained referring toFIG. 16. In this time, it is found that the spring expands. When the round object just moves downward, it is found that a part corresponding to the present uncovered occlusion region is appears in the time t+2lΔt. In other words, if the reference frames t+lΔt and t+2lΔt are subjected to matching, a region corresponding to the occlusion region can be found. This is referred to as occlusion matching.
• The stretch of the spring corresponds to the uncovered occlusion region as shown inFIG. 26. In this case, because the matching is carried out in a forward direction with respect to a time axis, it corresponds to the forward matching. However, the covered occlusion region of the front part of the object is not considered. When a covered occlusion region is watched in the backward matching as shown inFIG. 27, it is a uncovered occlusion region. Accordingly, it is possible to devise a countermeasure on both occlusions by combining the forward matching and the backward matching together.
• With the above it is found that the occlusion region is detected and a region corresponding to the occlusion region can be subjected to the matching.
• It is found that the uncovered occlusion region can be detected by the expansion of the spring. This means what can compute surface reliability. If the surface reliability can be computed not only by the expansion of the spring but also by plural information, it is thought that the uncovered occlusion region can be detected at higher precision by the joint probability of these reliabilities.
• The following characteristics should be noted.
• 1. An occlusion occurs before and after movement of the object. The occlusion appearing before movement of the object is covered occlusion and the occlusion appearing thereafter is uncovered occlusion.
• 2. In dynamic matching, there are geometric characteristics such that the spring stretches on the rear of the object and shortens on the front thereof.
• According to thecharacteristics 1 and 2, the uncovered occlusion can be detected by expansion of the spring, and the covered occlusion can be detected by the contraction thereof.FIG. 28 shows this state. It is described referring toFIG. 29 why the occlusion cannot be detected correctly only by detecting the uncovered occlusion by expansion of the spring.
• It is assumed that the matching is carried out with respect to the bidirection of the time axis to execute the forward matching and backward matching. In this time, one occlusion region is provided on the rear of the object by the forward matching, and the other occlusion region is provided on the front thereof by the backward matching. If these two occlusion regions are added, an ideal result as shown on the upper right ofFIG. 29 is provided. However, an occlusion map may not be always provided justly as shown on the under side ofFIG. 29. In that case, when the forward and backward occlusion regions are subjected to OR-gating simply, the object is missing resulting in deteriorating the image quality.
• In contrast, the case that the occlusion disappeared due to contraction of the spring is detected at the same time is considered. An example of this case is explained referred toFIG. 30. In this case, the matching is assumed to be carried out in bidirection with respect to the time axis. In this time, two occlusion maps corresponding to the expansion and contraction of the spring are obtained. The two occlusion maps can be provided from a result of the backward matching. Then, as shown in the under side ofFIG. 29, it may be said that the uncovered occlusion region (uncovered occlusion) due to the forward matching and the covered occlusion region (covered occlusion) due to the backward matching have the same meaning. In other words, the same occlusion region is watched from two viewpoints. These two maps are ones viewed from two viewpoints with respect to one event. The result of higher robust that is not depended upon a single result by obtaining the joint probability of these maps can be acquired. Accordingly, the joint probability of two maps is obtained as shown inFIG. 30 (AND), and the robust occlusion map can be provided by ORing two results.
• As shown inFIG. 31 even if the occlusion maps are not really provided justly similarly to the above, this method can provide a robust occlusion map. By obtaining reliability of the mapping as a result of matching, it is thought that more robust result can be obtained by applying the joint probability of the mapping.
• How to carry out matching is explained. A method for matching corresponding points from an source image to a reference image when the source image and reference image are input, that is, generating a relation between matching images is explained referring to a block circuit ofFIG. 32. InFIG. 32, the output of a discretevariable unit51 supplied with the reference image and source image is provided to aframe memory52, an image correlation potentialforce computation unit53, a elasticenergy computation unit54 and a frictionalforce computation unit55. A problem setting is done on this block circuit. The following modes are conceived as an source image.
• A Continuous Image Model
• $\begin{array}{cc}{s}_c\left(x,t\right),\left[\begin{array}{c}x\\ t\end{array}\right]\in \left(X\times T\right)⋐{\mathrm{<figure-callout\; label="R"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">R</figure-callout>}}^3& 76\end{array}$
• X⊂R²is a set expressing the whole image, and T⊂R is a set expressing a time to be intended.
• This is a continuous image model of a real number base. In the present embodiment, because a digital image is regarded as an source image, the following model obtained by sampling the above model is used.
• Sampling Image Model
• $\begin{array}{cc}{s}_p\left(x,t\right),\left[\begin{array}{c}x\\ t\end{array}\right]=\left[\begin{array}{cc}\mathrm{<figure-callout\; label="v"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">v</figure-callout>}& 0\\ 0& \Delta t\end{array}\right]\left[\begin{array}{c}n\\ k\end{array}\right]\in \left(N\times K\right)⋐{\mathrm{<figure-callout\; label="\Lambda "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\Lambda </figure-callout>}}^3& 77\end{array}$
• where N⊂Λ²is a lattice set of the whole image, K⊂Λ is a lattice set of a time to be intended, and V is a sampling matrix, and N and X are connected to each other. For example,
• $V=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$
• is used.
• The image matching is a problem to find which point of the reference image corresponds to a point on the source image In this time, the image matching problem is formulated as a problem to search for a motion vector d:X×T→X satisfying the following conditional expression.
• 
  s_p(x,t)=s_c(x+d(x,t;lΔt),t+lΔt)  78
• It should be noted that since a motion vector d is a real number, the right side of the equation uses notation of the continuous image model equation 77. Because the image matching between two images of the source image and reference image is thought about, the following equivalent problem is considered.
• 
  s_k1,p(x)=s_k2,c(x+d(x))  79
• where
• s_k1,p(x)=s_p(x,t)|_t=k1Δt
• s_k2,c(x)=s_c(x,t)|_t=k2Δt
• k₁,k₂εK are variables representing the source image and reference image respectively.
• The motion vector is simplified to be d:X→X. Since only one motion vector d may be decided with respect to the point x, the motion vector d is assumed to be a unique mapping. Since the image x+d(x) can be considered to be a mapping with respect to the point x, it is defined by the followingequation 80.
• 
  g(x)=x+d(x)  80
• where it is a unique mapping of g:X→X. The above image matching problem results in a problem for searching for the mapping g satisfying the following equation 81.
• Image Matching Problem
• 
  s_k1,p(x)=s_k2,c(g(x))  81
• where a point determined by the mapping g is defined by the following equation 82.
• 
  y=g(x)  82
• where yεX.
• Because x=Vn, the point x corresponds uniquely to a point n on the lattice space. Because the mapping g is a unique mapping, the point y corresponds uniquely to the point x, too. Therefore, the point y corresponds uniquely to the point n. This is expressed byFIG. 5. In other words, the space which will be dealt with here is a transformation lattice space corresponding one-to-one to the point n on the lattice space.
• Because y=g(x)=g(Vn) as described above, this is redefined by the following equation in order to make clearly understandable to correspond one-to-one to the point n.
• 
  y_n=g(x)  83
• Then, the image matching problem (equation 3) result in a problem searching for yn satisfying the next expression.
• Image Matching Problem
• 
  s_k1,p(x)=s_k2,c(y_n)  84
• Measure for solving the problem is explained. Energy function is defined by the following equation to solve the equation 84 of the image matching problem.
• Energy Function
• 
  U(y_n)=E_u(y_n)+E_k(y_n)  85
• The equation 84 of the image matching problem is formulated to the following optimization problem as the minimization problem of the energy function of the whole screen.
• Optimization Problem
• $\begin{array}{cc}\underset{\left\{y_an\in N\right\}}{\min }\sum _{n\in N}U\left(y_n\right)& 86\end{array}$
• where Eu is assumed to be an image energy, and Ek to be an elastic energy. This is used in Snakes [“M. Kass, A. Witkin and D. Terzopoulos, “Snakes: Active Contour Models”, International Journal of Computer Vision, 1, 4, pp. 321-331, 1988”], Active Net [“Sakaue, Yamamoto”, Model Active Net of a dynamic network and Application to Region extraction, Television studies magazine, 45, 10, pp. 1155-1163, 1991”] and CPF [Y. Shinagawa and T. L. Kunii, “Unconstrained Automatic Image Matching Using Multiresolutional Critical-Point Filters”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 20, No. 9, 1998”].
• Eu is explained as follows. The equation 84 is transformed into the following equation.
• 
  s_k2,c(y_n)−s_k1,p(x)=0  87
• It is usually difficult by noise components included in an image to search for a point to satisfy the equation 87 strictly. Accordingly, the following energy function is conceived.
• $\begin{array}{cc}{E}_u\left(n\right)=\frac{1}{2}{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_n\left(T\right)\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(x\right)\right)}^2& 88\end{array}$
• A point that this energy function Eu becomes minimum is searched for. Ek is explained as follows. The image to be subjected to matching is assumed to be one obtained by projecting a three-dimensional space to a two-dimensional area. When an object on the three-dimensional space is assumed to be a rigid body, the surface of the rigid body is observed as an object on the two-dimensional image. If an object on the three-dimensional space is observed with the source image and reference image, the probability that the phase of the object observed on each image is kept is high. As shown inFIG. 7, the positional relation of points x on the source image should be kept on the point yn on the reference image object point, too. It is convenient that the point yn seems to be a net formed of springs for such a characteristic to be reproduced. In other words, it is preferable that the elastic energy is introduced.
• Therefore, the energy minimization equation 86 due to theequation 85 of energy function is to search for solution such that the equation 84 is satisfied as much as possible and the whole image becomes smooth.
• The optimality requirement of the following equation is found from the equation 86 in Snakes and Active Net.
• Optimality Requirement
• $\begin{array}{cc}\frac{\partial U\left(y_n\right)}{\partial y_n}=0& 89\end{array}$
• A solution is obtained by solving this. This provides a condition of static balance to satisfy the energy function (equation 5).
• This method builds up a dynamic solution method by introducing kinetic energy T. A dynamic behavior of the point y is interested, so that the function yn(i) is defined by introducing a new time axis τεR to the point y. Because the new time axis is introduced, a differentiation concerning a time can be defined as follows.
• $\begin{array}{cc}\frac{}{\tau }y_n\left(T\right)={\stackrel{.}{y}}_n\left(T\right),\frac{{\mathrm{<figure-callout\; label="d"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">d</figure-callout>}}^2}{{\mathrm{<figure-callout\; label="\tau "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\tau </figure-callout>}}^2}y_n\left(\tau \right)={\ddot{y}}_n\left(T\right)& 90\end{array}$
• The kinetic energy is defined as the following expression.
• Kinetic Energy
• $\begin{array}{cc}T\left({\stackrel{.}{y}}_n\left(T\right)\right)=\frac{1}{2}{{\stackrel{.}{y}}_n\left(T\right)}^T{\stackrel{.}{y}}_n\left(T\right)& 91\end{array}$
• A Lagrangian is defined as follows.
• Lagrangian
• 
  L(y_n(T),{dot over (y)}_n(T))=T({dot over (y)}_n(T))−U(y_n(T))  92
• An equation of motion to satisfy a variation principle then is provided by the following equation 93.
• Euler Equation
• $\begin{array}{cc}\frac{}{\tau }\frac{\partial L}{\partial {\stackrel{.}{y}}_n}-\frac{\partial L}{\partial y_n}=0& 93\end{array}$
• When this is developed, the equation of motion of the following expression is provided.
• An Equation of Motion
• $\begin{array}{cc}{\ddot{y}}_n\left(T\right)=-\frac{\partial U\left(y_n\right)}{\partial y_n}& 94\end{array}$
• However, the motion according to the equation of motion 94 is a conservative system, and a solution does not converge by an energy conservation law. Therefore, the friction energy of the following equation 95 is introduced to make the system a non-conservative system.
• Friction Energy
• $\begin{array}{cc}\Phi \left({\stackrel{.}{y}}_n\left(T\right)\right)=\frac{1}{2}\mu {{\stackrel{.}{y}}_n\left(T\right)}^T{\stackrel{.}{y}}_n\left(T\right)& 95\end{array}$
• where μ>0 is a coefficient of dynamic friction. The Euler equation when the friction energy is introduced is expressed as follows.
• The Euler Equation Including the Friction Energy.
• $\begin{array}{cc}\frac{}{T}\frac{\partial L}{\partial {\stackrel{.}{y}}_n}-\frac{\partial L}{\partial y_n}+\frac{\partial \Phi }{\partial {\stackrel{.}{y}}_n}=0& 96\end{array}$
• The equation of motion 97 is provided by solving this equation 96.
• The equation of motion including friction.
• $\begin{array}{cc}{\ddot{y}}_n\left(T\right)=-\frac{\partial U\left(y_n\right)}{\partial y_n}-\mu {\stackrel{.}{y}}_n\left(T\right)& 97\end{array}$
• These static energy function and dynamic energy function are intuitively compared with each other as follows. In optimization according to the static energy function as shown inFIG. 33, a search is done for acquiring a position of a static balance. Therefore, a position of balance which is basically the nearest to an initial value is obtained. In other words, the local optimal point which is the nearest to the initial value is obtained. However, the local optimal point is not always a global optimal point as shown inFIG. 33. As thus described, the optimization due to the static energy function accompanies strong initial value dependence.
• In contrast, in the dynamic optimization of the present embodiment a search is done in a dynamic motion. More specifically, the search is expressed as motion. Therefore, the first potential energy is converted into the kinetic energy as shown inFIG. 34 so that the search can jump out the local optimal point. As a result, a wider range than used of the static energy function can be searched. Therefore, in comparison with use of the static energy function, the present method can lower the initial value dependence and have robustness against influence of noise, etc.
• It will be proved that the dynamic search, that is, a solution according to the equation of motion (equation 15) satisfies an optimality requirement condition for an optimization problem (equation 6).
• 1. The search is assumed to be completed. Then, the point y stops completely, that is, the following equation 98 is established.
• 
  {dot over (y)}=0,ÿ=0  98
• Then, the following equation is established by the equation of motion 97.
• $\begin{array}{cc}0=-\frac{\partial U\left(y\right)}{\partial y}-\mu ·0& 99\end{array}$
• Thus, the following equation 100 is established.
• $\begin{array}{cc}0=\frac{\partial U\left(y\right)}{\partial y}& 100\end{array}$
• Therefore, the optimality requirement equation 89 is satisfied. When the search is completed, the point is optimum.
• 2. The point y is assumed to be stationary. Then, the following equation 101 is established.
• 
  {dot over (y)}=0,ÿ≠0  101
• Then, the following equation 102 is obtained by the equation of motion 97.
• $\begin{array}{cc}\ddot{y}=-\frac{\partial U\left(y\right)}{\partial y}& 102\end{array}$
• If the point y is an optimal point now, the following equation 103 is established by the optimality requirement equation 89, and the search is completed.
• 
  ÿ=0  103
• If the point y is not an optimal point, the search is not completed. In other words, the search is not completed except for the optimal point.
• 3. The following equation is established by a property of friction energy.
• $\begin{array}{cc}\underset{t\to \infty }{\lim }T\left(\stackrel{.}{y}\right)=0& 104\end{array}$
• In other words, the motion stops. Thus, the search is completed sometime.
• The motion according to the equation of motion (equation 15) satisfies a requirement condition of optimality for the optimization problem equation 86 by theconditions 1, 2 and 3 as mentioned above.
• The secondary optimality requirement is not satisfied partly. The reason is because the search may be finished on the top of potential hill. Therefore, if the condition that the search is not completed on the top of potential hill, that is, the following equation 105 is assumed, the secondary optimality requirement is proved, too.
• $\begin{array}{cc}\frac{{\partial }^2U\left(y\right)}{\partial {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">y</figure-callout>}}^2}<0\to \stackrel{.}{y}\ne 0,\ddot{y}\ne 0& 105\end{array}$
• This additional assumption can be achieved by requesting for the next condition for an algorithm.
• “if
• $\frac{{\partial }^2U\left(y\right)}{\partial {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">y</figure-callout>}}^2}<0$
• in {dot over (y)}=0,ÿ=0, random perturbation is given.”
• As described above, the search according to the equation of motion 97 provides a solution for the optimization problem equation 86, and suggests to have a robust in comparison with a conventional static search.
• The structure of
• $\frac{\partial U\left(y_n\right)}{\partial y_n}$
• will be conceived successively.
• Since this can be developed like the following equation 106, this equation will be discussed.
• $\begin{array}{cc}\frac{\partial U\left(y_n\right)}{\partial y_n}=\frac{\partial E_u\left(y_n\right)}{\partial y_n}+\frac{\partial E_k\left(y_n\right)}{\partial y_n}& 106\end{array}$
• $\frac{\partial E_u\left(y_n\right)}{\partial y_n}$
• represents a force of a reverse direction with respect to a gradienta gradient of the point yn(T) of image energy. The following equation is given to make that it is a force clear.
• $F_u\left(n\right)=-\frac{\partial E_u\left(y_n\right)}{\partial y_n}$
• Various methods are conceived to compute the gradient, but the following method is adopted. As shown in theequation 7, the gradient is obtained by local optimization directly. The image model Sc is a continuous image model, but in practice only the sampled image model Sp can be used. Thus, the local optimization is done based on the sampled image model, too.
• Because the sampling point which is at the nearest to yn(T) should be a local space center yc, it is obtained by the following equation 107.
• $\begin{array}{cc}{y}_c=\lfloor y_n\left(T\right)+\left[\begin{array}{c}0.5\\ 0.5\end{array}\right]\rfloor .& 107\end{array}$
• ^└•┘ is an operator to be converted to an integer (omission of fractions).
• When the adjacency space is defined by the following equation 108, a local search set Q can be defined by the following equation 109.
• $\begin{array}{cc}L=\left\{\begin{array}{c}\left[\begin{array}{c}-1\\ -1\end{array}\right],\left[\begin{array}{c}0\\ -1\end{array}\right],\left[\begin{array}{c}1\\ -1\end{array}\right],\left[\begin{array}{c}-1\\ 0\end{array}\right],\\ \left[\begin{array}{c}0\\ 0\end{array}\right],\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}-1\\ 1\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right],\left[\begin{array}{c}1\\ 1\end{array}\right]\end{array}\right\}& 108\\ \Omega =\left\{y_n\left(T\right),y_c+1|\forall 1\in L\right\}& 109\end{array}$
• The gradient is subjected to the local optimization, and a vector in the gradient direction is obtained. The vector is normalized and multiplied by the size of the gradient. As a result, the following equation is provided.
• A Force Due to the Image Energy (Square Error Energy)
• $\begin{array}{cc}{F}_u\left(n,y·\left(T\right)\right)=\frac{{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_{\min }\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_n\left(T\right)\right)\right)}^2}{u_d}u_du_d=y_{\min }-y_n\left(T\right)y_{\min }=\arg \underset{y\in \Omega }{\min }{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,p}\left(y\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(x\right)\right)}^2& 110\end{array}$
• In view of handling easiness in mounting, the equation 112 of the force due to the image energy that the energy function is defined by the following equation 111 can be used.
• 
  E_u(n)=|s_k2,c(y_n(T))−s_k1,p(x)|  111
• Force Due to the Image Energy (Absolute Value Difference Error Energy)
• $\begin{array}{cc}{F}_u\left(n\right)=\frac{{s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_{\min }\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_n\left(T\right)\right)}^2}{u_d}u_du_d=y_{\min }-y_n\left(T\right)y_{\min }=\arg \underset{y\in \Omega }{\min }s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,p}\left(y\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(x\right)& 112\end{array}$
• $\frac{\partial E_k\left(y_n\right)}{\partial y_n}$
• represents a force due to the gradient in a point y_n(T) of elastic energy.
• The following equation is given to make that it is a force clear.
• The elastic body is assumed to be configured by a manifold M. A point on the manifold M is assumed to be expressed by the following equation 113.
• $\begin{array}{cc}y=\left[\begin{array}{c}u\\ v\end{array}\right]\in {\mathrm{<figure-callout\; label="R"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">R</figure-callout>}}^2& 113\end{array}$
• The manifold is assumed to have coordinate neighborhood. In this time, partial differentiation is defined by the equation 114.
• $\begin{array}{cc}{y}_p={\left(\frac{\partial u}{\partial p},\frac{\partial v}{\partial p}\right)}^Ty_q={\left(\frac{\partial u}{\partial p},\frac{\partial v}{\partial q}\right)}^T& 114\end{array}$
• The primary differential term of the elastic energy is expressed by the following equation 115.
• $\begin{array}{cc}{E}_k=\frac{1}{2}k\left({y_p}^2+{y_q}^2\right)& 115\end{array}$
• This is energy to express the shortening of the elastic member. This is differentiated by y as shown by the following equation 116.
• $\begin{array}{cc}\begin{array}{c}\frac{\partial E_k}{\partial y}=k\left(-y_p\frac{\partial }{\partial y}\left(\frac{\partial y}{\partial p}\right)-y_q\frac{\partial }{\partial y}\left(\frac{\partial y}{\partial q}\right)\right)\\ =-k\left(\frac{\partial }{\partial p}\left(\frac{\partial y}{\partial p}\right)+\frac{\partial }{\partial q}\left(\frac{\partial y}{\partial q}\right)\right)\\ =-k\left(y_{\mathrm{pp}}+y_{\mathrm{qq}}\right)\end{array}\mathrm{where}& 116\\ y_{\mathrm{pp}}={\left(\frac{{\partial }^2u}{\partial {\mathrm{<figure-callout\; label="p"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">p</figure-callout>}}^2},\frac{{\partial }^2v}{\partial p^2}\right)}^Ty_{\mathrm{qq}}={\left(\frac{{\partial }^2u}{\partial {\mathrm{<figure-callout\; label="q"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">q</figure-callout>}}^2},\frac{{\partial }^2v}{\partial q^2}\right)}^T& 117\end{array}$
• The manifold M is assumed to be realized by the spring model coupling lattice points, and the partial differentiation is approximated by the difference equation.
• 
  y_pp≅y_n+(−1,0)_T−2y_n+y_n+(1,0)_T
• 
  y_qq≅y_n+(−1,0)_T−2y_n+y_n+(1,0)_T  118
• If this is substituted for the equation 117 and arranged, 4-point connection spring model shown by the following equation 119 is provided.
• 4-Point Connection Spring Models
• $\begin{array}{cc}{F}_k\left(n,y·\left(\tau \right)\right)=k\left(y_{n+\left[\begin{array}{c}0\\ -1\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)+k\left(y_{n+\left[\begin{array}{c}-1\\ 0\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)+k\left(y_{n+\left[\begin{array}{c}1\\ 0\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)+k\left(y_{n+\left[\begin{array}{c}0\\ 1\end{array}\right]}\left(T\right)-y_n\left(T\right)\right)& 119\end{array}$
• It is natural that an initial condition is expressed by the following equation 120, if y_n(0)|∀nεN is assumed to be equal to a square lattice and the kinetic energy is 0, that is, the speed is 0.
• $\begin{array}{cc}\left[\begin{array}{c}{y}_n\left(0\right)\\ {\stackrel{.}{y}}_n\left(0\right)\end{array}\right]=\left[\begin{array}{c}x\\ 0\end{array}\right]& 120\end{array}$
• When the above matters are summarized, the equation of motion is expressed by the following equation.
• Equation of Motion
• $\begin{array}{cc}{\ddot{y}}_n\left(T\right)=F_u\left(n\right)+F_k\left(n\right)-\mu {\stackrel{.}{y}}_n\left(T\right)\left[\begin{array}{c}{y}_n\left(0\right)\\ {\stackrel{.}{y}}_n\left(0\right)\end{array}\right]=\left[\begin{array}{c}x\\ 0\end{array}\right],\forall n\in N& 121\end{array}$
• Since the force Fu due to the image energy is not solved in analysis, the normal differential equation (equation 19) is not solved in analysis. Therefore, it is difficult to take the limit inτ→∞ of the system.
• For this reason, an enough time necessary for converging the system is considered, and the convergence state of the system is estimated by computing t=(0,T) interval by numeric analysis.
• If the initial value is determined, the normal differential equation provides a solution uniquely by a discrete variable method. Generally, it is said an initial value problem of the normal differential equation. There are many numerical solutions for solving this problem, for example, an Euler method, a Runge Kutta method, a Bulirsch-Soer method, a predictor-corrector method, a hidden Runge Kutta method as famous methods. The Runge Kutta method is most famous and has a high use frequent. However, the following equation 121 is hard to fit a complicated numerical solution since it has a dimension for the size of an image. Therefore, it is thought that the Euler method that realization is most simplified is applied.
• Since the Euler method is a numerical solution for a normal differential equation of one time, at first, theequation 19 is converted to the ordinary differential equation of one time. When transformation of variable is carried out by the following equation 122, a transformed equation of motion 123 is provided.
• Transformation of Variable
• $\begin{array}{cc}\{\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}\left(T\right)=y_n\left(T\right)\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(T\right)={\stackrel{.}{y}}_n\left(T\right)\end{array}& 122\end{array}$
• Transformed Equation of Motion
• $\begin{array}{cc}\left[\begin{array}{c}{\stackrel{.}{\psi }}_{1,n}\left(T\right)\\ {\stackrel{.}{\psi }}_{2,n}T\end{array}\right]=\left[\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(T\right)\\ F_u\left(n\right)+F_k\left(n\right)-\mu {\psi }_{2,n}\left(T\right)\end{array}\right]\left[\begin{array}{c}{\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{1,n}\left(0\right)\\ {\mathrm{<figure-callout\; label="\psi "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\psi </figure-callout>}}_{2,n}\left(0\right)\end{array}\right]=\left[\begin{array}{c}\mathrm{Vn}\\ 0\end{array}\right],\forall n\in N& 123\end{array}$
• where ψ_1,n(T),ψ_2,n(T) indicate a position and rate of yn(T) respectively.
• A scheme of the Euler method for the normal differential equation 124 is represented by the following equation 125.
• [Equation 122]]
• 
  {dot over (x)}(t)=F(x(t),t)  124
• A Scheme of Euler Method
• 
  x⁽ⁿ⁺¹⁾=x⁽ⁿ⁾+hF(x⁽ⁿ⁾,t)  125
• This progresses a solution from t⁽ⁿ⁾to t⁽ⁿ⁺¹⁾≡t⁽ⁿ⁾+h.
• x(n) shows a step and h indicates a step size. When the scheme of the Euler method is applied to the equation 123, an updated equation of the following equation 126 is provided.
• Update Equation Due to Euler Method
• The update equation is solved repeatedly, so that a convergence state can be obtained.
• The interpolation frame generation step is explained. This step adopts a method similar to Digital Image Warping1 [“J. Nieweglowski, P. Haavisto, “Motion Compensated Video Sequence Interpolation Using Digital Image Warping”, IEEE Acoustics, Speech, and Signal Processing, vol. 5, p 205-208, 1994”, “J. Nieweglowski, T. George Campbell, P. Haavisto, “A Novel Video Coding Sheme Based on Temporal Prediction Using Digital Image Warping”, IEEE Transactions on, Volume: 39 Issue: 3, Aug. 1993”].
• In the Digital Image Warping, a motion vector is obtained for each point of a rectangle, the rectangle is divided into two triangles, and affine transformation is computed on the basis of three vertexes for each triangle. A triangular image to be interpolated by the affine transformation is obtained.
• The affine transformation from a triangle having vertexes V={(u_i,v_i)^T,(i=0, 1, 2)} to a triangle having vertexes Y={(x_i,y_i)^T,(i=0, 1, 2)} by Digital Image Warping can be expressed by the following equation 126.
• Affine Transformation
• $\begin{array}{cc}T\left(V,Y\right)=\frac{1}{\det \left(U\right)}B\left[\begin{array}{ccc}{x}_0& {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">y</figure-callout>}}_0& 1\\ x_1& {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ x_2& {\mathrm{<figure-callout\; label="y"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\end{array}\right]& 126\\ B=\left[\begin{array}{ccc}{v}_1-v_2& v_2-v_0& v_0-v_1\\ u_2-u_1& u_0-u_2& u_1-u_0\\ {\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_1v_2-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_2v_1& {\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_2v_0-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">u</figure-callout>}}_0v_2& {\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">u</figure-callout>}}_0v_1-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_1v_0\end{array}\right]\det \left(U\right)={\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">u</figure-callout>}}_0\left(v_1-v_2\right)+{\mathrm{<figure-callout\; label="v"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">v</figure-callout>}}_0\left(u_2-u_1\right)+\left({\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_1v_2-{\mathrm{<figure-callout\; label="u"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">u</figure-callout>}}_2v_1\right)& 127\end{array}$
• It can be computed using the affine transformation that the point (u,v)^Tin the triangular V corresponds to the point (x,y)^Tin the triangle expressed by the following equation 128.
• 
  [x,y,1]=[u,v,1]T(V,Y)  128
• An interpolation frame is obtained by the affine transformation. The time position of the reference frame t is set to 0, the time position of the reference frame t+lΔt is set to 1, and the time position of the interpolation frame to be interpolated is assumed to be Δt_i(0{tilde under (<)}Δt_i{tilde under (<)}1). When the mapping g(,t;lΔt) is subjected to the linear transformation according to Δti, the following equation 129 is established.
• 
  g_i(x,t;lΔt)=x+Δt_id(x,t;lΔt)  129
• The interpolation frame is generated as shown inFIG. 20. Assuming that a triangle having vertexes {g_i(x,t;lΔt),g_i(x₂,t;lΔt),g_i(x₃,t;lΔt)} is W, a triangle having vertexes {x₁,x₂,x₃} is X, and a triangle having vertexes {g(x₁,t;lΔt),g(x₂,t;lΔt),g(x₃,t;lΔt)} is Y. A pixel value of a point within a triangle on the interpolation frame will be obtained. An affine transformation from the triangular W to the triangular X and an affine transformation to the triangular Y are obtained by theequation 24, and a pixel value corresponding to the point w is derived from each affine transformation. In other words, the point within the triangular x corresponding to the point w can be computed by the following equation 130.
• 
  [x^T1]=[w^T1]T(W,X)  130
• Similarly, the point within the triangle Y corresponding to the point w can compute by the following equation 131.
• 
  [y^T1]=[w^T1]T(W,Y)  131
• If a weight average of both frames is computed according to the time position, a pixel value of the interpolation frame can be computed by the following equation 132.
• Interpolation Frame Due to a Mapping g(•,t;lΔt)
• 
  s_p(w,t+lΔt_i)=Δt_is_c(x,t)+(1−Δt_i)s_c(y,t+lΔt)  132
• The right side of the equation 132 is written in a form of a continuous image model. Therefore, a pixel value is computed by being interpolated by a bi-linear method to compute the right side from the discrete image model. An interpolation frame based on the mapping g(•,t;lΔt) is generated by doing this processing for all triangles in a screen.
• The step of thesurface reliability computation1 is explained. The part which the spring expands is assumed to be a region having a high reliability as occlusion. On the contrary, the part which the spring does not change has a high reliability as the surface.
• In practice, since the reliability in the region must be computed, the area of a triangle surrounded by the region expressed by, for example, the following equation as shown inFIG. 17 is used.
• 
  y_n,y_n+(0,−1)T,y_{n+(−1,−1)T}
• The area of the original triangle is derived from the absolute value of outer product of a vector according to the following equation 133.
• $\begin{array}{cc}{\mathrm{<figure-callout\; label="S"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">S</figure-callout>}}_1\left(n\right)=\frac{1}{2}\left(x_{n+{\left(0,-1\right)}^T}-x_n\right)\times \left(x_{n+{\left(-1,-1\right)}^T}-x_n\right)& 133\end{array}$
• Similarly, the area after mapping is obtained by the following equation 134.
• $\begin{array}{cc}{\mathrm{<figure-callout\; label="S"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">S</figure-callout>}}_2\left(n\right)=\frac{1}{2}\left(y_{n+{\left(0,-1\right)}^T}-y_n\right)\times \left(y_{n+{\left(-1,-1\right)}^T}-y_n\right)& 134\end{array}$
• Therefore, the differential value of the area is assumed to be expressed by the following equation 135.
• 
  R(n)=S₂(n)−S₁(n)  135
• Assuming that the probability of surface enhances as the differential value nears to 0, and the probability of surface lowers as the differential value increases. In this case, it is convenient that if the differential value is greater than a threshold, it is set to 0, and if the differential value is smaller than the threshold, it is set to 1.
• The step of thesurface reliability computation2 is explained.
• The lattice point of the covered occlusion region (covered occlusion region) has a geometric characteristic to be shortened to 0 ideally. This is symmetrical with that the uncovered occlusion region (uncovered occlusion region) stretches.
• The probability of reliability given by the uncovered occlusion region is associated with the expansion of the spring, so that it will be natural that the probability of reliability given by the covered occlusion region is associated with the contraction of the spring. In other words, the expansion and contraction of the spring are dealt with equivalently till now. Hereinafter, the different confidence probabilities generate according to the expansion and contraction of the spring. This seems to be natural from the symmetry of a theory
• Using the difference of area used in the step of thesurface reliability computation1, if the difference is greater than the threshold, it is set to 0, and if the difference is smaller than the threshold, it is set to 1. The step of thesurface reliability computation3 is explained hereinafter.
• If the reliability of the mapping is positive, it is probability to seem to be reliable as the surface. This is modeled as probability of the image Sk2 under a state that the mapping g(x) is provided to the lattice point x. Generally, this is called likelihood.
• Assuming that a noise is added to an observed image, the pixel values of the lattice point x and mapping g(x) are expressed by the following equation 136.
• 
  g₁=s_k1(x)+ε₁
• 
  g₂=s_k2(g(x)+ε₂  136
• where ε₁,ε₂are additive noises.
• Displaced Pixel Difference (referred to as DPD hereinafter) is modeled as shown by the following equation 137.
• $\begin{array}{cc}\begin{array}{c}r\left(g\left(x\right),x\right)=g_2-{\mathrm{<figure-callout\; label="g"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">g</figure-callout>}}_1\\ =s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2}\left(g\left(x\right)\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1}\left(x\right)+\left({\varepsilon }_2-{\varepsilon }_1\right)\\ =s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2}\left(g\left(x\right)\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1}\left(x\right)+\varepsilon \end{array}& 137\end{array}$
• If this DPD is smaller than the threshold, the mapping is probable. Accordingly, the probability is set to 1. If DPD is greater than the threshold, the probability is set to 1. A configuration is explained referring toFIG. 25.
• Because the forward matching (42) and the backward matching (43) differ only in an input frame, the matching step with occlusion is summed up as follows. In the case of the forward matching, the input frame needs not re-read, but in the backward matching, the input frame is re-read as indicated by the following equation:
• 
  t+lΔt→t
• 
  t→t+lΔt
• 
  t−lΔt→t+2lΔt
• A subscript f is added to an output result in the forward matching, and a subscript b is added to the output result in the backward matching.
• Matching Step with Occlusion (FIG. 35)
• The matching g(Vn,t;lΔt),∀nεN is derived from t

  

  t+lΔt by the matching step (later described) (S11).
• 2. The occlusion matching g(Vn,t+lΔt;lΔt),∀nεN is derived from t+lΔt

  

  t+2lΔt by an occlusion matching step (the same as the matching step) (S12).
• A surface reliability map is derived from g(Vn,t;lΔt),∀nεN by the step of the surface reliability computation1 (S13).
• (a) The area S₁(n) of a quadrilateral Quad(Vn,V(n+(1,0)^T),V(n+(1,1)^T),V(n+(0,1)^T)) is obtained, the area S₂(n) of a quadrilateral Y=Quad(g(Vn,t;lΔt),g(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt),g(V(n+(0,1)^T),t;lΔt)) is obtained, and a difference R(n)=S₂(n)−S₁(n) between the areas is computed.
• (b) The surface probability inside the quadrilateral is obtained with respect to a threshold T1.
• ${\mathrm{Pr}}_o=\{\begin{array}{cc}1& R\left(n\right)<{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">T</figure-callout>}}_1\\ 0& R\left(n\right)>{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">T</figure-callout>}}_1\end{array}$
• (c) A result is stored in the surface reliability map.
• 
  Pr_o(y)=Pr_o,∀yε{Y∩Vn′|∀n′εN}
• (e) The above-mentioned processing is carried out for the whole screen**.
• 2. A mapping reliability map is derived from g(Vn,t;lΔt),∀nεN by the step of the surface reliability computation3 (S14).
• (a) The probability of the mapping is obtained with respect to a threshold T3.
• ${\mathrm{Pr}}_g=\{\begin{array}{cc}1& r\left(n\right)<{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">T</figure-callout>}}_3\\ 0& r\left(n\right)>{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">T</figure-callout>}}_3\end{array}r\left(n\right)=s_p\left(\mathrm{Vn},t\right)-s_c\left(g\left(\mathrm{Vn},t;l\Delta t\right),t+l\Delta t\right)$
• (b) A result is stored in the mapping reliability map.
• 
  Pr_g(Vn)=Pr_g
• (c) The above-mentioned processing is carried out for the whole screen ∀nεN.
• 3. A mapping reliability map is derived from g(Vn,t+lΔt;lΔt),∀nεN by a step of the surface reliability computation3 (S15).
• (a) The probability of the mapping is obtained with respect to the threshold T3.
• ${\mathrm{Pr}}_g=\{\begin{array}{cc}1& r\left(n\right)<{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">T</figure-callout>}}_3\\ 0& r\left(n\right)>{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">T</figure-callout>}}_3\end{array}r\left(n\right)=s_p\left(\mathrm{Vn},t+l\Delta t\right)-s_c\left(g\left(\mathrm{Vn},t+l\Delta t;l\Delta t\right),t+2l\Delta t\right)$
• (b) A result is stored to a mapping reliability map.
• 
  Pr_go(Vn)=Pr_g
• (c) The above-mentioned processing is done for the whole screen ∀nεN.
• 4. A covered surface reliability map is derived from g(Vn,t;lΔt),∀nεN by the step of the surface reliability computation (S16).
• (a) The area S₁(n) of a quadrilateral X=Quad(Vn,V(n+(1,0)^T),V(n+(1,1)^T),V(n+(0,1)^T)) is obtained, the area S₂(n) of a quadrilateral Quad(g(Vn,t;lΔt),g(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt),g(V(n+(0,1)^T),t;lΔt)) is obtained, and a difference R(n)=S₂(n)−S₁(n) between the areas is computed.
• (b) The surface probability inside the quadrilateral is obtained with respect to a threshold T2.
• ${\mathrm{Pr}}_{\mathrm{co}}=\{\begin{array}{cc}1& R\left(n\right)>{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">T</figure-callout>}}_2\\ 0& R\left(n\right)<{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">T</figure-callout>}}_2\end{array}$
• (c) A result is stored in the covered surface reliability map.
• 
  Pr_co(x)=Pr_co,∀xε{X∩Vn′|∀n′εN}
• (d) The above-mentioned processing is done for the whole screen ∀nεN.
• 5. g_i(x,t;lΔt)=x+Δt_id(x,t;lΔt) is obtained for ∀nεN by the interpolation frame generation step (described later), and an interpolation frame s_p(•,t+lΔt_i) is generated (S17).
• 6. g_i,o(x,t+lΔt;−lΔt)=x−(1−Δt_i)d(x,t+lΔt;lΔt) is obtained for ∀nεN by the interpolation frame generation step (described later), and an interpolation frame s_p,o(•,t+lΔt_i) is generated (S18).
• 7. Surface reliability map Pr_o,i(•) of lΔt_iis generated.
• Covered surface reliability map Pr_co,i(•) of lΔt_iis generated.
• Mapping reliability map Pr_g,i(•) of lΔt_iis generated.
• Mapping reliability map Pr_go,i(•) of lΔt_iis generated. (S19)
• (a) The following triangles are obtained.
• 
  X=Tri(Vn,V(n+(1,0)^T),V(n+(1,1)^T))
• 
  W=Tri(g_i(Vn,t;lΔt),g_i(V(n+(1,0)^T),t;lΔt),g_i(V(n+(1,1)^T),t;lΔt))
• 
  Y=Tri(g(Vn,t;lΔt),g_i(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt))
• (b) Affine transformation T(W,Y),T(W,X) is obtained.
• (c) [y^T1]=[w^T1]T(W,Y) is obtained for ∀Wε{W∩Vn′|∀n′εN} and stored in Pr_o,i(w)=Pr_o(y).
• [x^T1]=[w^T1]T(W,X) is obtained and stored in Pr_g,i(w)=Pr_g(x).
• It is stored in Pr_go,i(w)=Pr_go(y).
• It is stored in Pr_co,i(w)=Pr_co(x).
• (d) The above-mentioned processing is done for the whole screen ∀nεN.
• A combination step is explained.
• 8. combination is carries out to generate an interpolation frame s_p,c(•,t+lΔt_i).
• (a) The probabilities of the surface side and the occlusion side are obtained as probability of the product event (S20).
• 
  Pr₁=Pr_f,co,i(Vn)Pr_f,o,i(Vn)Pr_f,g,i(Vn)
• 
  Pr₂=(1−Pr_b,co,i(Vn))Pr_f,co,i(Vn)(1−Pr_f,o,i(Vn))Pr_f,go,i(Vn)
• 
  Pr₃=Pr_b,co,i(Vn)Pr_b,o,i(Vn)Pr_b,g,i(Vn)
• 
  Pr₄=(1−Pr_f,co,i(Vn))Pr_b,co,i(Vn)(1−Pr_b,o,i(Vn))Pr_b,go,i(Vn)
• (b) An alpha blend coefficient for composition is lost as follows.
• $\begin{array}{cc}{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_1=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_2=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_3=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_3}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_4=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_4}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}s_{p,c}\left(\mathrm{Vn},t+l\Delta t_i\right)={\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_1s_{p,f}\left(\mathrm{Vn},t+l\Delta t_i\right)+{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_2s_{p,f,o}\left(\mathrm{Vn},t+l\Delta t_i\right){\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_3s_{p,b}\left(\mathrm{Vn},t+l\Delta t_i\right)+{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_4s_{p,b,o}\left(\mathrm{Vn},t+l\Delta t_i\right)& \left(c\right)\end{array}$
• (d) The above-mentioned processing is done for the whole screen ∀nεN.
• Each step is described concretely according to the block diagram shown inFIG. 32.
• In a discrete variable method step, a position and a speed are updated according to an update equation 126 of the discrete variable method, and a computation is done from the initial value to a predetermined time. In this embodiment, a discrete variable method based on Euler method is used. Concretely, computation is performed according to a flow shown inFIG. 36. An image force computation step, an elastic energy force computation step, and a frictional force computation step are used.
• In the image force computation step, a force Fu occurring by the potential energy due to the image correlation is computed.
• If a position and mapping n,ψ_1,•⁽ⁿ⁾in a certain step are input, Fu is computed according to the equation 110 by the following equation 138.
• $\begin{array}{cc}{F}_u\left(n\right)=\frac{{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left(y_{\min }\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,c}\left({\psi }_{l,n}^{\left(n\right)}\right)\right)}^2}{u_d}u_du_d=y_{\min }-{\psi }_{l,n}^{\left(n\right)}y_{\min }=\arg \underset{y\in \Omega }{\min }{\left(s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2,p}\left(y\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1,p}\left(\mathrm{Vn}\right)\right)}^2\Omega =\left\{{\psi }_{l,n}^{\left(n\right)},y_c+1\forall 1\in L\right\}y_c=\lfloor {\psi }_{l,n}^{\left(n\right)}+\left[\begin{array}{c}0.5\\ 0.5\end{array}\right]\rfloor L=\left\{\left[\begin{array}{c}-1\\ -1\end{array}\right],\left[\begin{array}{c}0\\ -1\end{array}\right],\left[\begin{array}{c}1\\ -1\end{array}\right],\left[\begin{array}{c}-1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 0\end{array}\right],\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}-1\\ 1\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right],\left[\begin{array}{c}1\\ 1\end{array}\right]\right\}& 138\end{array}$
• Concretely, computation is carried out according to a flow chart shown inFIG. 37. This computation may be carried out according to the equation 112. In the elastic energy force computation step, a force Fk occurring by the elastic energy is computed.
• If a position and mapping n,ψ_1,•⁽ⁿ⁾in a certain step are input, Fk is computed according to theequation 18 by the following equation 139.
• $\begin{array}{cc}{F}_k\left(n\right)=k\left({\psi }_{1,n+\left[\begin{array}{c}0\\ -1\end{array}\right]}^{\left(n\right)}-{\psi }_{1,n}^{\left(n\right)}\right)+k\left({\psi }_{1,n+\left[\begin{array}{c}-1\\ 0\end{array}\right]}^{\left(n\right)}-{\psi }_{1,n}^{\left(n\right)}\right)+k\left({\psi }_{1,n+\left[\begin{array}{c}1\\ 0\end{array}\right]}^{\left(n\right)}-{\psi }_{1,n}^{\left(n\right)}\right)+k\left({\psi }_{1,n+\left[\begin{array}{c}0\\ 1\end{array}\right]}^{\left(n\right)}-{\psi }_{1,n}^{\left(n\right)}\right)& 139\end{array}$
• Concretely, computation is carried out according to a flow shown inFIG. 38.
• In the frictional force computation step, the force F_μ occurring by dynamical friction is computed.
• When a speed ψ_2,n⁽ⁿ⁾in a certain step is input, Fu is computed by the following equation 140.
• 
  F_μ(ψ_2,n⁽ⁿ⁾)=−μψ_2,n⁽ⁿ⁾  140
• Concretely, computation is carried out according to a flow shown inFIG. 39.
• In an interpolation frame generation step, an interpolation image of a 0≦Δt_i≦1 position is generated by the obtained mapping. Computation is carried out according to a flow ofFIG. 40 concretely.
• The present embodiment can provide a robust and high-quality interpolation frame and interpolate an appropriate image in an occlusion region as mentioned above.
Fifth Embodiment
• The present embodiment is explained referring to the block diagram ofFIG. 25. In the fourth embodiment, randomization with respect to the threshold is done in surfacereliability computation steps1,2,3. In this case, if the surface reliability is greater than a threshold, the probability is set at 1, and if it is not more than the threshold, the probability is set at 0. In other words, the probability can be expressed by the binary only. For this reason, the present embodiment uses the following sigmoid function in order to provide a more smooth probability expression of high precision.
• $\begin{array}{cc}\mathrm{Pr}\left(x\right)=\frac{1}{1+\exp \left(-\alpha \left(x-\beta \right)\right)}& 141\end{array}$
• where α indicates a gradient and β indicates a center. For example, α=30.0 and β=0.25 can be used. The following part is changed from the third embodiment.
• The Step of theSurface Reliability Computation1.
• (a) The area S₁(n) of quadrilateral Quad(Vn,V(n+(1,0)^T),V(n+(1,1)^T),V(n+(0,1)^T)) and the area S₂(n) of quadrilateral Y=Quad(g(Vn,t;lΔt),g(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt),g(V(n+(0,1)^T),t;lΔt)) are obtained, and a difference R(n)=S₂(n)−S₁(n) between the areas is computed.
• (b) The surface probability of the quadrilateral inside is obtained by the next expression.
• ${\mathrm{Pr}}_o=\frac{1}{1+\exp \left({\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_1\left(R\left(n\right)-{\beta }_1\right)\right)}$
• (c) A result is stored in the surface reliability map as follow:
• 
  Pr_o(y)=Pr_o,∀yε{Y∩Vn′|∀n′εN}
• (d) The above processing is done about the whole screen ∀nεN.
• A mapping reliability map is obtained from g(Vn,t;lΔt),∀nεN by the step ofsurface reliability computation3.
• (a) The probability of mapping is obtained according to the following equation:
• ${\mathrm{Pr}}_g=\frac{1}{1+\exp \left({\alpha }_3\left(r\left(n\right)-{\beta }_3\right)\right)}$$r\left(n\right)=s_p\left(\mathrm{Vn},t\right)-s_c\left(g\left(\mathrm{Vn},t;l\Delta t\right),t+l\Delta t\right)$
• (b) A result is stored in the mapping reliability map as shown by the following equation.
• 
  Pr_g(Vn)=Pr_g
• (c) The above processing is done about the whole screen ∀nεN.
• (d) A mapping reliability map is obtained from g(Vn,t+lΔt;lΔt),∀nεN by the step ofsurface reliability computation3.
• (e) The probability of mapping is obtained with respect to the threshold T3 as represented by the following equation:
• ${\mathrm{Pr}}_g=\frac{1}{1+\exp \left({\alpha }_3\left(r\left(n\right)-{\beta }_3\right)\right)}$$r\left(n\right)=s_p\left(\mathrm{Vn},t+l\Delta t\right)-s_c\left(g\left(\mathrm{Vn},t+l\Delta t;l\Delta t\right),t+2l\Delta t\right)$
• (f) The result is stored in the mapping reliability map as represented by the following equation.
• 
  Pr_go(Vn)=Pr_g
• (g) The above processing is done about the whole screen ∀nεN.
• Step ofSurface Reliability Computation2.
• (a) The area S₁(n) of quadrilateral X=Quad(Vn,V(n+(1,0)^T),V(n+(1,1)^T),V(n+(0,1)^T)) is obtained and the area S₂(n) of quadrilateral Quad(g(Vn,t;lΔt),g(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt),g(V(n+(0,1)^T),t;lΔt)) is obtained. A difference between these areas is computed.
• (b) Surface probability of the quadrilateral inside is obtained with respect to the threshold T2 by the following equation.
• ${\mathrm{Pr}}_{\mathrm{co}}=\{\begin{array}{cc}1& R\left(n\right)>{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">T</figure-callout>}}_2\\ 0& R\left(n\right)<{\mathrm{<figure-callout\; label="T"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">T</figure-callout>}}_2\end{array}$
• (c) A result is stored in a covered surface reliability map.
• 
  Pr_co(x)=Pr_co,∀xε{X∩Vn′|∀n′εN}
• (e) The above-mentioned processing is done about the whole screen ∀nεN.
• The present embodiment can provide an interpolation frame of higher image quality because the more smooth probability map is provided.
Sixth Embodiment
• The present embodiment is explained referring to the block diagram ofFIG. 25.
• In the present embodiment, the more robust surface reliability is obtained by describing the probability of surface reliability as a probability distribution according to a Gaussian distribution.
• About the Step ofSurface Reliability Computation1.
• The part that a spring expands is assumed to be a region having a high reliability as occlusion, in other words, the part which the spring does not change has a high reliability as the surface.
• Actually, an area of the triangle surrounded by, for example, three points y_n, y_n+(0,−1)T, y_{n+(−1,−1)T}as shown inFIG. 17 are used, because reliability in the region must be computed. An area of an original triangle is obtained by an absolute value of outer product of a vector as follows;
• $S_1\left(n\right)=\frac{1}{2}\left(x_{n+{\left(0,-1\right)}^T}-x_n\right)\times \left(x_{n+{\left(-1,-1\right)}^T}-x_n\right)$
• Similarly, the area of the triangle after mapping is obtained as follows:
• $S_2\left(n\right)=\frac{1}{2}\left(y_{n+{\left(0,-1\right)}^T}-y_n\right)\times \left(y_{n+{\left(-1,-1\right)}^T}-y_n\right)$
• Therefore, a differential value between the areas is computed by the following equation.
• 
  R(n)=S₂(n)−S₁(n)
• Assuming that the surface probability increases as the differential value draws closer to 0, and the surface probability decreases as the differential value increases. If the differential value is 0 in average and accords to the Gaussian distribution of dispersion σ², it is proper. Therefore, the probability density function of the differential value R is modeled as follows.
• $p\left(R\right)=\frac{1}{\sqrt{2{\mathrm{<figure-callout\; label="\pi \sigma "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\pi \sigma </figure-callout>}}^2}}\exp \left(-\frac{{\mathrm{<figure-callout\; label="R"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">R</figure-callout>}}^2}{2{\sigma }^2}\right)$
• Because this is a probability density function, a distribution function is expressed by the following equation:
• 
  P(x)=∫_∞^xp(v)dv
• Therefore, the probability that the differential value R(n) is between a and b is given by the following equation:
• 
  Pr(a≦R(n)<b)=P(b)−P(a)
• If the reliability when the differential value R(n)=0 is provided will be 95%, the following equation may be concluded.
• 
  Pr(−1.96σ≦R(n)<1.96σ)=P(1.96σ)−P(−1.96σ)=0.95
• Accordingly, the surface probability when a certain differential value is given is defined by the following equation 142:
• Surface Probability
• 
  Pr(R(n)−1.96σ≦R(n)<R(n)+1.96σ)=P(R(n)+1.96σ)−P(R(n)−1.96σ)  142
• Because this is the reliability of the surface inside the triangle, this is assigned to the inside of the triangle and stored in a memory.
• Because when the triangle y_n, y_n+(0,−1)T, y_{n+(−1,−1)T}having the lattice point n as a reference is assumed to be Y, the lattice point inside this triangle can be expressed by yε{Y∩Vn′|∀n′εN}. Accordingly, the reliability map of the surface is built up using this lattice point as follows.
• 
  Pr_o(y)=Pr(R(n)−1.96σ≦R(n)<R(n)+1.96σ),∀yε{Y∩Vn′|Vn′εN}
• Surface reliability computation3 is described. The reliability of mapping is modeled as probability.
• This is modeled as plausibility of the image S_k2(.) under the state that the mapping g(x) is given to the lattice point x. Generally, this is called likelihood.
• Assuming that an observed image contains a noise, pixel values of the lattice point x and mapping g(x) are expressed by the following equations:
• 
  g₁=s_k1(X)+ε₁
• 
  g₂=s_k2(g(X))+ε₂
• where ε1, ε2 indicate additivity noises.
• Displaced Pixel Difference (referred to as DPD hereinafter) is modeled as follows.
• $\begin{array}{c}r\left(g\left(x\right),x\right)=g_2-{\mathrm{<figure-callout\; label="g"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">g</figure-callout>}}_1\\ =s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2}\left(g\left(x\right)\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1}\left(x\right)+\left({\varepsilon }_2-{\varepsilon }_1\right)\\ =s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2}\left(g\left(x\right)\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1}\left(x\right)+\varepsilon \end{array}$
• Assuming that the observed noise is 0 in average and accords to the Gaussian distribution of dispersion σ2, likelihood of a certain DPDx can be described as a probability density function (pdf) as expressed by the following equation 143.
Likelihood
• $\begin{array}{cc}p\left(x\right)={\left(2{\mathrm{\pi \sigma }}^2\right)}^{-\frac{1}{2}}\exp \left\{-\frac{x^2}{2{\sigma }^2}\right\}& 143\end{array}$
• Because this is a probability density function, the distribution function is described by the following equation.
• 
  P(x)=∫_∞^xp(v)dv
• Therefore, a probability that the DPD r(g(x),x) exists between a and b is determined by the following equation.
• 
  Pr(a≦r(g(x),x)<b)=P(b)−P(a)
• If in true mapping {tilde over (g)}(x) the differential value between corresponding pixels becomes zero, DPD in {tilde over (g)}(x), that is, the true DPD{tilde over (r)}({tilde over (g)}(x),x) is {tilde over (r)}({tilde over (g)}(x),x)=ε. In other words, the true DPD{tilde over (r)}({tilde over (g)}(x),x) is equal to a noise component. Therefore, {tilde over (r)}({tilde over (g)}(x),x) is to be 0 in average and accord to the Gaussian distribution of dispersion σ². The reliability in the true DPD{tilde over (r)}({tilde over (g)}(x),x) is set by the confidence interval of the Gaussian distribution.
• For example, when the reliability of 95% will be given to the true DPD{tilde over (r)}({tilde over (g)}(x),x), it may be expressed by the following equation.
• 
  Pr(−1.96σ≦{tilde over (r)}(g(x),x)<1.96σ)=P(1.96σ)−P(−1.96σ)=0.95
• Therefore, the probability of the mapping g(x) when DPD is provided may be defined by the following equation 144.
• Probability of Mapping g(x)
• 
  Pr(r(g(x),x)−1.96σ≦r(g(x),x)<r(g(x),x)+1.96σ)=P(r(g(x),x)+1.96σ)−P(r(g(x),x)−1.96σ)  144
• Surface reliability computation2 is described.
• Covered occlusion (covered occlusion) is a transformed and compressed region.
• The quadrilateral is defined as follows:
• 
  Quad₁=Quad(x_n,x_n+(1,0)T,x_n+(1,1)T,x_n+(0,1)T)
• The transformed quadrilateral is defined as follows
• 
  Quad₂=Quad(y_n,y_n+(1,0)T,y_n+(1,1)T,y_n+(0,1)T)
• The difference between these areas is represented by the following equation.
• 
  R(n)=S(Quad₂)−S(Quad₁)
• where S represents an operator for computing an area. If this differential value r(n) is a negative, the occlusion is compressed.
• If the whole screen is a single area, the distribution of differential value R(n) looks good according to the Gaussian distribution. If the screen comprises a plurality of regions, and the covered occlusion occurs, the differential value R(n) between the regions may be distributed as outlier. Therefore, the probability that it is the surface builds reliability to be high so that the differential value R(n) is almost zero.
• Probability density function of differential value R is modeled as follows.
• $p\left(R\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{{\mathrm{<figure-callout\; label="R"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">R</figure-callout>}}^2}{2{\sigma }^2}\right)$
• Because this is a probability density function, a distribution function is expressed by the following equation:
• 
  P(x)=∫_∞^xp(v)dv
• Therefore, the probability that the differential value R(n) is between b and a is represented by the following equation:
• 
  Pr(a≦R(n)<b)=P(b)−P(a)
• If the reliability when the differential value R(n)=0 is obtained will be 95%, the following equation is preferable.
• 
  Pr(−1.96σ≦R(n)<1.96σ)=P(1.96σ)−P(−1.96σ)=0.95
• Surface probability when the differential value is obtained is defined by the following equation 145.
• Surface Probability
• 
  Pr(R(n)−1.96σ≦R(n)<R(n)+1.96σ)=P(R(n)+1.96σ)−P(R(n)−1.96σ)  145
• Real constitution seems to become not more than.
• One Surface Reliability Computation Step
• (a) An area S₁(n) of the quadrilateral Quad(Vn,V(n+(1,0)^T),V(n+(1,1)^T),V(n+(0,1)^T)) and an area S₂(n) of the quadrilateral Y=Quad(g(Vn,t;lΔt),g(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt),g(V(n+(0,1)^T),t;lΔt)) are obtained. A difference R(n)=S₂(n)−S₁(n) between these areas is computed.
• (b) (a sample average μ of R(n),∀nεN and a sample variance a σ2 are computed from g(Vn,t;lΔt),∀nεN)
• (c) Surface probability of the quadrilateral inside is obtained.
• ${\mathrm{Pr}}_o=P\left(R\left(n\right)+1.96\sigma \right)-P\left(R\left(n\right)-1.96\sigma \right)$$P\left(x\right)={\int }_{-\infty }^xp\left(v\right)v$$p\left(R\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{{\left(R-\mu \right)}^2}{2{\sigma }^2}\right)$
• (d) A result is stored in the surface reliability map.
• 
  Pr_o(y)=Pr_o,∀yε{Y∩Vn′|∀n′εN}
• (e) The above processing is done about the whole screen ∀nεN.
• Surface Trust Computation3
• (a) Probability of mapping is obtained.
• ${\mathrm{Pr}}_g=P\left(r\left(n\right)+1.96\sigma \right)-P\left(r\left(n\right)-1.96\sigma \right)$$P\left(x\right)={\int }_{-\infty }^xp\left(v\right)v$$p\left(r\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{{\mathrm{<figure-callout\; label="r"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">r</figure-callout>}}^2}{2{\sigma }^2}\right)$$r\left(n\right)=s_p\left(\mathrm{Vn},t\right)-s_c\left(g\left(\mathrm{Vn},t;l\Delta t\right),t+l\Delta t\right)$
• (b) A result is stored in the mapping reliability map.
• 
  Pr_g(Vn)=Pr_g
• (c) The above processing is done about the whole screen ∀nεN.
• A mapping reliability map is obtained from g(Vn,t+lΔt;lΔt),∀nεN by thesurface reliability computation3.
• (a) Probability of representation is found.
• ${\mathrm{Pr}}_g=P\left(r\left(n\right)+1.96\sigma \right)-P\left(r\left(n\right)-1.96\sigma \right)$$P\left(x\right)={\int }_{-\infty }^xp\left(v\right)v$$p\left(r\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{{\mathrm{<figure-callout\; label="r"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">r</figure-callout>}}^2}{2{\sigma }^2}\right)$$r\left(n\right)=s_p\left(\mathrm{Vn},t+l\Delta t\right)-s_c\left(g\left(\mathrm{Vn},t+l\Delta t;l\Delta t\right),t+2l\Delta t\right)$
• (b) A result is stored in the mapping reliability map.
• 
  Pr_go(Vn)=Pr_g
• (c) The above processing is done about the whole screen ∀nεN.
• Surface Reliability Computation2
• (a) An area S₁(n) of the quadrilateral X=Quad(Vn,V(n+(1,0)^T),V(n+(1,1)^T),V(n+(0,1)^T)) and an area S₂(n) of the quadrilateral Quad(g(Vn,t;lΔt),g(V(n+(1,0)^T),t;lΔt),g(V(n+(1,1)^T),t;lΔt),g(V(n+(0,1)^T),t;lΔt)). A difference R(n)=S₂(n)−S₁(n) between the areas is computed.
• (b) Surface probability of the quadrilateral inside is obtained with respect to the threshold T2.
• (c) A result is stored in the covered surface reliability map.
• 
  Pr_co(x)=Pr_co,∀xε{X∩Vn′|∀n′εN}
• (d) The above processing is done about the whole screen ∀nεN.
• As described above, according to the present embodiment, the probability of the surface reliability is described as a probability distribution according to the Gaussian distribution, whereby the more robust surface reliability can be obtained and the interpolation frame of higher image quality more can be provided.
Seventh Embodiment
• The present embodiment is explained referring to the block diagram ofFIG. 25.
• In the case of the image wherein the background remains stationary, a part that an occlusion region can occur is only a part which changes between frames. The part which changes between frames can be detected by detecting a difference between the frames. Therefore, the surface reliability can be computed from the difference between the frames.
• Surface reliability computation4 is explained. An observed image is assumed to contain a noise, and a pixel value between frames is expressed by the following equation.
• 
  g₁=s_k1(x)+ε₁
• 
  g₂=s_k2(x)+ε₂
• where ε1, ε2 each are an additivity noise.
• Pixel Difference (referred to as PD hereinafter) is modeled as follows.
• $\begin{array}{c}r\left(x\right)=g_2-{\mathrm{<figure-callout\; label="g"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">g</figure-callout>}}_1\\ =s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2}\left(x\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1}\left(x\right)+\left({\varepsilon }_2-{\varepsilon }_1\right)\\ =s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_2}\left(x\right)-s_{{\mathrm{<figure-callout\; label="k"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">k</figure-callout>}}_1}\left(x\right)+\varepsilon \end{array}$
• If this PD is smaller than a threshold, the probability can be set to 1, and if it is greater than the threshold, it can be set to 0.
• The modeling is not the above threshold modeling but may be the following method according to the Gaussian distribution model. Assuming that an observed noise is 0 in average, and accords to the Gaussian distribution of dispersion σ2, likelihood of a certain PDx can be described as a probability density function (pdf) by the following equation 146.
• Likelihood.
• $\begin{array}{cc}p\left(x\right)={\left(2\pi {\sigma }^2\right)}^{-\frac{1}{2}}\exp \left\{-\frac{x^2}{2{\sigma }^2}\right\}& 146\end{array}$
• Because this is a probability density function, the distribution function is expressed by the following equation.
• $P\left(x\right)={\int }_{-\infty }^xp\left(v\right)v$
• Therefore, the probability that PDγ(x) is between a and b is expressed by the following equation.
• 
  Pr(a≦r(x)<b)=P(b)−P(a)
• Assuming that a differential value between corresponding pixels becomes zero in a stationary region, the stationary PD is equal to a noise component. Therefore, the stationary PD is 0 in average and accords to the Gaussian distribution of dispersion σ2 from definition of a noise. The reliability in the stationary PD is set by a confidence interval of the Gaussian distribution. For example, if the reliability of 95% will be given to the stationary PD, the following equation may be established.
• 
  Pr(−1.96σ≦r(x)<1.96σ)=P(1.96σ)−P(−1.96σ)=0.95
• Accordingly, the surface probability when a certain PD is provided is defined by the following equation 147.
• Surface Probability
• 
  Pr(r(x)−1.96σ≦r(x)<r(x)+1.96σ)=P(r(x)+1.96σ)−P(r(x)−1.96σ)  147
• The dispersion σ2 may set to a given value, and it can be obtained repeatedly from an image given by an M estimation.
• Therefore, thesurface reliability computation4 is configured as follows.
• Surface Trust Computation4
• (d) Probability of the stationary surface is computed.
• ${\mathrm{Pr}}_{\mathrm{so}}=P\left(r\left(n\right)+1.96\sigma \right)-P\left(r\left(n\right)-1.96\sigma \right)$$P\left(x\right)={\int }_{-\infty }^xp\left(v\right)v$$p\left(r\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{{\mathrm{<figure-callout\; label="r"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">r</figure-callout>}}^2}{2{\sigma }^2}\right)$$r\left(n\right)=s_p\left(\mathrm{Vn},t+l\Delta t\right)-s_p\left(\mathrm{Vn},t\right)$
• (e) A result is stored in a stationary surface reliability map.
• 
  Pr_so(Vn)=Pr_so
• (f) The above processing is done about the whole screen ∀nεN.
• If the background completely stands still, it may use as it is without occlusion matching.
• A combination step is improved for the fourth embodiment.
• Combination is performed to generate an interpolation frame s_p,c(•,t+lΔt_i).
• (a) Probabilities on the surface side and the occlusion side are obtained as a probability of product event.
• 
  Pr₁=Pr_so(Vn)Pr_f,co,i(Vn)Pr_f,o,i(Vn)Pr_f,g,i(Vn)
• 
  Pr₂=(1−Pr_so(Vn))(1−Pr_b,co,i(Vn))Pr_f,co,i(Vn)(1−Pr_f,o,i(Vn))Pr_f,go,i(Vn)
• 
  Pr₃=Pr_so(Vn)Pr_b,co,i(Vn)Pr_b,o,i(Vn)Pr_b,g,i(Vn)
• 
  Pr₄=(1−Pr_so(Vn))(1−Pr_f,co,i(Vn))Pr_b,co,i(Vn)(1−Pr_b,o,i(Vn))Pr_b,go,i(Vn)
• (b) An alpha blend coefficient for combination is determined by the following equation.
• $\begin{array}{cc}<figure-callout\; label="\; \alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}"></figure-callout>{\alpha }_{}{}_1=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{\alpha }_2=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{\alpha }_3=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_3}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}{\alpha }_4=\frac{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_4}{{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_1+{\mathrm{<figure-callout\; label="Pr"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">Pr</figure-callout>}}_2}s_{p,c}\left(\mathrm{Vn},t+l\Delta t_i\right)={\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_1s_{p,f}\left(\mathrm{Vn},t+l\Delta t_i\right)+{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_2s_{p,f,o}\left(\mathrm{Vn},t+l\Delta t_i\right){\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_3s_{p,b}\left(\mathrm{Vn},t+l\Delta t_i\right)+{\mathrm{<figure-callout\; label="\alpha "\; filenames="US20090067728A1-20090312-D00002.png,US20090067728A1-20090312-D00004.png"\; state="\{\{state\}\}">\alpha </figure-callout>}}_4s_{p,b,o}\left(\mathrm{Vn},t+l\Delta t_i\right)& \left(c\right)\end{array}$
• (d) The above processing is done about the whole screen ∀nεN.
• According to the present embodiment, there is provided a high-quality interpolation frame in the case that the background stands still as mentioned above.
Eighth Embodiment
• The embodiment may accord to the Gaussian distribution as in the sixth embodiment in the case that the whole screen is a single area, but needs not always accord to the Gaussian distribution in the case that it comprises a plurality of areas. For this reason, the precision may fall. Accordingly, in the present embodiment, a precision is maintained for a plurality of regions by flipping the region coming off from the Gaussian distribution by M estimation.
• Covered occlusion is assumed to be a transformed and compressed region.
• A quadrilateral is defined as Quad₁=Quad(x_n,x_n+(1,0)T,x_n+(1,1)T,x_n+(0,1)T) and a transformed quadrilateral as Quad₂=Quad (y_n,y_n+(1,0)T,y_n+(1,1)T,y_n+(0,1)T).
• The difference between the areas of these quadrilateral is assumed to be R(n)=S(Quad₂)−S(Quad₁).
• S indicates an operator for computing the areas. If this differential value R(n) is negative, the region is compressed.
• If the whole screen is a single region, the distribution of differential value R(n) is seemed to be good according to the Gaussian distribution. If the screen is formed of a plurality of regions and the covered occlusion occurs, the possibility that the differential value R(n) between the regions is distributed as an outlier is high. Therefore, it is possible to built up the reliability that the probability that the region is the surface increases as the differential value R(n) nears zero.
• The probability density function of the differential value R is modeled as follows.
• $p\left(R\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{{\mathrm{<figure-callout\; label="R"\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">R</figure-callout>}}^2}{2{\sigma }^2}\right)$
• Because this is a probability density function, the distribution function is expressed by the following equation.
• $P\left(x\right)={\int }_{-\infty }^xp\left(v\right)v$
• Therefore, the probability that the differential value R(n) exists between a and b is given by the following equation.
• 
  Pr(a≦R(n)<b)=P(b)−P(a)
• If the reliability when the differential value R(n)=0 is provided will be 95%, the following equation may be established.
• 
  Pr(−1.96σ≦R(n)<1.96σ)=P(1.96σ)−P(−1.96σ)=0.95
• Accordingly, the surface probability when the differential value is acquired is defined by the following equation 148.
• The surface probability is defined by the following equation.
• 
  Pr(R(n)−1.96σ≦R(n)<R(n)+1.96σ)=P(R(n)+1.96σ)−P(R(n)−1.96σ)  148
• This surface probability is one obtained when the whole screen is a single region.
• When the probability which is not an outlier is assumed to be w:N→R:n

  

  w(n), the surface probability in the case of a plurality of regions can be defined by the following equation 149.
• 
  Pr(R(n)−1.96σ≦R(n)<R(n)+1.96σ)=w(n)(P(R(n)+1.96σ)−P(R(n)−1.96σ))  149
• This probability w can be associated with the weight function of the M estimation. The weight function of the M estimation nears 1 as the probability nears to an average, and nears zero as the probability falls.
• A modeling will be done within a frame of the M estimation.
• Because it is desired to discuss on the covered occlusion, this modeling is intended for only a compression direction. In other words, the expanded region should be ignored, the modeling is intended for a point of R(n)<1 (R(n)≧1 represents the expanded region).
• The surface on the covered occlusion is an average of the Gaussian distribution. In other words, a gap between the model and the differential value R(n) can be defined by the following equation.
• $z\left(n\right)=\frac{R\left(n\right)-\stackrel{\_}{R}}{\sigma }$
• According to the TuKey's Biweight method, the weight function can be expressed by the following equation.
• $w\left(n\right)=\{\begin{array}{cc}\left(1-\left(\frac{{z\left(n\right)}^2}{c^2}\right)\right)& z\left(n\right)<\mathrm{<figure-callout\; label="c"\; filenames="US20090067728A1-20090312-D00003.png,US20090067728A1-20090312-D00005.png"\; state="\{\{state\}\}">c</figure-callout>}\\ 0& z\left(n\right)\ge c\end{array}$
• where c>0 is a constant, and c=6.0 in the case of normal distribution. A sample average and a specimen deviation can be computed using the weight function by the following equation.
• ${\stackrel{\_}{R}}^{\left(i\right)}=\frac{1}{\sum _{n\in N}w\left(n\right)}\sum _{n\in N}w\left(n\right)R\left(n\right)$${\sigma }^{\left(i\right)}=\sqrt{\frac{1}{\sum _{n\in N}w\left(n\right)}\sum _{n\in N}w\left(n\right)\left(R\left(n\right)-{\stackrel{\_}{R}}^{\left(i\right)}\right)}$
• where the superscript expresses iteration. The M estimation is done by iterating a computation of the sample average & deviation and a computation of the weight function alternately once. When the probability is developed using a result of this M estimation, the modeling can be carried out as follows.
• $\mathrm{Pr}=\{\begin{array}{cc}1& z\left(n\right)>0\\ \mathrm{Pr}\left(R\left(n\right)-1.96{\sigma }^{\left(I\right)}\le R\left(n\right)<R\left(n\right)+1.96{\sigma }^{\left(I\right)}\right)& z\left(n\right)\le 0\end{array}\mathrm{Pr}\left(\begin{array}{c}R\left(n\right)-1.96{\sigma }^{\left(I\right)}\le \\ R\left(n\right)<R\left(n\right)+1.96{\sigma }^{\left(I\right)}\end{array}\right)=w\left(n\right)\left(\begin{array}{c}P\left(R\left(n\right)+1.96{\sigma }^{\left(I\right)}\right)-\\ P\left(R\left(n\right)-1.96{\sigma }^{\left(I\right)}\right)\end{array}\right)P\left(x\right)={\int }_{-\infty }^xp\left(v\right)vp\left(R\right)=\frac{1}{\sqrt{2\pi {\sigma }^{2\left(I\right)}}}\exp \left(-\frac{{\left(R-{\stackrel{\_}{R}}^{\left(I\right)}\right)}^2}{2{\mathrm{<figure-callout\; label="\sigma "\; filenames="US20090067728A1-20090312-D00001.png,US20090067728A1-20090312-D00002.png"\; state="\{\{state\}\}">\sigma </figure-callout>}}^{2\left(I\right)}}\right)$
• As described above, the present embodiment can provide an interpolation frame with a good precision even if the screen is formed of a plurality of regions.
Ninth Embodiment
• The present embodiment is explained referring to the block diagram ofFIG. 25, too. The above embodiment provides a method of detecting an occlusion region by the stretch and shrinkage of a spring. If the matching is done precisely, the stretch and shrinkage of the spring can be determined as a occlusion region. However, if the matching is done incorrectly, the stretch and shrinkage of the spring may cause a matching error. The present embodiment provides a method of generating a high-quality interpolation frame by detecting a matching error and compensating it.
• FIG. 41 shows a block diagram of an image matching apparatus enforcing picture matching method according to the ninth embodiment. According to this, aframe memory41 for storing an input image signal is provided. Thisframe memory41 is coupled with aforward matching unit42 and abackward matching unit43 to the stored image signal to these matching units. The forward and backward matchingunits42 and43 are supplied with the input image signal, too, to carry out a matching process based on both signals.
• The outputs of the forward and backward matchingunits42 and43 are connected to surfacereliability computation units45 and46 respectively. The outputs of the surfacereliability computation units45 and46 each are connected to thematching unit47 and thecombination unit48.
• Assuming that the image wherein an object moves from the left to the right is subjected to matching as shown inFIG. 42. In this time, if the matching is correctly done, the lattice of the object should move to a corresponding object. However, if the object cannot be detected due to a matching error, the lattice point on the object is located on the background of the reference image. In this situation, the lattice point is very unstable because the image energy is high. Therefore, the lattice point moves to a location of a lower image energy on its circumference. As a result, the stretch of the spring is observed there.
• The similar action occurs even if the matching is carried out in an opposite direction. Therefore, the regions which the forward and backward matching could not be carried out are detected as the expansion and shrinkage of the spring respectively. In other words, it is very likely that the region which could not be detected is contraction with being combined with the occlusion region. The matching error can be compensated by matching the occlusion regions together again.
• Asurface reliability computation4 is explained.
• A step corresponding to theforward matching unit42 is assumed to be a first surface reliability computation step and a step corresponding to thebackward matching unit43 is assumed to be a second surface reliability computation step. At first a complement of the surface reliability map of the surfacereliability computation step1 is computed to obtain a back side reliability. Alternatively, a product probability of a surface reliability map of the firstreliability computation step1 and that of the surfacereliability computation step2 may be obtained to acquire a complement thereof.
• A region that the back side reliability is more than a threshold (0.5, for example) is detected as the back side. These regions are assumed to be the first back side and the second back side respectively. A frame obtained by masking the reference frame t with the second back side is assumed to be a mask reference frame t, and a frame obtained by masking the reference frame t+lΔt with the second side is assumed to be a mask reference frame t+lΔt.
• The matching is done using the mask reference frame t as an source image and the mask reference frame t+lΔt as a reference image to produce a forward matching image. The matching is done using the mask reference frame t+lΔt as a source image and the mask reference frame t as a reference image to produce a backward matching image. For example, the reliabilities of the forward matching image and backward matching image are obtained using the step of thesurface reliability computation3 to obtain a surface reliability at the time of matching error.
• By doing combination as in the fourth embodiment together with the result of thesurface reliability computation4, a higher-quality interpolation frame can be generated with the matching error being compensated well.
• According to the present invention, there can be produced an image wherein even if the image is smoothed off, expression of discontinuity is not lost, and even if discontinuity is expressed, the smoothness is obtained.
• Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

Claims (25)

1-5. (canceled)

6. An image interpolation method for interpolating an interpolation image between a first reference image and a second reference image using the first reference image, the second reference image and a third reference image which are arranged in time series, the method comprising:

computing, using a gradient of an image correlation potential energy obtained by a position of each first lattice point of a plurality of first lattice points of a first lattice on a first reference image and pixel information of the each first lattice point and a position of each second lattice point of a plurality of second lattice points of a second lattice on a second reference image, which correspond to each of the first lattice points, and pixel data of the each second lattice point, potential force applied to the each second lattice point;

computing elasticity force applied to the each second lattice point from an elasticity energy between an adjacent lattice point pair of the second lattice points;

computing frictional force occurring at the each second lattice point;

performing a numerical analysis of an equation of motion regarding the second lattice points based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points;

detecting an occlusion region based on distribution of the positions which the second lattice points exist in the convergence state of the second lattice points;

detecting mapping from the first reference image to the third reference image by an image matching method; and

producing an interpolation image using corresponding relation of pixels of the first reference image and the second reference image which is determined by the second lattice points, the corresponding relation of pixels of the second reference image and the third reference image and information regarding the occlusion region.

7. The image interpolation method according toclaim 6, wherein the image matching method includes:

mapping fourth lattice points of a third lattice set on a third reference image to third lattice points of the second lattice on the second reference image respectively;

computing potential force applied to one fourth lattice point of the fourth lattice points by a gradient of an image correlation potential energy obtained by a position of corresponding one third lattice point of the third lattice points and a position of the one fourth lattice point;

computing elasticity force applied to the one fourth lattice point from an elasticity energy between the one fourth lattice point and the fourth lattice points adjacent to the one fourth lattice point,

computing frictional force occurring at the one fourth lattice point,

subjecting a equation of motion regarding the fourth lattice points and based on the potential force, the elasticity force and the frictional force to a numerical analysis to obtain a convergence state of the fourth lattice point, and

obtaining corresponding relation of pixels of the second reference image and the third reference image from relation between the third lattice point and the fourth lattice point.

8. The image interpolation method according toclaim 6, wherein the detecting the occlusion region includes:

dividing the second lattice in the convergence state of the second lattice point into a plurality of regions each having the second lattice points as vertexes,

computing occlusion reliability of each of the divided regions, and

detecting one region of the regions, which has the occlusion reliability greater than a threshold as the occlusion region, and

the producing the interpolation image includes:

obtaining a pixel value of each pixel in the occlusion region by weighting a first pixel value deriving from corresponding relation of pixels of the first reference image and the second reference image and a second pixel value derived from corresponding relation of pixels of the second reference image and the third reference image according to the occlusion reliability and combining them.

9. An image interpolation method for interpolating, using a first reference image, a second reference image, a third reference image and a fourth reference image which are arranged in time series, an interpolation image between the second reference image and the third reference image, the method comprising:

mapping, to each first lattice point of first lattice points of a first lattice on the second reference image, each second lattice point of second lattice points of a second lattice on the third reference image;

computing potential force applied to the each second lattice point of the second lattice points by a gradient of an image correlation potential energy based on a position of the each first lattice point and image information of the each first lattice point and a position of the each second lattice point and image information of the each second lattice point;

computing elasticity force applied to the each second lattice point from an elasticity energy between the each second lattice point and each of the second lattice points adjacent to the each second lattice point;

computing frictional force occurring at the each second lattice point;

performing a numerical analysis of an equation of motion regarding the second lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points; and

detecting a first occlusion region based on distribution of the second lattice points in the convergence state of the second lattice points,

mapping each fourth lattice point of fourth lattice points of the first lattice on the second reference image to each third lattice point of third lattice points of the second lattice on the third reference image,

computing potential force applied to the each fourth lattice point by a gradient of an image correlation potential energy based on a position of the each third lattice point and image information of the each third lattice point and a position of the each fourth lattice point and image information of the each fourth lattice point, computing elasticity force applied to the each fourth lattice point from an elasticity energy from the each fourth lattice point to the fourth lattice points adjacent to the one fourth lattice point,

computing frictional force occurring at the fourth lattice points,

performing a numerical analysis of an equation of motion regarding the fourth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the fourth lattice points; and

detecting a second occlusion region based on distribution of the fourth lattice points in the convergence state of the fourth lattice points;

performing first image matching for obtaining corresponding relation of pixels of the first reference image and the second reference image,

performing a second image matching for obtaining corresponding relation of pixels of the third reference image and the fourth reference image; and

generating an interpolation image using corresponding relation of pixels of the second reference image and the reference image, which is settled by the second lattice points, corresponding relation of pixels of the third reference image and the second reference image, which is settled by the fourth lattice points, corresponding relation of pixels of the second reference image and the first reference image, corresponding relation of pixels of the third reference image and the fourth reference image, information of the first occlusion region, and information of the second occlusion region.

10. The image interpolation method according toclaim 9, wherein:

the detecting the first occlusion region includes dividing the second lattice of the second lattice points in the convergence state into a plurality of first evaluation regions each having the second lattice points as vertexes,

computing occlusion reliability of each of the first evaluation regions;

detecting one of the first evaluation regions that has occlusion reliability greater than a threshold as a first occlusion region;

the detecting the second occlusion region includes dividing the fourth lattice in the convergence state of the fourth lattice points into a plurality of second evaluation regions each having the fourth lattice points as vertexes,

computing occlusion reliability of each of the second evaluation regions,

detecting one of the second evaluation regions that has occlusion reliability greater than a threshold as a second occlusion region; and

the generating the interpolation image includes:

acquiring a pixel value of each pixel in the first occlusion region by weighting a first pixel and a second pixel according to the occlusion reliability and combining them, the first pixel value being derived from corresponding relation of pixels of the second reference image and the third reference image and the second pixel value being derived from corresponding relation of pixels of the second reference image and the first reference image; and

acquiring a pixel value of each pixel in the second occlusion region by weighting a third pixel and a fourth pixel according to the occlusion reliability and combining them, the third pixel value being derived from corresponding relation of pixels of the second reference image and the third reference image and the fourth pixel value being derived from corresponding relation of pixels of the third reference image and the fourth reference image.

11. The image interpolation method according toclaim 6, wherein the first image matching and the second image matching each include acquiring the corresponding relation by a block matching method.

12. The image interpolation method according toclaim 6, wherein the first image matching and the second image matching each include acquiring the corresponding relation by an optical flow estimation method. By the first and the second image matching step, corresponding relation is demanded by an optical flow estimation method.

13. The image interpolation method according toclaim 6, wherein the first image matching and the second image matching each include acquiring the corresponding relation by a Bayesian method.

14. The image interpolation method according toclaim 6, wherein the first image matching and the second image matching each include acquiring the corresponding relation by a gradient method.

15. The image interpolation method according toclaim 9, which further comprises:

computing surface reliability of the second reference image based on distribution of the second lattice points in the convergence state;

computing surface reliability of the third reference image based on distribution of the fourth lattice points in the convergence state;

generating a first intermediate interpolation image based on distribution of the second lattice points in the convergence state;

generating a second intermediate interpolation image based on corresponding relation of pixels of the first reference image and the second reference image and the first occlusion region;

generating a third intermediate interpolation image based on distribution of the fourth lattice points in the convergence state;

generating a fourth intermediate interpolation image based on corresponding relation of pixels of the third reference image and the fourth reference image and the second occlusion region, and wherein the generating the interpolation image includes

generating the interpolation image using the surface reliability of the third reference image, the surface reliability of the second reference image, the first intermediate interpolation image, the second intermediate interpolation image, the third intermediate interpolation image and the fourth intermediate interpolation image.

16. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image and the computing the surface reliability of the third reference image each include determining as a non-surface region on which a distance between adjacent lattice points is greater than a reference value and determining as a surface region on which the distance nears the reference value, to output binary surface reliability representing the non-surface or the surface.

17. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image and the computing the surface reliability of the third reference image each include determining as a non-surface a region on which a distance between adjacent lattice points is less than a first threshold value and determining as a surface a region on which the distance is more than a second threshold value and less than a third threshold, to output a binary surface reliability representing the non-surface or the surface.

18. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image includes computing surface reliability of the second reference image based on difference between an area of a polygonal region having, as vertexes, the second lattice points of an initial state and an area of the polygonal region having, as vertexes, the second lattice points of the convergence state, and the computing the surface reliability of the third reference image includes computing surface reliability of the third reference image based on difference between an area of a polygonal region having, as vertexes, the fourth lattice points of an initial state and an area of a polygonal region having, as vertexes, the fourth lattice points of the convergence state.

19. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image and the computing the surface reliability of the third reference image each include determining as a non-surface a region on which a difference between the images is more than a threshold value and determining as a surface a region on which the difference is less than the threshold value, to output a binary surface reliability representing the non-surface or the surface.

20. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image and the computing the surface reliability of the third reference image each include determining a probability of reliability as 1 when the surface reliability is more than the threshold value and a probability of reliability as 0 when the surface reliability is less than the threshold value.

21. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image and the computing the surface reliability of the third reference image each includes computing a provability based on a sigmoid function.

22. The image interpolation method according toclaim 15, wherein the computing the surface reliability of the second reference image and the computing the surface reliability of the third reference image each include computing a probability based on the Gaussian distribution as the surface reliability.

23. The image interpolation method according toclaim 15, wherein:

the first image matching includes performing image matching using the first reference image as an original image and the second reference image as a target image;

the second image matching includes performing image matching using the third reference image as a target image and the fourth reference image as a target image; and

each of the first image matching and the second image matching includes:

computing potential gradient force due to the gradient of the image correlation potential energy based on pixels of the original image in an initial position of one of lattice points set to the original image and pixels of the target image in a current position of the one of the lattice points;

computing elasticity force due to an elasticity energy between one of the lattice points and the lattice points adjacent to the one of the lattice points;

computing frictional force to act on the one of the lattice points,

solving a equation of motion numerically by a discrete variable method to obtain a convergence state of each of the lattice points, the equation of motion regarding the lattice points and including the potential gradient force, the elasticity force and the frictional force; and

performing matching for obtaining corresponding relation between the original image and the target image using the initial position and the position in the convergence state.

24. The image interpolation method according toclaim 15, wherein the first matching and the second matching each are done by a block matching method.

25. The image interpolation method according toclaim 15, wherein the first matching and the second matching each are done by an optical flow estimation method.

26. The image interpolation method according toclaim 15, wherein the first matching and the second matching each are done by a Bayesian method.

27. The image interpolation method according toclaim 15, wherein;

the computing the surface reliability of the second reference image and the computing surface reliability of the third reference image each include obtaining a back side reliability that is a complement set of surface reliability and detecting, as a back side region, a region having the back side reliability greater than a given value;

performing matching using a first back side region detected by the computing the surface reliability of the second reference image and a second back side region detected by the computing the surface reliability of the third reference image; and

detecting, as a surface region, a region on which the first back side region and the second back side region correctly correspond to each other.

28. An image interpolation method for interpolating, using a first reference image, a second reference image and a third reference image and a fourth reference image, which are arranged in time series, an interpolation image between the second reference image and the third reference image, the method comprising:

mapping first lattice points of a first lattice on the second reference image onto second lattice points of a second lattice on the third reference image, respectively;

computing potential force applied to each second lattice point of the second lattice points by a gradient of an image correlation potential energy based on a position of each first lattice point of the first lattice points that corresponds to the each second lattice point and image information the each first lattice point and a position of the each second lattice point and image information of the each second lattice point;

computing elasticity force applied to the each second lattice point from an elasticity energy from the each second lattice point to the second lattice points adjacent to the each second lattice point;

computing frictional force occurring at the each second lattice point,

performing a numerical analysis of an equation of motion regarding the second lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points;

detecting a first occlusion region based on distribution of the second lattice points in the convergence state thereof;

mapping, third lattice points of the second lattice on the third reference image, onto fourth lattice points of the first lattice on the second reference image respectively,

computing potential force applied to each fourth lattice point of the fourth lattice points by a gradient of an image correlation potential energy based on a position of each third lattice point of the third lattice points and image information thereof and a position of the each fourth lattice point and image information of the each fourth lattice point;

computing elasticity force applied to the reference fourth lattice point from an elasticity energy from the each fourth lattice point to the fourth lattice points adjacent to the each fourth lattice point;

computing frictional force occurring at the each fourth lattice point;

performing a numerical analysis of an equation of motion regarding the fourth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the fourth lattice points; and

detecting a second occlusion region based on distribution of the fourth lattice points in the convergence state thereof;

mapping, fifth lattice points of the second lattice on the third reference image, onto sixth lattice points of the first lattice on the second reference image respectively,

computing potential force applied to each sixth lattice point of the sixth lattice points by a gradient of an image correlation potential energy based on a position of each fifth lattice point of the fifth lattice points and a position of the each sixth lattice point;

computing elasticity force applied to the each sixth lattice point by an elasticity energy from the each sixth lattice point to the sixth lattice points adjacent to the each sixth lattice point;

computing frictional force occurring at the each sixth lattice point;

performing a numerical analysis of an equation of motion regarding the sixth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the sixth lattice points; and

mapping, seventh lattice points of the second lattice on the third reference image, onto eighth points of the first lattice on the second reference image respectively,

computing potential force applied to each eighth lattice point of the eight lattice points by a gradient of an image correlation potential energy based on a position of each seventh lattice point of the seventh lattice points and a position of the each eighth lattice point;

computing elasticity force applied to the each eighth lattice point by an elasticity energy from the each eighth lattice point to the eighth lattice points adjacent to the each eighth lattice point;

computing frictional force occurring at the each eighth lattice point;

performing a numerical analysis of an equation of motion regarding the eighth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the eighth lattice points; and

generating an interpolation image using corresponding relation settled by the each second lattice point, corresponding relation settled by the each fourth lattice point, corresponding relation settled by the each sixth lattice point, corresponding relation settled by the each eighth lattice point, information of the first occlusion region, and information of the each second occlusion region.

29-35. (canceled)

Images