US20140132599A1 - Image processing apparatus and method - Google Patents

Abstract

An image processing apparatus that configures a single frame by determining a central image of a certain viewpoint as an original resolution, and frame another single frame by combining a left image of a left viewpoint and a right image of a right viewpoint. The image processing apparatus may generate three-dimensional (3D) image data configured using the frames, and may encode, decode, and render an image based on the 3D image data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a divisional of U.S. application Ser. No. 12/923,828, filed on Oct. 8, 2010, which claims the benefit of Korean Patent Application No. 10-2010-0010096, filed on Feb. 3, 2010, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference.
BACKGROUND
• 1. Field
• One or more embodiments relate to an image processing apparatus and method, and more particularly, to technology that may process an image based on three-dimensional (3D) image data containing an original resolution of a central image, and a processed left image and a processed right image.
• 2. Description of the Related Art
• Three-dimensional (3D) image data may include video data photographed at a plurality of viewpoints, and depth data indicating a distance between a subject and a camera. The 3D image data may be transformed to a predetermined format and be stored. The 3D image data may be compressed and be transmitted to a user. A reception end may play the 3D image data using one of a two-dimensional (2D) image, astereo type 3D image, and a multi-view image depending on the user's purpose and a type of a display.
• Accordingly, there is a desire for method and apparatus that may play one of a 2D image, astereo type 3D image, and a multi-view image using an existing broadcasting infra and without loss of resolution.
SUMMARY
• According to an aspect of one or more embodiments, there may be provided an image processing apparatus including an image input unit to receive an original left image, an original central image, and an original right image, an image processor to generate a processed left image and a processed right image by processing the original left image and the original right image, and an image generator to generate three-dimensional (3D) image data including a first frame image and a second frame image, the first frame image configured using the original central image and the second frame image configured using the processed left image and the processed right image.
• The image processor may generate the processed left image and the processed right image by reducing, by half, a vertical resolution of the original left image and a vertical resolution of the original right image.
• The 3D image data may further include auxiliary data including at least one of depth image data corresponding to each viewpoint image, segmentation information, and transparency information, and metadata including at least one of camera parameter information and depth range information.
• The 3D image data may further include filter information including history information regarding the original left image and the original right image being processed to the processed left image and the processed right image.
• The image processing apparatus may further include an image adder to add two-dimensional (2D) image data to the original central image when a 2D image is added, to add stereo image data to the original left image and the original right image when a stereo image is added, and to add multi-view image data to the original central image, the original left image, and the original right image when a multi-view image is added.
• The image processing apparatus may further include an encoder to encode the 3D image data.
• According to another aspect of one or more embodiments, there may be provided an image processing apparatus including an image separator to separate a first frame image and a second frame image from an encoded image, a mode selector to select an output mode, a 2D output unit to output a 2D image by decoding the first frame image when the selected output mode is a 2D output mode, a stereo output unit to output a stereo image by decoding the second frame image when the selected output mode is a stereo output mode, and a multi-view output unit to output a multi-view image by decoding the first frame image and the second frame image and by generating at least one new viewpoint image when the selected output mode is a multi-view output mode.
• The first frame image may be configured using an original central image and the second frame image may be configured using a processed left image and a processed right image obtained by processing an original left image and an original right image.
• The processed left image may correspond to an image generated by reducing a vertical resolution of the original left image by half. The processed right image may correspond to an image generated by reducing a vertical resolution of the original right image by half.
• The multi-view output unit may include a multi-view decoder to decode the first frame image and the second frame image, and an image generator to generate at least one new viewpoint image. The image generator may include an image selector to determine, as a first image, one image selected from the processed left image and the processed right image, a warping unit to three-dimensionally warp the first image and the original central image, and an image blender to blend an image for correcting a hole of the warped original central image.
• The image blender may include an image interpolator to restore an original resolution by interpolating the warped first image. A new viewpoint image may be generated by blending the interpolated first image and the warped original central image.
• The image interpolator may interpolate the first image based on filter information containing history information regarding the original left image or the original right image being processed.
• According to still another aspect of one or more embodiments, there may be provided an image processing apparatus including a warping unit to three dimensionally warp a first viewpoint image of a first resolution and a second viewpoint image of a second resolution, an image processor to generate a correction image by processing the warped second viewpoint image, and an image blender to generate a new viewpoint image by blending the warped first viewpoint image and the correction image.
• According to yet another general aspect of one or more embodiments, there may be provided an image processing method including receiving an original left image, an original central image, and an original right image, generating a processed left image and a processed right image by processing the original left image and the original right image, and generating 3D image data including a first frame image and a second frame image, the first frame image configured using the original central image and the second frame image configured using the processed left image and the processed right image.
• Additional aspects, features, and/or advantages of exemplary embodiments will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
• These and/or other aspects and advantages will become apparent and more readily appreciated from the following description of the exemplary embodiments, taken in conjunction with the accompanying drawings of which:
• FIG. 1 illustrates an image processing apparatus to generate three-dimensional (3D) image data according to an embodiment;
• FIG. 2 illustrates a process of generating 3D image data using interlacing according to an embodiment;
• FIG. 3 illustrates a process of generating 3D image data using a top and bottom scheme according to an embodiment;
• FIG. 4 illustrates a process of adding various data according to an embodiment;
• FIG. 5 illustrates an image processing apparatus to decode and output 3D image data according to another embodiment;
• FIG. 6 illustrates an image processing apparatus to generate a new viewpoint image according to still another embodiment;
• FIG. 7 illustrates a process of generating a new viewpoint image based on 3D image data according to an embodiment; and
• FIG. 8 illustrates an image processing method to generate 3D image data according to an embodiment.
DETAILED DESCRIPTION
• Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Exemplary embodiments are described below to explain the present disclosure by referring to the figures.
• FIG. 1 illustrates animage processing apparatus100 to generate three-dimensional (3D) image data according to an embodiment.
• Referring toFIG. 1, theimage processing apparatus100 to generate the 3D image data may include animage input unit110, animage processor120, and animage generator130.
• Theimage input unit110 may receive an original left image, an original central image, and an original right image. The original central image denotes an original image photographed at a central viewpoint. The original left image denotes an original image photographed at a left viewpoint based on the central viewpoint. The original right image denotes an original image photographed at a right viewpoint based on the central viewpoint.
• Theimage processor120 may generate a processed left image and a processed right image by processing the original left image and the original right image. The original left image and the original right image may be transformed and be used. For example, theimage processor120 may generate the processed left image and the processed right image by performing a transformation of reducing a resolution of the original left image and the original right image. A resolution of the processed left image and the processed right image may correspond to half a resolution of the original left image and the original right image.
• To reduce the resolution of the original left image and the original right image, theimage processor120 may generate an odd image including odd lines and an even image including even lines by interlacing the original left image and the original right image, and may determine the odd image and the even image as the processed left image and the processed right image.
• To reduce the resolution of the original left image and the original right image, theimage processor120 may generate a checkerboard left image having half a resolution of the original left image and a checkerboard right image having half a resolution of the original right image by arranging the original left image and the original right image in a checkerboard type, and may determine the checkerboard left image and the checkerboard right image as the processed left image and the processed right image.
• To reduce the resolution of the original left image and the original right image, theimage processor120 may generate the processed left image and the processed right image by reducing a vertical resolution of the original left image and the original right image, and may also generate the processed left image and the processed right image using various types of schemes, for example, a top and down scheme of arranging the generated processed left image and processed right image in top and down, a side by side scheme of arranging the generated processed left image and processed right image in left and right, and the like.
• Hereinafter, an example of generating the processed left image and the processed right image will be further described with reference toFIGS. 2 and 3.
• FIG. 2 illustrates a process of generating 3D image data using interlacing according to an embodiment.
• Referring toFIG. 2, aneven image220 and anodd image230 may be generated by interlacing an original left image and an original right image of an originalcentral image210. Specifically, an even image or an odd image among images interlaced from the original left image may be selected as a processed left image. Also, an even image or an odd image among images interfaced from the original right image may be selected as a processed right image. When the even image among the images interlaced from the original left image is selected as the processed left image, the odd image among the images interlaced from the original right image may be selected as the processed right image. Accordingly, when one of the processed left image and the processed right image corresponds to the even image, the remaining image may be selected as the odd image.
• FIG. 3 illustrates a process of generating 3D image data using a top and bottom scheme according to an embodiment.
• Referring toFIG. 3, atop image320 and abottom image330 may be generated by reducing a resolution of an original left image and an original right image of an originalcentral image310 by half. Specifically, an image generated by reducing the resolution of the original left image by half may be selected as a processed left image, and an image generated by reducing the resolution of the original right image by half may be selected as a processed right image. The selected processed left image or processed right image may be determined as thetop image320 or thebottom image330 and thereby be configured as a single frame.
• Referring again toFIG. 1, theimage generator130 may generate a single frame configured using the processed left image and the processed right image, and another single frame configured using an original central image, and may generate 3D image data including the two frames.
• The 3D image data may further include auxiliary data and metadata in addition to generated frame data. The auxiliary data may include at least one of depth image data corresponding to each viewpoint image, segmentation information, and transparency information. The metadata may include at least one of camera parameter information and depth range information.
• The 3D image data may further include filter information including history information regarding the original left image and the original right image being processed to the processed left image and the processed right image. For example, an original resolution may be easily restored during a decoding process by including, in the 3D image data, filter information containing history information regarding a change in of a resolution of the processed left image and the processed right image generated by changing a resolution of the original left image and the original right image.
• Theimage processing apparatus100 may further include animage adder150 to add two-dimensional (2D) image data to the original central image when a 2D image is added, to add stereo image data to the original left image and the original right image when a stereo image is added, and to add multi-view image data to the original central image, the original left image, and the original right image when a multi-view image is added. While outputting an image, theimage processing apparatus100 may play the image by adding another image of a different mode to the image or by editing the image. It will be further described with reference toFIG. 4.
• FIG. 4 illustrates a process of adding various data according to an embodiment.
• Referring toFIG. 4, when a 2D advertisement is desired to be added while outputting a multi-view 3D image, the 2D image may be added and be played by adding only acentral image410 without adding a left image and a right image. When a stereo advertisement is desired to be added while outputting the multi-view 3D image, the stereo advertisement may be added and be played by adding only aleft image421 and aright image422 and by adding a central image as a null image. When a multi-view advertisement is desired to be added while outputting the multi-view 3D image, the multi-view advertisement may be added and be played by adding all of aleft image431, acentral image432, and aright image433.
• Referring again toFIG. 1, theimage processing apparatus100 may further include anencoder140 to encode the 3D image data.
• As described above, an image may be processed based on 3D image data where an original central image is configured as a single frame and a processed left image and a processed right image are configured as another frame. Accordingly, a high definition of image may be processed using minimum amounts of data.
• FIG. 5 illustrates animage processing apparatus500 to decode andoutput 3D image data according to another embodiment.
• Referring toFIG. 5, theimage processing apparatus500 to decode and output the 3D image data may include animage separator510, amode selector520, a2D output unit530, astereo output unit540, and amulti-view output unit550.
• Theimage separator510 may separate a first frame image and a second frame image from an encoded image. The first frame may be configured using an original central image, and the second frame may be configured using a processed left image and a processed right image processed from an original left image and an original right image. The processed left image may correspond to an image generated by reducing a vertical resolution of the original left image by half. The processed right image may correspond to an image generated by reducing a vertical resolution of the original right image by half. As described above, the processed left image and the processed right image may be generated by interlacing the original left image and the original right image, and may also be generated by arranging the original left image and the original right image into a checkerboard type. Also, the processed left image and the processed right image may be generated by reducing, by half, a vertical resolution of the original left image and the original right image. The generated processed left image and processed right image may be arranged within a single frame using various schemes, for example, a top and down scheme, a side by side scheme, and the like.
• Themode selector520 may select an output mode to output an image in various image output devices enabling the image to be output in various modes. For example, the various output modes may include a 2D output mode, a stereo output mode, a multi-view output mode, and the like.
• The2D output unit530 may output a 2D image by decoding the first frame image when the selected output mode is a 2D output mode. Specifically, in the case of a 2D image output mode, since the 2D image may be configured by outputting only a central image, the 2D image may be output by decoding the first frame image including the central image.
• Thestereo output unit540 may output a stereo image by decoding the second frame image when the selected output mode is a stereo output mode. Specifically, in the case of a stereo image output mode, since the stereo image may be configured by outputting only a left image and a right image, the stereo image may be output by decoding the second frame image including the left image and the right image.
• Themulti-view output unit550 may output a multi-view image by decoding the first frame image and the second frame image and by generating at least one new viewpoint image when the selected output mode is a multi-view output mode. Specifically, in the case of a multi-view image output mode, since the multi-view image may be configured by outputting the central image, the left image, and the right image, the multi-view image may be output by decoding the first frame image including the central image and the second frame image including the left image and the right image.
• Themulti-view output unit550 may include amulti-view decoder551 to decode the first frame image and the second frame image, and animage generator552 to generate at least one new viewpoint image. Theimage generator552 may include animage selector553, awarping unit554, and animage blender555.
• To generate a new viewpoint image, theimage selector553 may determine, as a first image, one image selected from the processed left image and the processed right image. When the new viewpoint image is desired to be generated between the processed left image and an original central image, theimage selector553 may determine the processed left image as the first image. When the new viewpoint image is desired to be generated between the processed right image and the original central image, theimage selector553 may determine the processed right image as the first mage.
• Thewarping unit554 may three-dimensionally warp the first image and the original central image. Thewarping unit554 may perform transformation to an inter-viewpoint based on depth image corresponding to the first image and the original central image.
• Theimage blender555 may blend an image for correcting a hole of the warped original central image. Theimage blender555 may include an image interpolator (not shown) to restore an original resolution by interpolating the warped first image. The new viewpoint image may be generated by blending the interpolated first image and the warped original central image. The image interpolator may interpolate the first image based on filter information containing history information regarding the original left image or the original right image being processed. For example, the image interpolator may perform interpolation of restoring an original resolution based on history information used to generate the processed left image or the processed right image by changing a resolution of the original left image or the original right image.
• As described above, an output image may be configured without losing an original resolution by processing an image based on image data maintaining the original resolution. Also, the image may be effectively processed without a change in an existing infra by generating 3D image data used to selectively play one of a 2D image, a stereo image, and a multi-view image, and by processing the image based on the 3D image data.
• FIG. 6 illustrates animage processing apparatus600 to generate a new viewpoint image according to still another embodiment.
• Referring toFIG. 6, theimage processing apparatus600 to generate the new viewpoint image in order to play a multi-view image may include awarping unit610, animage processor620, and animage blender630.
• Thewarping unit610 may three dimensionally warp a first viewpoint image of a first resolution and a second viewpoint of a second resolution. For example, inter-viewpoint images may be generated by performing 3D warping based on the first resolution of a central image, the second resolution of a right image, and corresponding depth image.
• Theimage processor620 may generate a correction image by processing the warped second viewpoint image. When the first viewpoint image corresponds to the central image, a resolution of the second viewpoint image reduced during an encoding process may become to be the same as the resolution of the original viewpoint image by interpolating the 3D warped second viewpoint image, for example, the right image in order to correct a hole area occurring during a 3D warping process of the central image.
• Theimage blender630 may generate a new viewpoint image by blending the warped first viewpoint image and the correction image. Accordingly, an enhanced new viewpoint image may be generated by correcting the hole area generated during the 3D warping process of the first viewpoint image.
• Hereinafter, an example of generating the new viewpoint image will be further described with reference toFIG. 7.
• FIG. 7 illustrates a process of generating a new viewpoint image based on 3D image data according to an embodiment.
• Referring toFIG. 7, the new viewpoint image may be generated between acentral image710 and a right image or aleft image720. The left image or theright image720 may be selected depending on whether a location of a viewpoint to be generated is right or left based on thecentral image710, and may be blended with the new viewpoint image.New viewpoint images711 and712 may be generated by three dimensionally warping thecentral image710 and the selected left image orright image720. Since a hole occurs during the 3D warping process of thecentral image710, a correction image may be generated to remove the hole. In this instance, the correction image may be animage722 generated by interpolating anew viewpoint image721 with respect to the 3D warped left image orright image720, and by restoring a resolution of a second viewpoint image reduced during an encoding process to be the same as a resolution of an original image.
• Accordingly, the enhanced new viewpoint image may be finally generated by blending the restoredimage722 and thenew viewpoint image711 with respect to the central image.
• FIG. 8 illustrates an image processing method to generate 3D image data according to an embodiment.
• Inoperation810, an original left image, an original central image, and an original right image may be received.
• Inoperation820, a processed left image and a processed right image may be processed by processing the original left image and the original right image. The processed left image may correspond to an image generated by reducing a vertical resolution of the original left image by half, and the processed right image may correspond to an image generated by reducing a vertical resolution of the original right image by half.
• Inoperation830, 3D image data including a first frame image and a second frame image may be generated. Here, the first frame image may be configured using the original central image and the second frame image may be configured using the processed left image and the processed right image. The 3D image data may further include filter information including history information regarding the original left image and the original right image being processed to the processed left image and the processed right image. Also, the 3D image data may further include auxiliary data including at least one of depth image data corresponding to each viewpoint image, segmentation information, and transparency information, and metadata including at least one of camera parameter information and depth range information.
• The generated 3D image data may be encoded through an encoding process and be transmitted to an image output device and the like.
• Matters not described above with reference toFIG. 8 may refer to descriptions made above with reference toFIGS. 1 through 7.
• By processing an image based on 3D image data where an original central image is configured as a single frame and a processed left image and a processed right image are configured as another frame, a high definition of image may be processed using minimum amounts of data, and an output image may be configured without loss in a resolution. Also, the image may be effectively processed without a change in an existing infra.
• The image processing method according to the above-described exemplary embodiments may be recorded in computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVDs; magneto-optical media such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described exemplary embodiments, or vice versa.
• Although a few exemplary embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these exemplary embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.

Claims (21)

What is claimed is:

1. An image processing apparatus, comprising:

an image input unit to receive an original left image, an original central image, and an original right image;

an image processor to generate a processed left image and a processed right image by processing the original left image and the original right image; and

an image generator to generate three-dimensional (3D) image data including a first frame image and a second frame image, the first frame image configured using the original central image and the second frame image configured using the processed left image and the processed right image.

2. The image processing apparatus ofclaim 1, wherein the image processor generates the processed left image and the processed right image by reducing, by half, a vertical resolution of the original left image and a vertical resolution of the original right image.

3. The image processing apparatus ofclaim 1, wherein the image processor generates an odd image including odd lines and an even image including even lines by interlacing the original left image and the original right image, and determines the odd image and the even image as the processed left image and the processed right image.

4. The image processing apparatus ofclaim 1, wherein the image processor generates a checkerboard left image having half a resolution of the original left image and a checkerboard right image having half a resolution of the original right image by arranging the original left image and the original right image in a checkerboard type, and determines the checkerboard left image and the checkerboard right image as the processed left image and the processed right image.

5. The image processing apparatus ofclaim 1, wherein the 3D image data further comprises auxiliary data including at least one of depth image data corresponding to each viewpoint image, segmentation information, and transparency information, and metadata including at least one of camera parameter information and depth range information.

6. The image processing apparatus ofclaim 1, wherein the 3D image data further comprises filter information including history information regarding the original left image and the original right image being processed to the processed left image and the processed right image.

7. The image processing apparatus ofclaim 1, further comprising:

an image adder to add two-dimensional (2D) image data to the original central image when a 2D image is added, to add stereo image data to the original left image and the original right image when a stereo image is added, and to add multi-view image data to the original central image, the original left image, and the original right image when a multi-view image is added.

8. The image processing apparatus ofclaim 1, further comprising:

an encoder to encode the 3D image data.

9. An image processing apparatus, comprising:

a warping unit to three dimensionally warp a first viewpoint image of a first resolution and a second viewpoint image of a second resolution;

an image processor to generate a correction image by processing the warped second viewpoint image; and

an image blender to generate a new viewpoint image by blending the warped first viewpoint image and the correction image.

10. The image processing apparatus ofclaim 9, wherein the image processor generates the correction image having the first resolution by interpolating the warped second viewpoint image.

11. The image processing apparatus ofclaim 9, wherein the image interpolator uses filter information containing processing history information of the second viewpoint image to interpolate the second viewpoint image.

12. An image processing method, comprising:

receiving an original left image, an original central image, and an original right image;

generating a processed left image and a processed right image by processing the original left image and the original right image; and

generating 3D image data including a first frame image and a second frame image, the first frame image configured using the original central image and the second frame image configured using the processed left image and the processed right image.

13. The image processing method ofclaim 12, wherein the generating of the processed left image and the processed right image comprises generating the processed left image and the processed right image by reducing, by half, a vertical resolution of the original left image and a vertical resolution of the original right image.

14. The image processing method ofclaim 12, wherein the generating of the processed left image and the processed right image comprises generating an odd image including odd lines and an even image including even lines by interlacing the original left image and the original right image, and determines the odd image and the even image as the processed left image and the processed right image.

15. The image processing method ofclaim 12, wherein the generating of the processed left image and the processed right image comprises generating a checkerboard left image having half a resolution of the original left image and a checkerboard right image having half a resolution of the original right image by arranging the original left image and the original right image in a checkerboard type, and determining the checkerboard left image and the checkerboard right image as the processed left image and the processed right image.

16. The image processing method ofclaim 12, wherein the 3D image data further comprises filter information including history information regarding the original left image and the original right image being processed to the processed left image and the processed right image.

17. The image processing method ofclaim 12, further comprising encoding the 3D image data.

18. A non-transitory computer-readable recording medium storing a program to implement the method ofclaim 12.

19. An image processing apparatus, comprising:

an image input unit to receive images denoting differing viewpoints with respect to a central one of the images and with respect to each other;

an image processor to generate a processed first image based on a first one of the images having a differing viewpoint than the central one of the images and a processed second image based on a second one of the images having a differing viewpoint with respect to the central one of the images; and

an image generator to generate three-dimensional (3D) image data including a first frame image and a second frame image, the first frame image configured using the central one of the images and the second frame image configured using the processed first image and the processed second image.

20. The image processing apparatus ofclaim 19, wherein the received images include a left image, the central one of the images, and a right image.

21. The image processing apparatus ofclaim 20, wherein the left image corresponds to the first one of the images, and the right image corresponds to the second one of the images.

Images