1477Accesses
348Citations
17 Altmetric
1Mention
Abstract
In this paper, we study face hallucination, or synthesizing a high-resolution face image from an input low-resolution image, with the help of a large collection of other high-resolution face images. Our theoretical contribution is a two-step statistical modeling approach that integrates both a global parametric model and a local nonparametric model. At the first step, we derive a global linear model to learn the relationship between the high-resolution face images and their smoothed and down-sampled lower resolution ones. At the second step, we model the residue between an original high-resolution image and the reconstructed high-resolution image after applying the learned linear model by a patch-based non-parametric Markov network to capture the high-frequency content. By integrating both global and local models, we can generate photorealistic face images. A practical contribution is a robust warping algorithm to align the low-resolution face images to obtain good hallucination results. The effectiveness of our approach is demonstrated by extensive experiments generating high-quality hallucinated face images from low-resolution input with no manual alignment.
This is a preview of subscription content,log in via an institution to check access.
Access this article
Subscribe and save
- Starting from 10 chapters or articles per month
- Access and download chapters and articles from more than 300k books and 2,500 journals
- Cancel anytime
Buy Now
Price includes VAT (Japan)
Instant access to the full article PDF.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, books and news in related subjects, suggested using machine learning.References
Baker, S. and Kanade, T. 2000a. Hallucinating faces. InIEEE International Conference on Automatic Face and Gesture Recognition.
Baker, S. and Kanade, T. 2000b. Limits on super-resolution and how to break them. InProc. IEEE Conf. Computer Vision and Pattern Recognition.
Baker, S. and Matthews, I. 2004. Lucas-kanade 20 years on: A unifying framework.International Journal on Compter Vision, 56(3):221–255.
Blake, A., Bascle, B., and Zisserman, A. 1996. Motion deblurring and super-resolution from an image sequence. InProc. European Conference on Computer Vision, pp. 312–320.
De Bonet, J. 1997. Multiresolution sampling procedure for analysis and synthesis of texture images. InProceedings of SIGGRAPH 97, pp. 361–368.
Capel, D. and Zisserman, A. 2001. Super-resolution from multiple views using learnt image models. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 627–634.
Chen, H., Xu, Y.Q., Shum, H.Y., Zhu, S.C., and Zheng, N.N. 2001. Example-based facial sketch generation with non-parametric sampling. InProc. IEEE Int’l Conf. Computer Vision, pp. 433– 438.
Cootes, T.F. and Taylor, C.J. 2000. Statistical models of appearance for computer vision. Technical report, University of Manchester.
Dedeoglu, G., Baker, S., and Kanade, T. 2006. Resolution-aware fitting of active appearance models to low-resolution images. InProc. European Conference on Computer Vision. Springer, pp. 83– 97.
Dedeoglu, G., Kanade, T., and August, J. 2004. High-zoom video hallucination by exploiting spatio-temporal regularities. InProc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 151–158.
Efros, A.A. and Freeman, W.T. 2001. Quilting for texture synthesis and transfer. InProceedings of SIGGRAPH 2001, pp. 341–346.
Fitzgibbon, A.W., Wexler, Y., and Zisserman, A. 2003. Image-based rendering using image-based priors. InProc. IEEE Int’l Conf. Computer Vision, pp. 1176–1183.
Freeman, W.T., Jones, T.R., and Pasztor, E.C. 2002. Example-based super-resolution.IEEE Computer Graphics and Applications, 22(2):56–65.
Freeman, W.T., Pasztor, E.C., and Carmichael, O.T. 2000. Learning low-level vision.International Journal on Compter Vision, 40(1):25–47.
Greenspan, H., Anderson, C., and Akber, S. 2000. Image enhancement by nonlinear extrapolation in frequency space.IEEE Trans. on Image Processing, 9(6).
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., and Salesin, D.H. 2001. Image analogies. InProceedings of SIGGRAPH 2001.
Hou, H.H. and Andrews, H.C. 1978. Cubic splines for image interpolation and digital filtering.IEEE Trans. Acoust. Speech Signal Proc., 26(6):508–517.
Huang, T.S. and Tsai, R.Y. 1984. Multi-frame image restoration and registration.Advances in Computer Vision and Image Processing, 1:317–339.
Jia, K. and Gong, S. 2005. Multi-modal tensor face for simultaneous super-resolution and recognition. InProc. IEEE Int’l Conf. Computer Vision, pp. 1683–1690.
Jordan, M.I. 1998. (ed.)Learning in Graphical Models. MIT Press.
Li, Y. and Lin, X. 2004. Face hallucination with pose variation. InIEEE International Conference on Automatic Face and Gesture Recognition, pp. 723–728.
Liang, L., Liu, C., Xu, Y.Q., Guo, B., and Shum, H.Y. 2001. Real-time texture synthesis by patch-based sampling.ACM Trans. Graph., 20(3):127–150.
Liu, C., Freeman, W.T., Szeliski, R., and Kang, S.B. 2006. Noise estimation from a single image. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 901–908.
Liu, C., Shum, H.Y., and Zhang, C.S. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 192–198.
Liu, C., Zhu, S.C., and Shum, H.Y. 2001. Learning inhomogeneous Gibbs model of faces by minimax entropy. InProc. IEEE Int’l Conf. Computer Vision, pp. 281–287.
Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. Inthe 7th International Joint Conference on Artificial Intelligence (IJCAI ’81), pp. 674–679.
Martinez, A. and Benavente, R. 1998. The AR face database. Technical report, CVC Techinical Report No. 24.
Martinez, D. 1986. Model-based Motion Estimation and its Application to Restoration and Interpolation of Motion Pictures. PhD thesis, Massachusetts Institute of Technology.
Morse, B. and Schwartzwald, D. 2001. Image magnification using level set reconstruction. InProc. IEEE Int’l Conf. Computer Vision, pp. 333–341.
Oppenheim, A.V., Willsky, A.S., and Nawab, S.H. 1997.Signal and Systems 2nd edition. Prentice Hall, Inc.
Philips, P., Moon, H., Pauss, P., and Rivzvi. S. 1997. The feret evaluation methodology for face-recognition algorithms. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 137–143.
Press, W., Teukolsky, S., Vetterling, W., and Flannery, B. 1992.Numerical Recipes in C 2nd edition. Cambridge University Press.
Rowley, H.A., Baluja, S., and Kanade, T. 1998. Neural network-based face detection.IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(1):23–38.
Schultz, R.R. and Stevenson, R.L. 1994. A Bayesian approach to image expansion for improved definition.IEEE Trans. Image Processing, 3(3):233–242.
Shechtman, E., Caspi, Y., and Irani, M. 2005. Space-time super-resolution.IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(4):531–545.
Strang, G. 1988.Linear Algebra and Its Applications 3rd edn. Thomson Learning, Inc.
Sun, J., Zheng, N.N., Tao, H., and Shum, H.-Y. 2003. Generic image hallucination with primal sketch prior. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 729– 736.
Tappen, M.F., Russell, B.C., and Freeman, W.T. 2004. Efficient graphical models for processing images. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 673–680.
Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images. InProc. IEEE Int’l Conf. Computer Vision, pp. 839– 846.
Turk, M. and Pentland, A. 1991. Eigenfaces for recognition.Journal of Cognitive Neurosciences, 3:71–86.
Viola, P. and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 511–518.
Wang, X.G. and Tang, X. 2005. Hallucinating face by eigentransformation.IEEE Transactions on Systems, Man, and Cybernetics, Part C, 35(3):425–434.
Xiao, R., Li, M.J., and Zhang, H.J. 2004. Robust multipose face detection in images.IEEE Trans. Circuits Syst. Video Techn., 14(1):31– 41.
Zhou, Y., Gu, L., and Zhang, H.J. 2003. Bayesian tangent shape model: Estimating shape and pose parameters via bayesian inference. InProc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 109–118.
Zhu, S., Wu, Y., and Mumford, D. Filters random fields and maximum entropy (FRAME): To a unified theory for texture modeling.International Journal on Compter Vision, 27:1–20, 1998.
Author information
Authors and Affiliations
Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Massachusetts, USA
Ce Liu & William T. Freeman
Microsoft Research Asia, Beijing, China
Heung-Yeung Shum
- Ce Liu
Search author on:PubMed Google Scholar
- Heung-Yeung Shum
Search author on:PubMed Google Scholar
- William T. Freeman
Search author on:PubMed Google Scholar
Corresponding author
Correspondence toCe Liu.
Rights and permissions
About this article
Cite this article
Liu, C., Shum, HY. & Freeman, W.T. Face Hallucination: Theory and Practice.Int J Comput Vis75, 115–134 (2007). https://doi.org/10.1007/s11263-006-0029-5
Received:
Accepted:
Published:
Issue date:
Share this article
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative