Full Text:   <1946>

CLC number: TP391.41

On-line Access: 

Received: 2008-02-26

Revision Accepted: 2008-06-22

Crosschecked: 2008-12-26

Cited: 1

Clicked: 3260

Citations:  Bibtex RefMan EndNote GB/T7714

-   Go to

Article info.
1. Reference List
Open peer comments

Journal of Zhejiang University SCIENCE A 2009 Vol.10 No.2 P.232~238


Patch-guided facial image inpainting by shape propagation

Author(s):  Yue-ting ZHUANG, Yu-shun WANG, Timothy K. SHIH, Nick C. TANG

Affiliation(s):  Institute of Artificial Intelligence, School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China; more

Corresponding email(s):   yzhuang@cs.zju.edu.cn, eawang@microsoft.com, tshih@cs.tku.edu.tw, nick.ctang@msa.hinet.net

Key Words:  Image inpainting, Face reconstruction, Feature point extraction

Yue-ting ZHUANG, Yu-shun WANG, Timothy K. SHIH, Nick C. TANG. Patch-guided facial image inpainting by shape propagation[J]. Journal of Zhejiang University Science A, 2009, 10(2): 232~238.

@article{title="Patch-guided facial image inpainting by shape propagation",
author="Yue-ting ZHUANG, Yu-shun WANG, Timothy K. SHIH, Nick C. TANG",
journal="Journal of Zhejiang University Science A",
publisher="Zhejiang University Press & Springer",

%0 Journal Article
%T Patch-guided facial image inpainting by shape propagation
%A Yue-ting ZHUANG
%A Yu-shun WANG
%A Timothy K. SHIH
%A Nick C. TANG
%J Journal of Zhejiang University SCIENCE A
%V 10
%N 2
%P 232~238
%@ 1673-565X
%D 2009
%I Zhejiang University Press & Springer
%DOI 10.1631/jzus.A0820138

T1 - Patch-guided facial image inpainting by shape propagation
A1 - Yue-ting ZHUANG
A1 - Yu-shun WANG
A1 - Timothy K. SHIH
A1 - Nick C. TANG
J0 - Journal of Zhejiang University Science A
VL - 10
IS - 2
SP - 232
EP - 238
%@ 1673-565X
Y1 - 2009
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/jzus.A0820138

Images with human faces comprise an essential part in the imaging realm. Occlusion or damage in facial portions will bring a remarkable discomfort and information loss. We propose an algorithm that can repair occluded or damaged facial images automatically, named ‘facial image inpainting’. Inpainting is a set of image processing methods to recover missing image portions. We extend the image inpainting methods by introducing facial domain knowledge. With the support of a face database, our approach propagates structural information, i.e., feature points and edge maps, from similar faces to the missing facial regions. Using the inferred structural information as guidance, an exemplar-based image inpainting algorithm is employed to copy patches in the same face from the source portion to the missing portion. This newly proposed concept of facial image inpainting outperforms the traditional inpainting methods by propagating the facial shapes from a face database, and avoids the problem of variations in imaging conditions from different images by inferring colors and textures from the same face image. Our system produces seamless faces that are hardly seen drawbacks.

Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article


[1] Criminisi, A., Perez, P., Toyama, K., 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Processing, 13(9):1200-1212.

[2] Ersotelos, N., Dong, F., 2007. Building highly realistic facial modeling and animation: a survey. The Visual Computer, 24(1):13-30.

[3] Gross, R., Matthews, I., Baker, S., 2004. Constructing and Fitting Active Appearance Models with Occlusion. Conf. on Computer Vision and Pattern Recognition Workshop, p.72.

[4] Hwang, B.W., Lee, S.W., 2003. Reconstruction of partially damaged face images based on a morphable face model. IEEE Trans. Pattern Anal. Mach. Intell., 25(3):365-372.

[5] Lu, J., Plataniotis, K.N., Venetsanopoulos, A.N., Li, S.Z., 2006. Ensemble-based discriminant learning with boosting for face recognition. IEEE Trans. Neural Networks, 17(1):166-178.

[6] Martínez, A., 1999. Face Image Retrieval Using HMMs. Proc. IEEE Workshop on Content-based Access of Image and Video Libraries, p.35-39.

[7] Mo, Z.Y., Lewis, J.P., Neumann, U., 2004. Face Inpainting with Local Linear Representations. British Machine Vision Conf., London, UK.

[8] Nielsen, F., Nock, R., 2005. ClickRemoval: Interactive Pinpoint Image Object Removal. Proc. 13th Annual ACM Int. Conf. on Multimedia, p.315-318.

[9] Oda, M., Kato, T., 1993. What Kinds of Facial Features Are Used in Face Retrieval. Proc. 2nd IEEE Int. Workshop on Robot and Human Communication, p.265-270.

[10] Shih, T.K., Lu, L.C., Chang, R.C., 2003. Multi-resolution Image Inpainting. Proc. IEEE Int. Conf. on Multimedia & Expo, Baltimore, USA, p.485-488.

[11] Stegmann, M.B., Ersboll, B.K., Larsen, R., 2003. FAME—a flexible appearance modeling environment. IEEE Trans. Med. Imag., 22(10):1319-1331.

[12] Sun, J., Yuan, L., Jia, J., Shum, H.Y., 2005. Image completion with structure propagation. ACM Trans. Graph., 24(3):861-868.

[13] Swets, D.L., Weng, J., 1996. Using discriminant eigenfeatures for image retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 18(8):831-836.

[14] Turk, M., Pentland, A., 1991. Eigenfaces for recognition. J. Cogn. Neurosci., 3(1):71-86.

[15] Wolberg, G., 1994. Digital Image Warping (1st Ed.). IEEE Computer Society Press, Los Alamitos, CA, USA.

[16] Zhang, L.Z., Yang, Q., Bao, T., Vronay, D., Tang, X., 2006. ImLooking: Image-based Face Retrieval in Online Dating Profile Search. ACM SIG CHI, Canada, p.1577-1582.

[17] Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A., 2003. Face recognition: a literature survey. ACM Comput. Surv., 35(4):399-458.

Open peer comments: Debate/Discuss/Question/Opinion


Please provide your name, email address and a comment

Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou 310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn
Copyright © 2000 - Journal of Zhejiang University-SCIENCE