Abstract: Under the perspective projection assumption, non-Lambertian photometric stereo is a highly non-linear problem. In this study, we present an optimized framework for reconstructing the surface normal and depth with non-Lambertian reflection models under perspective projection. By decomposing the images into diffuse and specular components, we compute the surface normal and reflectance simultaneously. We also propose a variational formulation that is robust and useful for surface reconstruction. The experiments show that our method accurately reconstructs both the surface shape and reflectance of colorful objects with non-Lambertian surfaces.

透视投影下的非朗伯光度立体技术

[1]Agrawal A, Raskar R, Nayar SK, et al., 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans Graph, 24(3):828-835.

[2]Agrawal A, Raskar R, Chellappa R, 2006a. Edge suppression by gradient field transformation using cross-projection tensors. IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.2301-2308.

[3]Agrawal A, Raskar R, Chellappa R, 2006b. What is the range of surface reconstructions from a gradient field? 9^th European Conf on Computer Vision, p.578-591.

[4]Barsky S, Petrou M, 2003. The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Trans Patt Anal Mach Intell, 25(10):1239-1252.

[5]Chandraker M, Agarwal S, Kriegman D, 2007. Shadowcuts: photometric stereo with shadows. IEEE Conf on Computer Vision and Pattern Recognition, p.1-8.

[6]Chung HS, Jia JY, 2008. Efficient photometric stereo on glossy surfaces with wide specular lobes. IEEE Conf on Computer Vision and Pattern Recognition, p.1-8.

[7]Coleman ENJr, Jain R, 1982. Obtaining 3-dimensional shape of textured and specular surfaces using four-source photometry. Comput Graph Image Process, 18(4):309-328.

[8]Donoho DL, 2006. Compressed sensing. IEEE Trans Inform Theory, 52(4):1289-1306.

[9]Esteban CH, Vogiatzis G, Cipolla R, 2008. Multiview photometric stereo. IEEE Trans Patt Anal Mach Intell, 30(3):548-554.

[10]Finlayson GD, Schaefer G, 2001a. Convex and non-convex illuminant constraints for dichromatic colour constancy. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.598-604

[11]Finlayson GD, Schaefer G, 2001b. Solving for colour constancy using a constrained dichromatic reflection model. Int J Comput Vis, 42(3):127-144.

[12]Gabay D, Mercier B, 1976. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput Math Appl, 2(1):17-40.

[13]Galliani S, Lasinger K, Schindler K, 2015. Massively parallel multiview stereopsis by surface normal diffusion. Proc IEEE Int Conf on Computer Vision, p.873-881.

[14]Georghiades AS, 2003. Incorporating the Torrance and Sparrow model of reflectance in uncalibrated photometric stereo. Proc 9^th IEEE Int Conf on Computer Vision, Article 816.

[15]Ghosh A, Chen TB, Peers P, et al., 2009. Estimating specular roughness and anisotropy from second order spherical gradient illumination. Comput Graph For, 28(4):1161-1170.

[16]Goldman DB, Curless B, Hertzmann A, et al., 2010. Shape and spatially-varying BRDFs from photometric stereo. IEEE Trans Patt Anal Mach Intell, 32(6):1060-1071.

[17]Guo XJ, Cao XC, Ma Y, 2014. Robust separation of reflection from multiple images. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2187-2194.

[18]Han BJ, Sim JY, 2018. Glass reflection removal using co-saliency-based image alignment and low-rank matrix completion in gradient domain. IEEE Trans Image Process, 27(10):4873-4888.

[19]Han TQ, Shen HL, 2015. Photometric stereo for general BRDFs via reflection sparsity modeling. IEEE Trans Image Process, 24(12):4888-4903.

[20]He XT, Peng YX, 2019. Fine-grained visual-textual representation learning. IEEE Trans Circ Syst Video Technol, online.

[21]He XT, Peng YX, Zhao JJ, 2019a. Fast fine-grained image classification via weakly supervised discriminative localization. IEEE Trans Circ Syst Video Technol, 29(5):1394-1407.

[22]He XT, Peng YX, Zhao JJ, 2019b. Which and how many regions to gaze: focus discriminative regions for fine-grained visual categorization. Int J Comput Vis, 127(9):1235-1255.

[23]Huynh CP, Robles-Kelly A, 2010. A solution of the dichromatic model for multispectral photometric invariance. Int J Comput Vis, 90(1):1-27.

[24]Ikehata S, 2018. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. European Conf on Computer Vision, p.3-19.

[25]Ikehata S, Aizawa K, 2014. Photometric stereo using constrained bivariate regression for general isotropic surfaces. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2179-2186.

[26]Ikehata S, Wipf D, Matsushita Y, et al., 2012. Robust photometric stereo using sparse regression. IEEE Conf on Computer Vision and Pattern Recognition, p.318-325.

[27]Kong N, Tai YW, Shin SY, 2012. A physically-based approach to reflection separation. IEEE Conf on Computer Vision and Pattern Recognition, p.9-16.

[28]Levin A, Weiss Y, 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans Patt Anal Mach Intell, 29(9):1647-1654.

[29]Li Y, Brown MS, 2013. Exploiting reflection change for automatic reflection removal. Proc IEEE Int Conf on Computer Vision, p.2432-2439.

[30]Mallick SP, Zickler TE, Kriegman DJ, et al., 2005. Beyond Lambert: reconstructing specular surfaces using color. IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.619-626.

[31]Nayar SK, Fang XS, Boult T, 1997. Separation of reflection components using color and polarization. Int J Comput Vis, 21(3):163-186.

[32]Papadhimitri T, Favaro P, 2013. A new perspective on uncalibrated photometric stereo. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1474-1481.

[33]Peng YX, He XT, Zhao JJ, 2018. Object-part attention model for fine-grained image classification. IEEE Trans Image Process, 27(3):1487-1500.

[34]Quéau Y, Wu T, Lauze F, et al., 2017. A non-convex variational approach to photometric stereo under inaccurate lighting. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.350-359.

[35]Quéau Y, Durix B, Wu T, et al., 2018. LED-based photometric stereo: modeling, calibration and numerical solution. J Math Imag Vis, 60(3):313-340.

[36]Santo H, Samejima M, Sugano Y, et al., 2017. Deep photometric stereo network. Proc IEEE Int Conf on Computer Vision, p.501-509.

[37]Schechner YY, Kiryati N, Basri R, 2000. Separation of transparent layers using focus. Int J Comput Vis, 39(1):25-39.

[38]Shafer SA, 1985. Using color to separate reflection components. Color Res Appl, 10(4):210-218.

[39]Shen HL, Han TQ, Li CG, 2017. Efficient photometric stereo using kernel regression. IEEE Trans Image Process, 26(1):439-451.

[40]Shi BX, Tan P, Matsushita Y, et al., 2014. Bi-polynomial modeling of low-frequency reflectances. IEEE Trans Patt Anal Mach Intell, 36(6):1078-1091.

[41]Shi BX, Mo ZP, Wu Z, et al., 2019. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Trans Patt Anal Mach Intell, 41(2):271-284.

[42]Shih Y, Krishnan D, Durand F, et al., 2015. Reflection removal using ghosting cues. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.3193-3201.

[43]Solomon F, Ikeuchi K, 1996. Extracting the shape and roughness of specular lobe objects using four light photometric stereo. IEEE Trans Patt Anal Mach Intell, 18(4):449-454.

[44]Sun C, Liu SC, Yang TT, et al., 2016. Automatic reflection removal using gradient intensity and motion cues. Proc 24^th ACM Int Conf on Multimedia, p.466-470. http://doi.org/10.1145/2964284.2967264

[45]Sunkavalli K, Zickler T, Pfister H, 2010. Visibility subspaces: uncalibrated photometric stereo with shadows. 11^th European Conf on Computer Vision, p.251-264.

[46]Tan RT, Ikeuchi K, 2008. Separating reflection components of textured surfaces using a single image. IEEE Trans Patt Anal Mach Intell, 27(2):353-384. https://10.1109/TPAMI.2005.36

[47]Tankus A, Kiryati N, 2005. Photometric stereo under perspective projection. 10^th IEEE Int Conf on Computer Vision, p.611-616.

[48]Wan RJ, Shi BX, Hwee TA, et al., 2016. Depth of field guided reflection removal. IEEE Int Conf on Image Processing, p.21-25.

[49]Wan RJ, Shi BX, Duan LY, et al., 2018. CRRN: multi-scale guided concurrent reflection removal network. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4777-4785.

[50]Wang F, Ainouz S, Petitjean C, et al., 2017. Specularity removal: a global energy minimization approach based on polarization imaging. Comput Vis Image Underst, 158:31-39.

[51]Woodham RJ, 1979. Photometric stereo: a reflectance map technique for determining surface orientation from image intensity. 22^nd Annual Technical Symp on Image Understanding Systems and Industrial Applications I, p.136-143.

[52]Wu L, Ganesh A, Shi B, et al., 2010. Robust photometric stereo via low-rank matrix completion and recovery. 10^th Asian Conf on Computer Vision, p.703-717.

[53]Yang J, Gong D, Liu LQ, et al., 2018. Seeing deeply and bidirectionally: a deep learning approach for single image reflection removal. Proc European Conf on Computer Vision, p.654-669.

[54]Yang QX, Tang JH, Ahuja N, 2015. Efficient and robust specular highlight removal. IEEE Trans Patt Anal Mach Intell, 37(6):1304-1311.

[55]Yeung SK, Wu TP, Tang CK, et al., 2015. Normal estimation of a transparent object using a video. IEEE Trans Patt Anal Mach Intell, 37(4):890-897.

[56]Yu C, Seo Y, Lee SW, 2010. Photometric stereo from maximum feasible Lambertian reflections. 11^th European Conf on Computer Vision, p.115-126.

[57]Zhang XE, Ng R, Chen QF, 2018. Single image reflection separation with perceptual losses. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4786-4794.

[58]Zhou ZL, Wu Z, Tan P, 2013. Multi-view photometric stereo with spatially varying isotropic materials. IEEE Conf on Computer Vision and Pattern Recognition, p.1482-1489.