© The Institution of Engineering and Technology
Although there is an increasing interest in employing the depth data in computer vision applications, the spatial resolution of depth maps is still limited compared with typical visible-light images. A novel method is proposed to synthetically improve the spatial resolution of a single depth image. It integrates the higher-order terms into the Markov random field (MRF) formulation of example-based methods in order to improve the representational power of those methods. The inference is performed by approximately minimising the higher-order multi-label MRF energies. In addition, to improve the efficiency of the inference algorithm, a hierarchical scheme on the number of MRF states is proposed. First, a large number of states are used to obtain an initial labelling by solving the minimisation problem of inference for only the first-order energies. Then, the problem is solved for the higher-order energies in a smaller number of states. Performance comparisons show that proposed method improves the results of first-order approaches that are based on simple four-connected MRF graph structure, both qualitatively and quantitatively.
References
-
-
1)
-
18. Xie, J., Chou, C.C., Feris, R., et al: ‘Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering’, IEEE Int. Conf. Multimedia and Expo (ICME), 2014, pp. 1–6.
-
2)
-
22. Tang, Y., Chen, H., Liu, Z., et al: ‘Example-based super-resolution via social images’, Neurocomputing, 2016, 172, pp. 38–47.
-
3)
-
34. Wang, C., Komodakis, N., Paragios, N.: ‘Markov random field modeling, inference & learning in computer vision & image understanding: a survey’, Comput. Vis. Image Underst., 2013, 117, pp. 1610–1627.
-
4)
-
36. Scharstein, D., Szeliski, R.: ‘A taxonomy and evaluation of dense two-frame stereo correspondence algorithms’, Int. J. Comput. Vis., 2002, 47, (1), pp. 7–42.
-
5)
-
24. Chen, Y.: ‘Higher-order MRFs based image super resolution: MMSE or MAP?’ (CoRR, 2014).
-
6)
-
19. Xie, J., Feris, R.S., Yu, S.S., et al: ‘Joint super resolution and denoising from a single depth image’, IEEE Trans. Multimed., 2015, 17, pp. 1525–1537.
-
7)
-
6. Fix, A., Gruber, A., Boros, E., et al: ‘A graph cut algorithm for higher-order Markov random fields’, IEEE Int. Conf. Computer Vision (ICCV), 2011, pp. 1020–1027.
-
8)
-
4. Mac Aodha, O., Campbell, N.D., Nair, A., et al: ‘Patch based synthesis for single depth image super-resolution’, European Conference on Computer Vision (Springer, Berlin, Heidelberg, 2012), pp. 71–84.
-
9)
-
8. Kolmogorov, V., Rother, C.: ‘Minimizing nonsubmodular functions with graph cuts-a review’, Pattern IEEE Trans. Anal. Mach. Intell., 2007, 29, (7), pp. 1274–1279.
-
10)
-
30. Choi, O., Jung, S.W.: ‘A consensus-driven approach for structure and texture aware depth map upsamplingvn’, IEEE Trans. Image Process., 2014, 23, pp. 3321–3335.
-
11)
-
33. Al Ismaeil, K., Aouada, D., Mirbach, B., et al: ‘Enhancement of dynamic depth scenes by upsampling for precise super-resolution (UP-SR)’ (Computer Vision and Image Understanding, 2016).
-
12)
-
14. Schuon, S., Theobalt, C., Davis, J., et al: ‘Lidarboost: depth superresolution for of 3d shape scanning’, IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 343–350.
-
13)
-
23. Xie, J., Feris, R.S., Sun, M.T.: ‘Edge-guided single depth image super resolution’, IEEE Trans. Image Process., 2016, 25, pp. 428–438.
-
14)
-
10. Tian, J., Ma, K.K.: ‘A survey on super-resolution imaging’, Signal Image Video Process., 2011, 5, (3), pp. 329–342.
-
15)
-
5. Ishikawa, H.: ‘Transformation of general binary MRF minimization to the first-order case’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (6), pp. 1234–1249.
-
16)
-
31. Yang, Y., Gao, M., Zhang, J., et al: ‘Depth map super-resolution using stereo-vision-assisted model’, Neurocomputing, 2015, 149, pp. 1396–1406.
-
17)
-
32. Al Ismaeil, K.: ‘Super-resolution approaches for depth video enhancement’ (University of Luxembourg, Bertrange, Luxembourg, 2015).
-
18)
-
38. Wang, Z., Bovik, A.C., Sheikh, H.R., et al: ‘Image quality assessment: from error visibility to structural similarity’, IEEE Trans. Image Process., 2004, 13, pp. 600–612.
-
19)
-
21. Wang, Q., Tang, X., Shum, H.: ‘Patch based blind image super resolution’, IEEE Int. Conf. Computer Vision (ICCV), October 2005, 1, pp. 709–716.
-
20)
-
37. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution via sparse representation’, IEEE Trans. Image Process., 2010, 19, (11), pp. 2861–2873.
-
21)
-
2. Freeman, W., Liu, C.: ‘Markov random fields for super-resolution and texture synthesis’, Adv. Markov Random Fields Vis. Image Process., 2011, 1, pp. 155–165..
-
22)
-
20. Sun, J., Zheng, N.N., Tao, H., et al: ‘Image hallucination with primal sketch priors’, IEEE Computer Society Conf. Computer Vision and Pattern Recognition, 2003, 2, pp. II–729.
-
23)
-
35. Kolmogorov, V.: ‘Convergent tree-reweighted message passing for energy minimization’, IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28, (10), pp. 1568–1583.
-
24)
-
29. Li, Y., Xue, T., Sun, L., et al: ‘Joint example-based depth map super-resolution’, Multimedia and Expo (ICME), 2012 IEEE International Conference on, (IEEE, 2012), pp. 152–157.
-
25)
-
15. Izadi, S., Kim, D., Hilliges, O., et al: ‘KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera’, Proc. 24th Annual ACM Symp. User Interface Softw. Technol. (ACM, 2011), pp. 559–568.
-
26)
-
12. Yuan, Q., Zhang, L., Shen, H.: ‘Multiframe super-resolution employing a spatially weighted total variation model’, IEEE Trans. Circuits Syst. Video Technol., 2012, 22, (3), pp. 379–392.
-
27)
-
27. Kopf, J., Cohen, M.F., Lischinski, D., et al: ‘Joint bilateral upsampling’, ACM Trans. Graph., (TOG), 2007, 26, (3), p. 96.
-
28)
-
3. Freeman, W.T., Jones, T.R., Pasztor, E.C.: ‘Example-based super-resolution’, IEEE Comput. Graph. Appl., 2002, 22, (2), pp. 56–65.
-
29)
-
7. Lempitsky, V., Rother, C., Roth, S., et al: ‘Fusion moves for Markov random field optimization’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (8), pp. 1392–1405.
-
30)
-
26. Yang, Q., Yang, R., Davis, J., et al: ‘Spatial-depth super resolution for range images’, Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on (IEEE, 2007), pp. 1–8.
-
31)
-
25. Roth, S., Black, M.J.: ‘Fields of experts’, Int. J. Comput. Vis., 2009, 82, (2), pp. 205–229.
-
32)
-
9. Nasrollahi, K., Moeslund, T.B.: ‘Super-resolution: a comprehensive survey’, Mach. Vis. Appl., 2014, 25, (6), pp. 1423–1468.
-
33)
-
39. Yuan, Y., Lin, J., Wang, Q.: ‘Hyperspectral image classification via multitask joint sparse representation and stepwise MRF optimization’, IEEE Trans. Cybern., 2016, 46, pp. 2966–2977.
-
34)
-
17. Zeyde, R., Elad, M., Protter, M.: ‘On single image scale-up using sparse-representations’, Int. Conf. Curves and Surfaces, Springer Berlin Heidelberg, June 2010, pp. 711–730.
-
35)
-
16. Yang, J., Wright, J., Huang, T., et al: ‘Image super-resolution as sparse representation of raw image patches’, IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8.
-
36)
-
13. Zhang, L., Yuan, Q., Shen, H., et al: ‘Multiframe image super-resolution adapted with local spatial information’, JOSA A, 2011, 28, (3), pp. 381–390.
-
37)
-
28. Lu, J., Min, D., Pahwa, R.S., et al: ‘A revisit to MRF-based depth map super-resolution and enhancement’, Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on (IEEE, 2011), pp. 985–988.
-
38)
-
11. Li, X., Hu, Y., Gao, X., et al: ‘A multi-frame image super-resolution method’, Signal Process., 2010, 90, (2), pp. 405–414.
-
39)
-
1. Xia, L., Chen, C.C., Aggarwal, J.: ‘View invariant human action recognition using histograms of 3d joints’, IEEE Computer Society Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), 2012, pp. 20–27.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2016.0373
Related content
content/journals/10.1049/iet-cvi.2016.0373
pub_keyword,iet_inspecKeyword,pub_concept
6
6