access icon free High-order Markov random field for single depth image super-resolution

Although there is an increasing interest in employing the depth data in computer vision applications, the spatial resolution of depth maps is still limited compared with typical visible-light images. A novel method is proposed to synthetically improve the spatial resolution of a single depth image. It integrates the higher-order terms into the Markov random field (MRF) formulation of example-based methods in order to improve the representational power of those methods. The inference is performed by approximately minimising the higher-order multi-label MRF energies. In addition, to improve the efficiency of the inference algorithm, a hierarchical scheme on the number of MRF states is proposed. First, a large number of states are used to obtain an initial labelling by solving the minimisation problem of inference for only the first-order energies. Then, the problem is solved for the higher-order energies in a smaller number of states. Performance comparisons show that proposed method improves the results of first-order approaches that are based on simple four-connected MRF graph structure, both qualitatively and quantitatively.

Inspec keywords: graph theory; Markov processes; computer vision; image resolution

Other keywords: single depth image superresolution; spatial resolution improvement; inference algorithm; depth data; MRF graph structure; computer vision applications; high-order Markov random field; depth maps

Subjects: Markov processes; Markov processes; Combinatorial mathematics; Combinatorial mathematics; Optical, image and video signal processing; Computer vision and image processing techniques

References

    1. 1)
      • 18. Xie, J., Chou, C.C., Feris, R., et al: ‘Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering’, IEEE Int. Conf. Multimedia and Expo (ICME), 2014, pp. 16.
    2. 2)
      • 22. Tang, Y., Chen, H., Liu, Z., et al: ‘Example-based super-resolution via social images’, Neurocomputing, 2016, 172, pp. 3847.
    3. 3)
      • 34. Wang, C., Komodakis, N., Paragios, N.: ‘Markov random field modeling, inference & learning in computer vision & image understanding: a survey’, Comput. Vis. Image Underst., 2013, 117, pp. 16101627.
    4. 4)
      • 36. Scharstein, D., Szeliski, R.: ‘A taxonomy and evaluation of dense two-frame stereo correspondence algorithms’, Int. J. Comput. Vis., 2002, 47, (1), pp. 742.
    5. 5)
      • 24. Chen, Y.: ‘Higher-order MRFs based image super resolution: MMSE or MAP?’ (CoRR, 2014).
    6. 6)
      • 19. Xie, J., Feris, R.S., Yu, S.S., et al: ‘Joint super resolution and denoising from a single depth image’, IEEE Trans. Multimed., 2015, 17, pp. 15251537.
    7. 7)
      • 6. Fix, A., Gruber, A., Boros, E., et al: ‘A graph cut algorithm for higher-order Markov random fields’, IEEE Int. Conf. Computer Vision (ICCV), 2011, pp. 10201027.
    8. 8)
      • 4. Mac Aodha, O., Campbell, N.D., Nair, A., et al: ‘Patch based synthesis for single depth image super-resolution’, European Conference on Computer Vision (Springer, Berlin, Heidelberg, 2012), pp. 7184.
    9. 9)
      • 8. Kolmogorov, V., Rother, C.: ‘Minimizing nonsubmodular functions with graph cuts-a review’, Pattern IEEE Trans. Anal. Mach. Intell., 2007, 29, (7), pp. 12741279.
    10. 10)
      • 30. Choi, O., Jung, S.W.: ‘A consensus-driven approach for structure and texture aware depth map upsamplingvn’, IEEE Trans. Image Process., 2014, 23, pp. 33213335.
    11. 11)
      • 33. Al Ismaeil, K., Aouada, D., Mirbach, B., et al: ‘Enhancement of dynamic depth scenes by upsampling for precise super-resolution (UP-SR)’ (Computer Vision and Image Understanding, 2016).
    12. 12)
      • 14. Schuon, S., Theobalt, C., Davis, J., et al: ‘Lidarboost: depth superresolution for of 3d shape scanning’, IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 343350.
    13. 13)
      • 23. Xie, J., Feris, R.S., Sun, M.T.: ‘Edge-guided single depth image super resolution’, IEEE Trans. Image Process., 2016, 25, pp. 428438.
    14. 14)
      • 10. Tian, J., Ma, K.K.: ‘A survey on super-resolution imaging’, Signal Image Video Process., 2011, 5, (3), pp. 329342.
    15. 15)
      • 5. Ishikawa, H.: ‘Transformation of general binary MRF minimization to the first-order case’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (6), pp. 12341249.
    16. 16)
      • 31. Yang, Y., Gao, M., Zhang, J., et al: ‘Depth map super-resolution using stereo-vision-assisted model’, Neurocomputing, 2015, 149, pp. 13961406.
    17. 17)
      • 32. Al Ismaeil, K.: ‘Super-resolution approaches for depth video enhancement’ (University of Luxembourg, Bertrange, Luxembourg, 2015).
    18. 18)
      • 38. Wang, Z., Bovik, A.C., Sheikh, H.R., et al: ‘Image quality assessment: from error visibility to structural similarity’, IEEE Trans. Image Process., 2004, 13, pp. 600612.
    19. 19)
      • 21. Wang, Q., Tang, X., Shum, H.: ‘Patch based blind image super resolution’, IEEE Int. Conf. Computer Vision (ICCV), October 2005, 1, pp. 709716.
    20. 20)
      • 37. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution via sparse representation’, IEEE Trans. Image Process., 2010, 19, (11), pp. 28612873.
    21. 21)
      • 2. Freeman, W., Liu, C.: ‘Markov random fields for super-resolution and texture synthesis’, Adv. Markov Random Fields Vis. Image Process., 2011, 1, pp. 155165..
    22. 22)
      • 20. Sun, J., Zheng, N.N., Tao, H., et al: ‘Image hallucination with primal sketch priors’, IEEE Computer Society Conf. Computer Vision and Pattern Recognition, 2003, 2, pp. II729.
    23. 23)
      • 35. Kolmogorov, V.: ‘Convergent tree-reweighted message passing for energy minimization’, IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28, (10), pp. 15681583.
    24. 24)
      • 29. Li, Y., Xue, T., Sun, L., et al: ‘Joint example-based depth map super-resolution’, Multimedia and Expo (ICME), 2012 IEEE International Conference on, (IEEE, 2012), pp. 152157.
    25. 25)
      • 15. Izadi, S., Kim, D., Hilliges, O., et al: ‘KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera’, Proc. 24th Annual ACM Symp. User Interface Softw. Technol. (ACM, 2011), pp. 559568.
    26. 26)
      • 12. Yuan, Q., Zhang, L., Shen, H.: ‘Multiframe super-resolution employing a spatially weighted total variation model’, IEEE Trans. Circuits Syst. Video Technol., 2012, 22, (3), pp. 379392.
    27. 27)
      • 27. Kopf, J., Cohen, M.F., Lischinski, D., et al: ‘Joint bilateral upsampling’, ACM Trans. Graph., (TOG), 2007, 26, (3), p. 96.
    28. 28)
      • 3. Freeman, W.T., Jones, T.R., Pasztor, E.C.: ‘Example-based super-resolution’, IEEE Comput. Graph. Appl., 2002, 22, (2), pp. 5665.
    29. 29)
      • 7. Lempitsky, V., Rother, C., Roth, S., et al: ‘Fusion moves for Markov random field optimization’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (8), pp. 13921405.
    30. 30)
      • 26. Yang, Q., Yang, R., Davis, J., et al: ‘Spatial-depth super resolution for range images’, Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on (IEEE, 2007), pp. 18.
    31. 31)
      • 25. Roth, S., Black, M.J.: ‘Fields of experts’, Int. J. Comput. Vis., 2009, 82, (2), pp. 205229.
    32. 32)
      • 9. Nasrollahi, K., Moeslund, T.B.: ‘Super-resolution: a comprehensive survey’, Mach. Vis. Appl., 2014, 25, (6), pp. 14231468.
    33. 33)
      • 39. Yuan, Y., Lin, J., Wang, Q.: ‘Hyperspectral image classification via multitask joint sparse representation and stepwise MRF optimization’, IEEE Trans. Cybern., 2016, 46, pp. 29662977.
    34. 34)
      • 17. Zeyde, R., Elad, M., Protter, M.: ‘On single image scale-up using sparse-representations’, Int. Conf. Curves and Surfaces, Springer Berlin Heidelberg, June 2010, pp. 711730.
    35. 35)
      • 16. Yang, J., Wright, J., Huang, T., et al: ‘Image super-resolution as sparse representation of raw image patches’, IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008, pp. 18.
    36. 36)
      • 13. Zhang, L., Yuan, Q., Shen, H., et al: ‘Multiframe image super-resolution adapted with local spatial information’, JOSA A, 2011, 28, (3), pp. 381390.
    37. 37)
      • 28. Lu, J., Min, D., Pahwa, R.S., et al: ‘A revisit to MRF-based depth map super-resolution and enhancement’, Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on (IEEE, 2011), pp. 985988.
    38. 38)
      • 11. Li, X., Hu, Y., Gao, X., et al: ‘A multi-frame image super-resolution method’, Signal Process., 2010, 90, (2), pp. 405414.
    39. 39)
      • 1. Xia, L., Chen, C.C., Aggarwal, J.: ‘View invariant human action recognition using histograms of 3d joints’, IEEE Computer Society Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), 2012, pp. 2027.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2016.0373
Loading

Related content

content/journals/10.1049/iet-cvi.2016.0373
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading