access icon free View synthesis method for 3D video coding based on temporal and inter view correlation

Recently, in three-dimensional (3D) television, the temporal correlation between consecutive frames of the intermediate view is used together with the inter-view correlation to improve the quality of the synthesised view. However, most temporal methods are based on the motion vector fields (MVFs) calculated by the optical flow or block-based motion estimation which has very high computational complexity. To alleviate this issue, the authors propose a temporal-disparity-based view synthesis (TDVS) method, which uses the MVFs extracted from the bitstreams of side views and motion warping technique to create the temporal correlation between views in the intermediate position. Then a motion compensation technique is used to create a temporal-based view. Finally, the temporal-based view is fused with a disparity-based view which is generated by a traditional depth image-based rendering technique to create the final synthesised view. The fusion of these views is performed based on the side information which is determined and encoded at the sender-side of the 3D video system using a dynamic programming algorithm and rate-distortion optimisation scheme. Experimental results show that the proposed method can achieve the synthesised view with appreciable improvements in comparison with the view synthesis reference software 1D fast (VSRS-1D Fast) for several test sequences.

Inspec keywords: dynamic programming; image fusion; motion estimation; rate distortion theory; rendering (computer graphics); computational complexity; feature extraction; three-dimensional television; image sequences; video coding

Other keywords: three-dimensional television; motion vector fields; 3D video system; block-based motion estimation; computational complexity; 3D television; dynamic programming algorithm; rate-distortion optimisation scheme; intermediate view; MVF; inter view correlation; temporal correlation; depth image-based rendering technique; optical flow; 3D video coding; motion warping technique; temporal-disparity-based view synthesis method; temporal-based view fusion

Subjects: Graphics techniques; Television and video equipment, systems and applications; Optimisation techniques; Image and video coding; Video signal processing; Optimisation techniques; Computer vision and image processing techniques; Image recognition; Sensor fusion

References

    1. 1)
      • 10. Daribo, I., Pesquet-Popescu, B.: ‘Depth-aided image inpainting for novel view synthesis’. 2010 IEEE Int. Workshop on Multimedia Signal Processing (MMSP), Saint Malo, France, 2010, pp. 167170.
    2. 2)
      • 12. Chen, K.Y., Tsung, P.K., Lin, P.C., et al: ‘Hybrid motion/depth-oriented inpainting for virtual view synthesis in multiview applications’. 2010 3DTV Conf.: The True Vision –Capture, Transmission and Display of 3D Video, Tampere, Finland, 2010, pp. 14.
    3. 3)
      • 15. Le Dinh, M., Tung, L.V., Van, X.H., et al: ‘Improving 3D-TV view synthesis using motion compensated temporal interpolation’. 2016 Int. Conf. on Advanced Technologies for Communications (ATC), Hanoi, Vietnam, 2016, pp. 312317.
    4. 4)
      • 11. Purica, A.I., Mora, E.G., Pesquet-Popescu, B., et al: ‘Improved view synthesis by motion warping and temporal hole filling’. 2015 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia, 2015, pp. 11911195.
    5. 5)
      • 4. Yang, L., Yendo, T., Tehrani, M.P., et al: ‘Artifact reduction using reliability reasoning for image generation of FTV’, J. Vis. Commun. Image Represent., 2010, 21, (5–6), pp. 542560.
    6. 6)
      • 22. Stockhammer, T., Kontopodis, D., Wiegand, T.: ‘Rate-distortion optimization for JVT/H. 26L video coding in packet loss environment’, 2002.
    7. 7)
      • 14. Purica, A.I., Mora, E.G., Pesquet-Popescu, B., et al: ‘Multiview plus depth video coding with temporal prediction view synthesis’, IEEE Trans. Circuits Syst. Video Technol., 2016, 26, (2), pp. 360374.
    8. 8)
      • 8. Guillemot, C., Le Meur, O.: ‘Image inpainting: overview and recent advances’, IEEE Signal Process. Mag., 2014, 31, (1), pp. 127144.
    9. 9)
      • 5. Lee, C., Ho, Y.S.: ‘Boundary filtering on synthesized views of 3D video’. Second Int. Conf. on Future Generation Communication and Networking Symposia, Sanya, China, 2008, pp. 1518.
    10. 10)
      • 16. Liu, C., Freeman, W.T., Adelson, E.H., et al: ‘Beyond pixels: exploring new representations and applications for motion analysis’. PhD thesis, Massachusetts Institute of Technology, 2009.
    11. 11)
      • 9. Bertalmio, M., Sapiro, G., Caselles, V., et al: ‘Image inpainting’. Proc. 27th Annual Conf. on Computer Graphics and Interactive Techniques, New Orleans, USA, 2000, pp. 417424.
    12. 12)
      • 13. Purica, A.I., Cagnazzo, M., Pesquet-Popescu, B., et al: ‘View synthesis based on temporal prediction via warped motion vector fields’. 2016 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 2016, pp. 11501154.
    13. 13)
      • 1. Smolic, A., Mueller, K., Merkle, P., et al: ‘3D video and free viewpoint video- technologies, applications and MPEG standards’. 2006 IEEE Int. Conf. on Multimedia and Expo, Toronto, Canada, 2006, pp. 21612164.
    14. 14)
      • 23. Wiegand, T., Girod, B.: ‘Lagrange multiplier selection in hybrid video coder control’. Proc. 2001 Int. Conf. on Image Processing, Thessaloniki, Greece, 2001, pp. 542545.
    15. 15)
      • 27. Hore, A., Ziou, D.: ‘Image quality metrics: PSNR vs. SSIM’. 2010 20th Int. Conf. on Pattern Recognition (ICPR), Istanbul, Turkey, 2010, pp. 23662369.
    16. 16)
      • 6. Mueller, K., Smolic, A., Dix, K., et al: ‘View synthesis for advanced 3D video systems’, EURASIP J. Image Video Process., 2009, 2008, (1), p. 438148.
    17. 17)
      • 18. Kim, I.K., Min, J., Lee, T., et al: ‘Block partitioning structure in the HEVC standard’, IEEE Trans. Circuits Syst. Video Technol., 2012, 22, (12), pp. 16971706.
    18. 18)
      • 17. Sze, V., Budagavi, M., Sullivan, G.J.: ‘High efficiency video coding (HEVC)’, in ‘Integrated circuit and systems, algorithms and architectures’ (Springer, Berlin Heidelberg, 2014), 39, p. 40.
    19. 19)
      • 3. Fehn, C.: ‘Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV’, Proc. SPIE Electronic Engineering, San Jose, USA, 2004, pp. 93105.
    20. 20)
      • 20. Di Stefano, L., Bulgarelli, A.: ‘A simple and efficient connected components labeling algorithm’. Proc. Int. Conf. on Image Analysis and Processing, 1999, Venice, Italy, 1999, pp. 322327.
    21. 21)
      • 19. Zhu, C., Zhao, Y., Yu, L., et al: ‘3D-TV system with depth-image-based rendering’ (Springer-Verlag, New York, 2014).
    22. 22)
      • 2. Smolic, A., Mueller, K., Merkle, P., et al: ‘Multi-view video plus depth (MVD) format for advanced 3D video systems’, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q, 2007, 6, p. 2127.
    23. 23)
      • 24. Chen, Y., Tech, G., Wegner, K., et al: ‘Test model 11 of 3D-HEVC and MV-HEVC’, JCT-3V Doc., JCT3V-J1003, Geneva, CH, 2015.
    24. 24)
      • 26. Sullivan, G.J., Ohm, J., Han, W.J., et al: ‘Overview of the high efficiency video coding (HEVC) standard’, IEEE Trans. Circuits Syst. Video Technol., 2012, 22, (12), pp. 16491668.
    25. 25)
      • 7. Criminisi, A., Pérez, P., Toyama, K.: ‘Region filling and object removal by exemplar-based image inpainting’, IEEE Trans. Image Process., 2004, 13, (9), pp. 12001212.
    26. 26)
      • 21. Sullivan, G.J., Wiegand, T.: ‘Rate-distortion optimization for video compression’, IEEE Signal Process. Mag., 1998, 15, (6), pp. 7490.
    27. 27)
      • 25. Rusanovskyy, D., Müller, K., Vetro, A.: ‘Common test conditions of 3DV core experiments’, ITU-T SG, 2013, p. 16.
    28. 28)
      • 28. Bjontegarrd, G.: ‘Calculation of average PSNR differences between RD-curves’, VCEG-M33, 2001.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.5390
Loading

Related content

content/journals/10.1049/iet-ipr.2018.5390
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading