access icon free Video retargeting through spatio-temporal seam carving using Kalman filter

© The Institution of Engineering and Technology
Received 26/02/2019, Accepted 29/05/2019, Revised 28/04/2019, Published 04/06/2019

A Kalman filter-based spatio-temporal seam determination scheme is proposed in this study for video retargeting through seam carving. In probably a first, the authors’ retargeting approach is designed to be hardware friendly, and to achieve spatial and temporal coherence through an optimal solution with explicit mechanisms to reduce jitter and structural distortion. In a video frame, while spatially coherent seam is determined by the popular approach of seam energy minimisation, temporally coherent seam is determined using Kalman prediction and updation processes. Further, these spatial and temporal seams are combined judiciously to obtain the spatio-temporal seam to be removed/repeated for decreasing/increasing the frame size. The authors show that the proposed Kalman filter-based approach has less theoretical complexity compared to the existing. Extensive experimental results show that the proposed approach consistently outperforms the state-of-the-art, both qualitatively and quantitatively in performance, and in computational time.

Inspec keywords: video signal processing; Kalman filters; jitter

Other keywords: authors; seam energy minimisation; spatial seams; seam carving; updation processes; temporally coherent seam; video retargeting; Kalman filter-based spatio-temporal seam determination scheme; video frame; Kalman filter-based approach; spatial coherence; spatially coherent seam; temporal seams; temporal coherence

Subjects: Video signal processing; Optical, image and video signal processing; Filtering methods in signal processing; Computer vision and image processing techniques

References

    1. 1)
      • 17. Graetzel, C.F., Nelson, B.J., Fry, S.N.: ‘A dynamic region-of-interest vision tracking system applied to the real-time wing kinematic analysis of tethered drosophila’, IEEE Trans. Autom. Sci. Eng., 2010, 7, (3), pp. 463–473.
    2. 2)
      • 34. Wang, B., Xiong, H., Ren, Z., et al: ‘Deformable shape preserving video retargeting with salient curve matching’, IEEE J. Emerg. Sel. Top. Circuits Syst., 2014, 4, (1), pp. 82–94.
    3. 3)
      • 26. Wang, Y.-S., Lin, H.-C., Sorkine, O., et al: ‘Motion-based video retargeting with optimized crop-and-warp’, ACM Trans. Graph. (Proc. ACM SIGGRAPH), 2010, 29, (4), pp. 90:1–90:9, Article 90, Doi: 10.1145/1778765.1778827.
    4. 4)
      • 19. Liu, F., Gleicher, M.: ‘Video retargeting: automating pan and scan’. Proc. ACM Int. Conf. on Multimedia, Santa Barbara, USA, 2006, pp. 241–250.
    5. 5)
      • 13. Grewal, M.S., Andrews, A.P.: ‘Kalman filtering: theory and practice with MATLAB’ (Wiley-Blackwell, New Jersey, USA, 2015, 4th edn.).
    6. 6)
      • 20. Wolf, L., Guttmann, M., Cohen-Or, D.: ‘Non-homogeneous content-driven video-retargeting’. Proc. IEEE Int. Conf. on Computer Vision (ICCV), Rio De Janeiro, Brazil, 2007, pp. 1–6.
    7. 7)
      • 39. Li, B., Lin, C.-W., Shi, B., et al: ‘Depth-aware stereo video retargeting’. Int. Conf. on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 6517–6525.
    8. 8)
      • 29. Chao, W.-L., Su, H.-H., Chien, S.-Y., et al: ‘Coarse-to-fine temporal optimization for video retargeting based on seam carving’. Proc. IEEE Int. Conf. Multimedia Expo (ICME), Barcelona, Spain, 2011, pp. 1–6.
    9. 9)
      • 36. Zhang, L., Wang, M., Nie, L., et al: ‘Retargeting semantically-rich photos’, IEEE Trans. Multimed., 2015, 17, (9), pp. 1538–1549.
    10. 10)
      • 7. Gal, R., Sorkine, O., Cohen-Or, D.: ‘Feature-aware texturing’. Proc. Eurographics Symp. on Rendering, Nicosia, Cyprus, 2006, pp. 297–303.
    11. 11)
      • 51. Hsu, C., Lin, C.: ‘Objective quality assessment for video retargeting based on spatio-temporal distortion analysis’. Proc. IEEE Int. Conf. on Visual Communications and Image Processing, St. Petersburg, USA, 2017, pp. 1–4.
    12. 12)
      • 40. Kang, K., Cao, Y., Wang, Z.: ‘Perceptually aware image retargeting for mobile devices’, IEEE Trans. Image Process., 2018, 27, (5), pp. 2301–2313.
    13. 13)
      • 48. Sen, D., Swamy, M.N.S., Ahmad, M.O.: ‘Computationally fast techniques to reduce AWGN and speckle in videos’, IET Image Process., 2007, 1, (4), pp. 319–334.
    14. 14)
      • 41. Zhang, L., Li, K., Ou, Z., et al: ‘Seam warping: a new approach for image retargeting for small displays’, Soft Comput., 2017, 21, (2), pp. 447–457.
    15. 15)
      • 50. Garcia-Diaz, A., Fdez-Vidal, X.R., Pardo, X.M., et al: ‘Saliency from hierarchical adaptation through decorrelation and variance normalization’, Image Vis. Comput., 2012, 30, (1), pp. 51–64.
    16. 16)
      • 47. Faragher, R.: ‘Understanding the basis of the Kalman filter via a simple and intuitive derivation’, IEEE Trans. Signal Process., 2012, 29, (5), pp. 128–132.
    17. 17)
      • 49. del Rincon, J.M., Makris, D., Urunuela, C.O., et al: ‘Tracking human position and lower body parts using Kalman and particle filters constrained by human biomechanics’, IEEE Trans. Syst. Man. Cybern., Part B, 2011, 41, (1), pp. 26–37.
    18. 18)
      • 25. Kim, J.S., Kim, J.H., Kim, C.S.: ‘Adaptive image and video retargeting technique based on Fourier analysis’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Miami, USA, 2009, pp. 1730–1737.
    19. 19)
      • 28. Hu, Y., Rajan, D.: ‘Hybrid shift map for video retargeting’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, San Francisco, USA, 2010, pp. 577–584.
    20. 20)
      • 46. Kaur, H., Kour, S., Sen, D.: ‘Prediction-based seam carving for video retargeting’. Proc. Int. Conf. on Pattern Recognition, Cancun, Mexico, 2016, pp. 877–882.
    21. 21)
      • 33. Li, B., Duan, L. Y., Wang, J., et al: ‘Spatiotemporal grid flow for video retargeting’, IEEE Trans. Image Process., 2014, 23, (4), pp. 1615–1628.
    22. 22)
      • 18. Liu, F., Gleicher, M.: ‘Automatic image retargeting with fisheye-view warping’. Proc. Annual ACM Symp. on User Interface Software and Technology, Seattle, USA, 2005, pp. 153–162.
    23. 23)
      • 9. Rubinstein, M., Shamir, A., Avidan, S.: ‘Improved seam carving for video retargeting’, ACM Trans. Graph. (Proc. ACM SIGGRAPH), 2008, 27, (3), pp. 16:1–16:9Article No. 16, Doi: 10.1145/1360612.1360615.
    24. 24)
      • 38. Kang, K., Cao, Y., Wang, Z.: ‘Simultaneously retargeting and super-resolution for stereoscopic video’, Multimedia Tools Appl., 2017, 76, (8), pp. 11081–11095.
    25. 25)
      • 42. Kohata, Y., Desaki, Y.: ‘Performance evaluation of hardware-oriented seam carving algorithm’. IEEE Global Conf. on Consumer Electronics, Tokyo, Japan, 2014, pp. 315–316.
    26. 26)
      • 45. Conge, D.D., Kumar, M., Miller, R.L., et al: ‘Improved seam carving for image resizing’. IEEE Workshop on Signal Processing Systems, San Francisco, USA, 2010, pp. 345–349.
    27. 27)
      • 27. Lu, T., Yuan, Z., Huang, Y., et al: ‘Video retargeting with nonlinear spatial–temporal saliency fusion’. Proc. IEEE Int. Conf. on Image Processing, Hong Kong, China, 2010, pp. 1801–1804.
    28. 28)
      • 1. Todd, M.: ‘Systems and methods for video processing, combination and display of heterogeneous sources’, US Patent: US20180316939A1, 2018.
    29. 29)
      • 31. Yen, T. C., Tsai, C. M., Lin, C.W.: ‘Maintaining temporal coherence in video retargeting using mosaic-guided scaling’, IEEE Trans. Image Process., 2011, 20, (8), pp. 2339–2351.
    30. 30)
      • 2. Liang, Y., Su, Z., Wang, C., et al: ‘Optimised image retargeting using aesthetic-based cropping and scaling’, IET Image Process., 2013, 7, (1), pp. 61–69.
    31. 31)
      • 6. Guo, Y., Liu, F., Shi, J., et al: ‘Image retargeting using mesh parametrization’, IEEE Trans. Multimed., 2009, 11, (5), pp. 856–867.
    32. 32)
      • 12. Greisen, P., Lang, M., Heinzle, S., et al: ‘Algorithm and VLSI architecture for real-time 1080p60 video retargeting’. Proc. of ACM SIGGRAPH/Eurographics Conf. on High-Performance Graphics, Paris, France, 2012, pp. 57–66.
    33. 33)
      • 15. Roy, A., Mitra, D.: ‘Multi-target trackers using cubature Kalman filter for Doppler radar tracking in clutter’, IET Signal Process., 2016, 10, (8), pp. 888–901.
    34. 34)
      • 37. Su, Y.-C., Jayaraman, D., Grauman, K.: ‘Pano2vid: automatic cinematography for watching 360° videos’. Asian Conf. on Computer Vision, Taipei, Taiwan, 2016, pp. 154–171.
    35. 35)
      • 11. Yan, B., Sun, K., Liu, L.: ‘Matching-area-based seam carving for video retargeting’, IEEE Trans. Circuits Syst. Video Technol., 2013, 23, (2), pp. 302–310.
    36. 36)
      • 14. Deergha Rao, K., Swamy, M.N.S., Plotkin, E.I.: ‘Adaptive filtering approaches for colour image and video restoration’, IEE Proc., Vis. Image Signal Process., 2003, 150, (3), pp. 168–177.
    37. 37)
      • 8. Zhang, G.-X., Cheng, M.-M., Hu, S.-M., et al: ‘A shape-preserving approach to image resizing’, Comput. Graph. Forum, 2009, 28, (7), pp. 1897–1906.
    38. 38)
      • 32. Lin, S.S., Lin, C.H., Yeh, I.C., et al: ‘Content-aware video retargeting using object-preserving warping’, IEEE Trans. Vis. Comput. Graph., 2013, 19, (10), pp. 1677–1686.
    39. 39)
      • 5. Vaquero, D., Turk, M., Pulli, K., et al: ‘A survey of image retargeting techniques’. Proc. SPIE-7798, 2010, pp. 14–1–14-15.
    40. 40)
      • 43. Itti, L., Koch, C., Niebur, E.: ‘A model of saliency-based visual attention for rapid scene analysis’, IEEE Trans. Pattern Anal. Mach. Intell., 1998, 20, (1), pp. 1254–1259.
    41. 41)
      • 44. Mylona, E.A., Savelonas, M.A., Maroulis, D.: ‘Entropy-based spatially-varying adjustment of active contour parameters’. Proc. of IEEE Int. Conf. Image Processing, Orlando, USA, 2012, pp. 2565–2568.
    42. 42)
      • 35. Yan, B., Yuan, B., Yang, B.: ‘Effective video retargeting with jittery assessment’, IEEE Trans. Multimed., 2014, 16, (1), pp. 272–277.
    43. 43)
      • 30. Wang, S. F., Lai, S.H.: ‘Compressibility-aware media retargeting with structure preserving’, IEEE Trans. Image Process., 2011, 20, (3), pp. 855–865.
    44. 44)
      • 10. Grundmann, M., Kwatra, V., Han, M., et al: ‘Discontinuous seam carving for video retargeting’. Proc. IEEE Conf. Computer Vision Pattern Recognition (CVPR), San Francisco, USA, 2010, pp. 569–576.
    45. 45)
      • 24. Rubinstein, M., Shamir, A., Avidan, S.: ‘Multioperator media retargeting’, ACM Trans. Graph. (Proc. ACM SIGGRAPH), 2009, 28, (3), pp. 23:1–23:11, Article 23, Doi: 10.1145/1576246.1531329.
    46. 46)
      • 22. Zhang, Y.F., Hu, S.M., Martin, R.R.: ‘Shrinkability maps for content-aware video resizing’, Comput. Graph. Forum, 2008, 27, (7), pp. 1797–1804.
    47. 47)
      • 4. Avidan, S., Shamir, A.: ‘Seam carving for content-aware image resizing’, ACM Trans. Graph. (Proc. of SIGGRAPH), 2007, 26, (3), pp. 10:1–10:10, Article No. 10.
    48. 48)
      • 3. Hsin, H.: ‘Saliency histogram equalisation and its application to image resizing’, IET Image Process., 2016, 10, (10), pp. 787–798.
    49. 49)
      • 21. Tao, C., Jia, J., Sun, H.: ‘Active window oriented dynamic video retargeting’. Proc. ICCV Workshop on Dynamical Vision, Rio De Janeiro, Brazil, 2007.
    50. 50)
      • 16. Ricco, M., Manganiello, P., Monmasson, E., et al: ‘FPGA-based implementation of dual Kalman filter for PV MPPT applications’, IEEE Trans. Ind. Inf., 2017, 13, (1), pp. 176–185.
    51. 51)
      • 23. Wang, Y.-S., Fu, H., Sorkine, O., et al: ‘Motion-aware temporal coherence for video resizing’. Proc. ACM SIGGRAPH Asia, Yokohama, Japan, 2009, p. 127, 1–10.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2019.0236
Loading

Related content

content/journals/10.1049/iet-ipr.2019.0236
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading