access icon free Infrared-visible video fusion based on motion-compensated wavelet transforms

Video data fusion should simultaneously take into account both temporal and spatial dimensions, and therefore a novel spatiotemporal video-fusion algorithm based on motion compensation in the wavelet-transform domain is proposed in this study. The fusion method incorporates motion compensation and the wavelet transform, thus making full use of spatial geometric information and inter-frame temporal information of input videos. The proposed method improves the temporal stability and consistency of the fused video compared to other existing individual frame-based fusion methods. The algorithm first decomposes the image frames which have been processed by an optic flow motion-compensation approach and then develops a spatiotemporal energy-based fusion rule to merge input videos. Experimental results demonstrate that the proposed fusion algorithm has superior visual and quantitative performance to traditional individual-frame-based and state-of-the-art three-dimensional-transform-based methods.

Inspec keywords: motion estimation; video signal processing; image fusion; infrared imaging; wavelet transforms

Other keywords: temporal dimensions; video data fusion; motion compensated wavelet transforms; individual frame based fusion methods; spatial geometric information; novel spatiotemporal video fusion algorithm; infrared visible video fusion; spatiotemporal energy; optic flow motion compensation approach; temporal stability; image frames; fusion method; motion compensation; interframe temporal information; spatial dimensions

Subjects: Computer vision and image processing techniques; Video signal processing; Integral transforms; Integral transforms; Optical, image and video signal processing

References

    1. 1)
      • 40. Do, M.N., Vetterli, M.: ‘The finite ridgelet transform for image representation’, IEEE Trans. Image Process., 2003, 12, (1), pp. 1628.
    2. 2)
      • 50. http://www.eeweb.poly.edu/iselesni/WaveletSoftware/index.html, accessed June 2013.
    3. 3)
      • 23. Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: ‘Highly accurate optic flow computation with theoretically justified warping’, Int. J. Comput. Vis., 2006, 67, (2), pp. 141158.
    4. 4)
      • 37. Rahman, S.M.M., Ahmad, M.O., Swamy, M.N.S.: ‘Contrast-based fusion of noisy images using discrete wavelet transform’, IET Image Process., 2010, 4, (5), pp. 374384.
    5. 5)
      • 11. Wang, Z., Li, Q.: ‘Video quality assessment using a statistical model of human visual speed perception’, J. Opt. Soc. Am. A, 2007, 24, (12), pp. B61B69.
    6. 6)
      • 7. Xiao, G., Wei, K., Jing, Z.: ‘Improved dynamic image fusion scheme for infrared and visible sequence based on image fusion system’. Proc. 11th Int. Conf. on Information Fusion, 2008, pp. 16.
    7. 7)
      • 38. Amolins, K., Zhang, Y., Dare, P.: ‘Wavelet based image fusion techniques – an introduction, review and comparison’, ISPRS J. Photogram. Remote Sens., 2007, 62, (4), pp. 249263.
    8. 8)
      • 42. Da Cunha, A.L., Zhou, J., Do, M.N.: ‘The nonsubsampled contourlet transform: theory, design, and applications’, IEEE Trans. Image Process., 2006, 15, (10), pp. 30893101.
    9. 9)
      • 34. Burt, P.J.: ‘A gradient pyramid basis for pattern selective image fusion’. Proc. of the Society for Information Display Conf., 1992, pp. 467470.
    10. 10)
      • 8. Li, J., Nikolov, S.G., Benton, C.P., Scott-Samuel, N.E.: ‘Motion-based video fusion using optical flow information’. Proc. Ninth Int. Conf. Information Fusion, Florence, Italy, July 2006, pp. 18.
    11. 11)
      • 13. Teodosio, L., Bender, W.: ‘Salient video stills: Content and context preserved’. Proc. the first ACM Int. Conf. Multimedia, Anaheim, California, 1993, pp. 3946.
    12. 12)
      • 17. Mahajan, D., Huang, F.C., Matusik, W., Ramamoorthi, R., Belhumeur, P.: ‘Moving gradients: A path-based method for plausible image interpolation’, ACM Trans. Graph., 2009, 28, (3), p. 42.
    13. 13)
      • 14. Matsushita, Y., Ofek, E., Ge, W., Tang, X., Shum, H.Y.: ‘Full-frame video stabilization with motion inpainting’, IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28, (7), pp. 11501163.
    14. 14)
      • 5. Rockinger, O.: ‘Image sequence fusion using a shift-invariant wavelet transform’. Proc. IEEE Int. Conf. Image Processing, 1997, vol. 3, pp. 288291.
    15. 15)
      • 25. Manduchi, R., Mian, G.A.: ‘Accuracy analysis for correlation-based image registration algorithms’. Proc. IEEE Int. Sym. Circuits Syst., 1993, pp. 834837.
    16. 16)
      • 19. Nagel, H.H., Enkelmann, W.: ‘An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences’, IEEE Trans. Pattern Anal. Mach. Intell., 1986, 8, (5), pp. 565593.
    17. 17)
      • 33. Toet, A.: ‘A morphological pyramidal image decomposition’, Pattern Recognit. Lett., 1989, 9, (4), pp. 255261.
    18. 18)
      • 26. Foroosh, H., Zerubia, J.B., Berthod, M.: ‘Extension of phase correlation to subpixel registration’, IEEE Trans. Image Process., 2002, 11, (3), pp. 188200.
    19. 19)
      • 16. Kokaram, A.C.: ‘On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach’, IEEE Trans. Image Process., 2004, 13, (3), pp. 397415.
    20. 20)
      • 51. Petrovic, V., Cootes, T., Pavlovic, R.: ‘Dynamic image fusion performance evaluation’. Proc. IEEE Int. Conf. on Information Fusion, 2007, pp. 17.
    21. 21)
      • 29. Stiller, C., Konrad, J., Bosch, R.: ‘Estimating motion in image sequences: a tutorial on modeling and computation of 2D motion’, IEEE Signal Process. Mag., 1999, 16, (4), pp. 7091.
    22. 22)
      • 47. Simoncelli, E.P., Freeman, W.T., Adelson, E.H., Heeger, D.J.: ‘Shiftable multiscale transforms’, IEEE Trans. Inf. Theory, 1992, 38, (2), pp. 587607.
    23. 23)
      • 28. Mitiche, A., Bouthemy, P.: ‘Computation and analysis of image motion: a synopsis of current problems and methods’, Int. J. Comput. Vis., 1996, 19, (1), pp. 2955.
    24. 24)
      • 10. Zhang, Q., Chen, Y., Wang, L.: ‘Multisensor video fusion based on spatial-temporal salience detection’, Signal Process., 2013, 93, (9), pp. 24852499.
    25. 25)
      • 31. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M., Szeliski, R.: ‘A database and evaluation methodology for optical flow’, Int. J. Comput. Vis., 2011, 92, (1), pp. 131.
    26. 26)
      • 6. Liu, C., Jing, Z., Xiao, G., Yang, B.: ‘Feature-based fusion of infrared and visible dynamic images using target detection’, Chin. Opt. Lett., 2007, 5, (5), pp. 274277.
    27. 27)
    28. 28)
      • 3. Pohl, C., Van Genderen, J.L.: ‘Multisensor image fusion in remote sensing: concepts, methods and applications’, Int. J. Remote Sens., 1998, 19, (5), pp. 823854.
    29. 29)
      • 36. Ranchin, T., Wald, L.: ‘The wavelet transform for the analysis of remotely sensed images’, Int. J. Remote Sens., 1993, 14, (3), pp. 615619.
    30. 30)
      • 43. Selesnick, I.W., Li, K.Y.: ‘Video denoising using 2D and 3D dual-tree complex wavelet transforms’. Proc. SPIE's 48th Annual Meeting. Int. Society for Optics and Photonics, 2003, pp. 607618.
    31. 31)
      • 48. Bruhn, A., Weickert, J., Schnörr, C.: ‘Lucas/Kanade meets Horn/Schunk: combining local and global optical flow methods’, Int. J. Comput. Vis., 2005, 61, (3), pp. 211231.
    32. 32)
      • 32. Burt, P.J., Adelson, E.H.: ‘The Laplacian pyramid as a compact image code’, IEEE Trans. Commun., 1983, 31, (4), pp. 532540.
    33. 33)
      • 45. Negi, P.S., Labate, D.: ‘3-D discrete shearlet transform and video processing’, IEEE Trans. Image Process., 2012, 21, (6), pp. 29442954.
    34. 34)
      • 21. Zhu, S., Ma, K.K.: ‘A new diamond search algorithm for fast block-matching motion estimation’, IEEE Trans. Image Process., 2000, 9, (2), pp. 287290.
    35. 35)
      • 15. Irani, M., Hsu, S., Anandan, P.: ‘Video compression using mosaic representations’, Signal Process., Image Commun., 1995, 7, pp. 529552.
    36. 36)
      • 24. Wedel, A., Cremers, D., Pock, T., Bischof, H.: ‘Structure- and motion-adaptive regularization for high accuracy optic flow’. Twelfth Int. Conf. on Computer Vision, Kyoto, Japan, 2009, pp. 16631668.
    37. 37)
      • 49. Liu, C.: ‘Beyond pixels: exploring new representations and applications for motion analysis’. PhD thesis, Massachusetts Institute of Technology, 2009.
    38. 38)
      • 27. Barjatya, A.: ‘Block matching algorithms for motion estimation’, IEEE Trans. Evol. Comput., 2004, 8, (3), pp. 225239.
    39. 39)
      • 18. Horn, B.K., Schunck, B.G.: ‘Determining optical flow’, Artif. Intell., 1981, 17, pp. 185203.
    40. 40)
      • 12. Varghese, G., Wang, Z.: ‘Video denoising based on a spatiotemporal Gaussian scale mixture model’, IEEE Trans. Circuits Syst. Video Technol., 2010, 20, (7), pp. 10321040.
    41. 41)
      • 4. Leykin, A., Ran, Y., Hammoud, R.: ‘Thermal-visible video fusion for moving target tracking and pedestrian classification’. Computer Vision and Pattern Recognition (CVPR'07), Minneapolis, Minnesota, USA, 2007, pp. 18.
    42. 42)
      • 35. Petrović, V.S., Xydeas, C.S.: ‘Gradient-based multiresolution image fusion’, IEEE Trans. Image Process., 2004, 13, (2), pp. 228237.
    43. 43)
      • 44. Lu, Y.M., Do, M.N.: ‘Multidimensional directional filter banks and surfacelets’, IEEE Trans. Image Process., 2007, 16, (4), pp. 918931.
    44. 44)
      • 30. Szeliski, R.: ‘Image alignment and stitching: a tutorial’, Found. Trends in Comput. Graph. Comput. Vis., 2006, 2, (1), pp. 1104.
    45. 45)
      • 20. Lucas, B.D., Kanade, T.: ‘An iterative image registration technique with an application to stereo vision’. Proc. of the Int. Joint Conf. on Artificial Intelligence, 1981, vol. 81, pp. 674679.
    46. 46)
    47. 47)
      • 9. Zhang, Q., Wang, L., Ma, Z., Li, H.: ‘A novel video fusion framework using surfacelet transform’, Opt. Commun., 2012, 285, (13), pp. 30323041.
    48. 48)
      • 41. Do, M.N., Vetterli, M.: ‘The contourlet transform: an efficient directional multiresolution image representation’, IEEE Trans. Image Process., 2005, 14, (12), pp. 20912106.
    49. 49)
      • 22. Fleet, D.J., Jepson, A.D.: ‘Computation of component image velocity from local phase information’, Int. J. Comput. Vis., 1990, 5, (1), pp. 77104.
    50. 50)
      • 39. Candès, E., Demanet, L., Donoho, D., Ying, L.: ‘Fast discrete curvelet transforms’, Multiscale Model. Simul., 2006, 5, (3), pp. 861899.
    51. 51)
      • 46. Xu, L., Du, J., Zhang, Z.: ‘Image sequence fusion and denoising based on 3D shearlet transform’, J. Appl. Math., 2014, 2014, pp. 110.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2014.0245
Loading

Related content

content/journals/10.1049/iet-ipr.2014.0245
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading