© The Institution of Engineering and Technology
Video data fusion should simultaneously take into account both temporal and spatial dimensions, and therefore a novel spatiotemporal video-fusion algorithm based on motion compensation in the wavelet-transform domain is proposed in this study. The fusion method incorporates motion compensation and the wavelet transform, thus making full use of spatial geometric information and inter-frame temporal information of input videos. The proposed method improves the temporal stability and consistency of the fused video compared to other existing individual frame-based fusion methods. The algorithm first decomposes the image frames which have been processed by an optic flow motion-compensation approach and then develops a spatiotemporal energy-based fusion rule to merge input videos. Experimental results demonstrate that the proposed fusion algorithm has superior visual and quantitative performance to traditional individual-frame-based and state-of-the-art three-dimensional-transform-based methods.
References
-
-
1)
-
40. Do, M.N., Vetterli, M.: ‘The finite ridgelet transform for image representation’, IEEE Trans. Image Process., 2003, 12, (1), pp. 16–28.
-
2)
-
3)
-
23. Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: ‘Highly accurate optic flow computation with theoretically justified warping’, Int. J. Comput. Vis., 2006, 67, (2), pp. 141–158.
-
4)
-
37. Rahman, S.M.M., Ahmad, M.O., Swamy, M.N.S.: ‘Contrast-based fusion of noisy images using discrete wavelet transform’, IET Image Process., 2010, 4, (5), pp. 374–384.
-
5)
-
11. Wang, Z., Li, Q.: ‘Video quality assessment using a statistical model of human visual speed perception’, J. Opt. Soc. Am. A, 2007, 24, (12), pp. B61–B69.
-
6)
-
7. Xiao, G., Wei, K., Jing, Z.: ‘Improved dynamic image fusion scheme for infrared and visible sequence based on image fusion system’. Proc. 11th Int. Conf. on Information Fusion, 2008, pp. 1–6.
-
7)
-
38. Amolins, K., Zhang, Y., Dare, P.: ‘Wavelet based image fusion techniques – an introduction, review and comparison’, ISPRS J. Photogram. Remote Sens., 2007, 62, (4), pp. 249–263.
-
8)
-
42. Da Cunha, A.L., Zhou, J., Do, M.N.: ‘The nonsubsampled contourlet transform: theory, design, and applications’, IEEE Trans. Image Process., 2006, 15, (10), pp. 3089–3101.
-
9)
-
34. Burt, P.J.: ‘A gradient pyramid basis for pattern selective image fusion’. Proc. of the Society for Information Display Conf., 1992, pp. 467–470.
-
10)
-
8. Li, J., Nikolov, S.G., Benton, C.P., Scott-Samuel, N.E.: ‘Motion-based video fusion using optical flow information’. Proc. Ninth Int. Conf. Information Fusion, Florence, Italy, July 2006, pp. 1–8.
-
11)
-
13. Teodosio, L., Bender, W.: ‘Salient video stills: Content and context preserved’. Proc. the first ACM Int. Conf. Multimedia, Anaheim, California, 1993, pp. 39–46.
-
12)
-
17. Mahajan, D., Huang, F.C., Matusik, W., Ramamoorthi, R., Belhumeur, P.: ‘Moving gradients: A path-based method for plausible image interpolation’, ACM Trans. Graph., 2009, 28, (3), p. 42.
-
13)
-
14. Matsushita, Y., Ofek, E., Ge, W., Tang, X., Shum, H.Y.: ‘Full-frame video stabilization with motion inpainting’, IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28, (7), pp. 1150–1163.
-
14)
-
5. Rockinger, O.: ‘Image sequence fusion using a shift-invariant wavelet transform’. Proc. IEEE Int. Conf. Image Processing, 1997, vol. 3, pp. 288–291.
-
15)
-
25. Manduchi, R., Mian, G.A.: ‘Accuracy analysis for correlation-based image registration algorithms’. Proc. IEEE Int. Sym. Circuits Syst., 1993, pp. 834–837.
-
16)
-
19. Nagel, H.H., Enkelmann, W.: ‘An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences’, IEEE Trans. Pattern Anal. Mach. Intell., 1986, 8, (5), pp. 565–593.
-
17)
-
33. Toet, A.: ‘A morphological pyramidal image decomposition’, Pattern Recognit. Lett., 1989, 9, (4), pp. 255–261.
-
18)
-
26. Foroosh, H., Zerubia, J.B., Berthod, M.: ‘Extension of phase correlation to subpixel registration’, IEEE Trans. Image Process., 2002, 11, (3), pp. 188–200.
-
19)
-
16. Kokaram, A.C.: ‘On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach’, IEEE Trans. Image Process., 2004, 13, (3), pp. 397–415.
-
20)
-
51. Petrovic, V., Cootes, T., Pavlovic, R.: ‘Dynamic image fusion performance evaluation’. Proc. IEEE Int. Conf. on Information Fusion, 2007, pp. 1–7.
-
21)
-
29. Stiller, C., Konrad, J., Bosch, R.: ‘Estimating motion in image sequences: a tutorial on modeling and computation of 2D motion’, IEEE Signal Process. Mag., 1999, 16, (4), pp. 70–91.
-
22)
-
47. Simoncelli, E.P., Freeman, W.T., Adelson, E.H., Heeger, D.J.: ‘Shiftable multiscale transforms’, IEEE Trans. Inf. Theory, 1992, 38, (2), pp. 587–607.
-
23)
-
28. Mitiche, A., Bouthemy, P.: ‘Computation and analysis of image motion: a synopsis of current problems and methods’, Int. J. Comput. Vis., 1996, 19, (1), pp. 29–55.
-
24)
-
10. Zhang, Q., Chen, Y., Wang, L.: ‘Multisensor video fusion based on spatial-temporal salience detection’, Signal Process., 2013, 93, (9), pp. 2485–2499.
-
25)
-
31. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M., Szeliski, R.: ‘A database and evaluation methodology for optical flow’, Int. J. Comput. Vis., 2011, 92, (1), pp. 1–31.
-
26)
-
6. Liu, C., Jing, Z., Xiao, G., Yang, B.: ‘Feature-based fusion of infrared and visible dynamic images using target detection’, Chin. Opt. Lett., 2007, 5, (5), pp. 274–277.
-
27)
-
8. Toet, A., Ijspeert, J.K., Waxman, A.M., Aguilar, M.: ‘Fusion of visible and thermal imagery improves situational awareness’, Displays, 1997, 18, (2), pp. 85–95 (doi: 10.1016/S0141-9382(97)00014-0).
-
28)
-
3. Pohl, C., Van Genderen, J.L.: ‘Multisensor image fusion in remote sensing: concepts, methods and applications’, Int. J. Remote Sens., 1998, 19, (5), pp. 823–854.
-
29)
-
36. Ranchin, T., Wald, L.: ‘The wavelet transform for the analysis of remotely sensed images’, Int. J. Remote Sens., 1993, 14, (3), pp. 615–619.
-
30)
-
43. Selesnick, I.W., Li, K.Y.: ‘Video denoising using 2D and 3D dual-tree complex wavelet transforms’. Proc. SPIE's 48th Annual Meeting. Int. Society for Optics and Photonics, 2003, pp. 607–618.
-
31)
-
48. Bruhn, A., Weickert, J., Schnörr, C.: ‘Lucas/Kanade meets Horn/Schunk: combining local and global optical flow methods’, Int. J. Comput. Vis., 2005, 61, (3), pp. 211–231.
-
32)
-
32. Burt, P.J., Adelson, E.H.: ‘The Laplacian pyramid as a compact image code’, IEEE Trans. Commun., 1983, 31, (4), pp. 532–540.
-
33)
-
45. Negi, P.S., Labate, D.: ‘3-D discrete shearlet transform and video processing’, IEEE Trans. Image Process., 2012, 21, (6), pp. 2944–2954.
-
34)
-
21. Zhu, S., Ma, K.K.: ‘A new diamond search algorithm for fast block-matching motion estimation’, IEEE Trans. Image Process., 2000, 9, (2), pp. 287–290.
-
35)
-
15. Irani, M., Hsu, S., Anandan, P.: ‘Video compression using mosaic representations’, Signal Process., Image Commun., 1995, 7, pp. 529–552.
-
36)
-
24. Wedel, A., Cremers, D., Pock, T., Bischof, H.: ‘Structure- and motion-adaptive regularization for high accuracy optic flow’. Twelfth Int. Conf. on Computer Vision, Kyoto, Japan, 2009, pp. 1663–1668.
-
37)
-
49. Liu, C.: ‘Beyond pixels: exploring new representations and applications for motion analysis’. PhD thesis, Massachusetts Institute of Technology, 2009.
-
38)
-
27. Barjatya, A.: ‘Block matching algorithms for motion estimation’, IEEE Trans. Evol. Comput., 2004, 8, (3), pp. 225–239.
-
39)
-
18. Horn, B.K., Schunck, B.G.: ‘Determining optical flow’, Artif. Intell., 1981, 17, pp. 185–203.
-
40)
-
12. Varghese, G., Wang, Z.: ‘Video denoising based on a spatiotemporal Gaussian scale mixture model’, IEEE Trans. Circuits Syst. Video Technol., 2010, 20, (7), pp. 1032–1040.
-
41)
-
4. Leykin, A., Ran, Y., Hammoud, R.: ‘Thermal-visible video fusion for moving target tracking and pedestrian classification’. Computer Vision and Pattern Recognition (CVPR'07), Minneapolis, Minnesota, USA, 2007, pp. 1–8.
-
42)
-
35. Petrović, V.S., Xydeas, C.S.: ‘Gradient-based multiresolution image fusion’, IEEE Trans. Image Process., 2004, 13, (2), pp. 228–237.
-
43)
-
44. Lu, Y.M., Do, M.N.: ‘Multidimensional directional filter banks and surfacelets’, IEEE Trans. Image Process., 2007, 16, (4), pp. 918–931.
-
44)
-
30. Szeliski, R.: ‘Image alignment and stitching: a tutorial’, Found. Trends in Comput. Graph. Comput. Vis., 2006, 2, (1), pp. 1–104.
-
45)
-
20. Lucas, B.D., Kanade, T.: ‘An iterative image registration technique with an application to stereo vision’. Proc. of the Int. Joint Conf. on Artificial Intelligence, 1981, vol. 81, pp. 674–679.
-
46)
-
L. Yang ,
B.L. Guo ,
W. Ni
.
Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform.
Neurocomputing
,
203 -
211
-
47)
-
9. Zhang, Q., Wang, L., Ma, Z., Li, H.: ‘A novel video fusion framework using surfacelet transform’, Opt. Commun., 2012, 285, (13), pp. 3032–3041.
-
48)
-
41. Do, M.N., Vetterli, M.: ‘The contourlet transform: an efficient directional multiresolution image representation’, IEEE Trans. Image Process., 2005, 14, (12), pp. 2091–2106.
-
49)
-
22. Fleet, D.J., Jepson, A.D.: ‘Computation of component image velocity from local phase information’, Int. J. Comput. Vis., 1990, 5, (1), pp. 77–104.
-
50)
-
39. Candès, E., Demanet, L., Donoho, D., Ying, L.: ‘Fast discrete curvelet transforms’, Multiscale Model. Simul., 2006, 5, (3), pp. 861–899.
-
51)
-
46. Xu, L., Du, J., Zhang, Z.: ‘Image sequence fusion and denoising based on 3D shearlet transform’, J. Appl. Math., 2014, 2014, pp. 1–10.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2014.0245
Related content
content/journals/10.1049/iet-ipr.2014.0245
pub_keyword,iet_inspecKeyword,pub_concept
6
6