Particle filter framework for salient object detection in videos

Karthik Muthuswamy; Deepu Rajan

Particle filter framework for salient object detection in videos

View Fulltext

Author(s): Karthik Muthuswamy ¹ and Deepu Rajan ¹
- Affiliations: 1: Centre for Multimedia and Network Technology, School of Computer Engineering, Nanyang Technological University, 50 Nanyang Avenue, N4-02C-92 639798, Singapore
Source: Volume 9, Issue 3, June 2015, p. 428 – 438
DOI: 10.1049/iet-cvi.2013.0298 , Print ISSN 1751-9632, Online ISSN 1751-9640

Received 07/11/2013, Accepted 08/10/2014, Revised 17/09/2014, Published 25/02/2015

Salient object detection in videos is challenging because of the competing motion in the background, resulting from camera tracking an object of interest, or motion of objects in the foreground. The authors present a fast method to detect salient video objects using particle filters, which are guided by spatio-temporal saliency maps and colour feature with the ability to quickly recover from false detections. The proposed method for generating spatial and motion saliency maps is based on comparing local features with dominant features present in the frame. A region is marked salient if there is a large difference between local and dominant features. For spatial saliency, hue and saturation features are used, while for motion saliency, optical flow vectors are used as features. Experimental results on standard datasets for video segmentation and for saliency detection show superior performance over state-of-the-art methods.

References

1. 1)
  - K. Nummiaro , E. Koller-Meierb , L.V. Gool . An adaptive colour-based particle filter. Image Vis. Comput. , 99 - 110
2. 2)
  - 41. Cheng, F.-C., Huang, S.-C., Ruan, S.-J.: ‘Advanced background subtraction approach using Laplacian distribution model’. Proc. IEEE Int. Conf. on Multimedia and Expo, Singapore, Singapore, July 2010, pp. 754–759.
3. 3)
  - 34. Lee, Y.J., Kim, J., Grauman, K.: ‘Key-segments for video object segmentation’. Proc. IEEE Conf. on Computer Vision: Barcelona, Spain, 2011 pp. 1995–2002.
4. 4)
  - 28. Li, W.-T., Chang, H.-S., Lien, K.-C., Chang, H.-T., Wang, Y.-C.F.: ‘Exploring visual and motion saliency for automatic video object extraction’, IEEE T. Image Process., 22, (7) pp. 2600–2610.
5. 5)
  - 25. Elgammal, A., Harwood, D., Davis, L.: ‘Non-parametric model for background subtraction’. ECCV, Lecture Notes in Computer Science, 2000, 1843, pp. 751–767.
6. 6)
  - M.S. Arulampalam , S. Maskell , N. Gordon , T. Clapp . A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Process. , 2 , 174 - 188
7. 7)
  - V. Gopalakrishnan , Y. Hu , D. Rajan . Salient region detection by modeling distributions of color and orientation. IEEE Trans. Multimedia , 5 , 892 - 905
8. 8)
  - 2. Duncan, K., Sarkar, S.: ‘Saliency in images and video: a brief survey’, IET Comput. Vis., 2012, 6, pp. 514–523 (doi: 10.1049/iet-cvi.2012.0032).
9. 9)
  - 22. Adam, A., Rivlin, E., Shimshoni, I., Reinitz, D.: ‘Robust real-time unusual event detection using multiple fixed-location monitors’, IEEE Trans. Pattern Anal. Mach. Intell., 2008, 30, (3), pp. 555–560 (doi: 10.1109/TPAMI.2007.70825).
10. 10)
  - S. Marat , T.H. Phuoc , L. Granjon , N. Guyader , D. Pellerin , A. Guérin-Dugué . Modelling spatio-temporal saliency to predict gaze direction for short videos. Int. J. Comput. Vis. , 3 , 231 - 243
11. 11)
  - 27. Monnet, A., Mittal, A., Paragios, N., Ramesh, V.: ‘Background modeling and subtraction of dynamic scenes’. Proc. 9th IEEE Int. Conf. on Computer Vision, Nice, France, 2003, pp. 1305–1312.
12. 12)
  - 55. Cheng, M.-M., Zhang, G.-X., Mitra, N.J., Huang, X., Hu, S.-M.: ‘Global contrast based salient region detection’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Providence, RI, USA, June 2011, pp. 409–416.
13. 13)
  - 1. Han, S.-H., Jung, G.-D., Lee, S.-Y., Hong, Y.-P., Lee, S.-H.: ‘Automatic salient object segmentation using saliency map and color segmentation’, J. Central South Univ., 2013, 20, (9), pp. 2407–2413 (doi: 10.1007/s11771-013-1750-1).
14. 14)
  - 42. Stauffer, C., Grimson, W.E.L.: ‘Adaptive background mixture models for real-time tracking’. Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, Fort Collins, CO, USA, June 1999, pp. 246–252.
15. 15)
  - 29. Lezama, J., Alahari, K., Sivic, J., Laptev, I.: ‘Track to the future: spatio-temporal video segmentation with long-range motion cues’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Providence, RI, USA, 2011.
16. 16)
  - 12. Ren, Z., Gao, S., Chia, L., Rajan, D.: ‘Regularized feature reconstruction for spatio-temporal saliency detection’, IEEE T. Image Process., 2013, 22, (8), pp. 3120–3132 (doi: 10.1109/TIP.2013.2259837).
17. 17)
  - 4. Hou, X., Zhang, L.: ‘Saliency detection: a spectral residual approach’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 2007.
18. 18)
  - 56. Gonzalez, R.C., Woods, R.E.: ‘Digital image processing’ (Prentice Hall, Upper Saddle River, NJ, 2006, 3rd edn.).
19. 19)
  - 15. Zhai, Y., Shah, M.: ‘Visual attention detection in video sequences using spatiotemporal cues’. Proc. 14th Annual Int. Conf. of Multimedia, Santa Barbara, CA, USA, 2006, pp. 815–824.
20. 20)
  - 8. Gopalakrishnan, V., Hu, Y., Rajan, D.: ‘Sustained observability for salient motion detection’. Proc. 10th Asian Conf. on Computer Vision, Queenstown, New Zealand, 2010, pp. 732–743.
21. 21)
  - 14. Guo, C., Ma, Q., Zhang, L.: ‘Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 2008.
22. 22)
  - 23. Kim, J., Grauman, K.: ‘Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009, pp. 2921–2928.
23. 23)
  - 54. Liu, F., Gleicher, M.: ‘Learning color and locality cues for moving object detection and segmentation’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Miami, FL, USA, June 2009, pp. 320–327.
24. 24)
  - 24. Mittal, A., Paragios, N.: ‘Motion-based background subtraction using adaptive kernel density estimation’. Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, Washington, DC, USA, 2004, vol. 2, pp. II–302-II–309.
25. 25)
  - 51. Morse, B.S., Thornton, D., Xia, Q., Uibel, J.: ‘Image-based color schemes’. Proc. IEEE Int. Conf. on Image Processing, San Antonio, TX, USA, 2007, vol 3, pp. 497–500.
26. 26)
  - 35. Paris, S.: ‘Edge-preserving smoothing and mean-shift segmentation of video streams’. Proc. 10th European Conf. on Computer Vision: Part II, 2008, pp. 460–473.
27. 27)
  - 52. Tsai, D., Flagg, M., Nakazawa, A., Rehg, J.M.: ‘Motion coherent tracking using multi-label MRF optimization’, Int. J. Comput. Vision, 2012, 100, (2), pp. 190–202 (doi: 10.1007/s11263-011-0512-5).
28. 28)
  - 20. Wixson, L.: ‘Detecting salient motion by accumulating directionally-consistent flow’, IEEE Trans. Pattern Anal. Mach. Intell., 2000, 22, (8), pp. 774–780 (doi: 10.1109/34.868680).
29. 29)
  - 38. Yin, P., Criminisi, A., Winn, J., Essa, I.A.: ‘Bilayer segmentation of webcam videos using tree-based classifiers’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (1), pp. 30–42 (doi: 10.1109/TPAMI.2010.65).
30. 30)
  - 31. Brox, T., Malik, J.: ‘Object segmentation by long term analysis of point trajectories’. Proc. European Conf. on Computer Vision, Crete, Greece, 2010, pp. 282–295.
31. 31)
  - 10. Xia, Y., Hu, R., Wang, Z.: ‘Salient map extraction based on motion history map’. Proc. 4th Int, Congress on Image and Signal Processing, Shanghai, China, 2011, pp. 427–430.
32. 32)
  - 17. Li, Q., Chen, S., Zhang, B.: ‘Predictive video saliency detection’. Proc. Chinese Conf. on Pattern Recognition, Beijng, China, 2012, pp. 178–185.
33. 33)
  - 6. Itti, L., Baldi, P.: ‘A principled approach to detecting surprising events in video’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, San Diego, CA, USA, 2005, pp. 631–637.
34. 34)
  - 37. Wu, B., Nevatia, R.: ‘Detection and segmentation of multiple, partially occluded objects by grouping, merging, assigning part detection responses’, Int. J. Comput. Vis., 2009, 82, (2), pp. 185–204 (doi: 10.1007/s11263-008-0194-9).
35. 35)
  - 46. Perez, P., Hue, C., Vermaak, J., Gangnet, J.: ‘Color-based probabilistic tracking’. 7th European Conf. on Computer Vision, Copenhagen, Denmark, 2002, pp. 661–675.
36. 36)
  - 9. Muthuswamy, K., Rajan, D.: ‘Salient motion detection through state controllability’. Proc. IEEE Int. Conf on Acoustics, Speech and Signal Processing, Kyoto, Japan, 2012, pp. 1465–1468.
37. 37)
  - L. Itti , C. Kock , E. Niebur . A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. , 11 , 1254 - 1259
38. 38)
  - 26. Wang, J., Bebis, G., Nicolescu, M., Nicolescu, M., Miller, R.: ‘Improving target detection by coupling it with tracking’, Mach. Vis. Appl., 2009, 20, (4), pp. 205–223 (doi: 10.1007/s00138-007-0118-7).
39. 39)
  - 53. Fukuchi, K., Miyazato, K., Kimura, A., Takagi, S., Yamato, J.: ‘Saliency-based video segmentation with graph cuts and sequentially updated priors’. Proc. IEEE Int. Conf. on Multimedia and Expo, New York, NY, USA, 2009, pp. 638–641.
40. 40)
  - 48. Yantis, S., Jonides, J.: ‘Abrupt visual onsets and selective attention: voluntary versus automatic allocation’, J. Exp. Psychol. Hum. Percept. Perform., 1990, 16, (1), pp. 121–134 (doi: 10.1037/0096-1523.16.1.121).
41. 41)
  - 33. Ochs, P., Brox, T.: ‘Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions’. Proc. IEEE Conf. on Computer Vision, Barcelona, Spain, 2011, pp. 1583–1590.
42. 42)
  - 16. Seo, H.J., Milanfar, P.: ‘Static and space-time visual saliency detection by self-resemblance’, J. Vis., 2009, 9, (12), pp. 1–27 (doi: 10.1167/9.12.15).
43. 43)
  - 21. Bugeau, A., Pérez, P.: ‘Detection and segmentation of moving objects in highly dynamic scenes’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 2007.
44. 44)
  - V. Mahadevan , N. Vasconcelos . Spatiotemporal saliency in dynamic scenes. IEEE Trans. Pattern Anal. Mach. Intell. , 1 , 171 - 177
45. 45)
  - 3. Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: ‘Summarizing visual data using bidirectional similarity’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 2008.
46. 46)
  - 43. Thrun, S., Burgard, W., Fox, D., et al: ‘Probabilistic robotics’ (MIT Press, Cambridge, 2005).
47. 47)
  - 39. Criminisi, A., Cross, G., Blake, A., Kolmogorov, V.: ‘Bilayer segmentation of live video’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, New York, NY, USA, 2006, pp. 53–60.
48. 48)
  - 32. Galasso, F., Iwasaki, M., Nobori, K., Cipolla, R.: ‘Spatio-temporal clustering of probabilistic region trajectories’. Proc. IEEE Conf. on Computer Vision, Barcelona, Spain, 2011, pp. 1738–1745.
49. 49)
  - 11. Luo, Y., Tian, Q.: ‘Spatio-temporal enhanced sparse feature selection for video saliency estimation’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA, 2012, pp. 33–38.
50. 50)
  - 47. Mahapatra, D., Winkler, S., Yen, S.-C.: ‘Motion saliency outweighs other low-level features while watching videos’, Proc. SPIE, 2008, 6806, pp. 68060P1–68060P10 (doi: 10.1117/12.792498).
51. 51)
  - 23. Lin, Z., Davis, L.S.: ‘Shape-based human detection and segmentation via hierarchical part-template matching’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (4), pp. 604–618 (doi: 10.1109/TPAMI.2009.204).
52. 52)
  - 49. Bouguet, J.-Y.: ‘Pyramidal implementation of the affine Lucas Kanade feature tracker description of the algorithm’, Open CV document, Intel, Microprocessor Research Labs, 2000.
53. 53)
  - 30. Grundmann, M., Kwatra, V., Han, M., Essa, I.: ‘Efficient hierarchical graph-based video segmentation’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 2141–2148.
54. 54)
  - 13. Li, Y., Zhou, Y., Xu, L., Yang, X., Yang, J.: ‘Incremental sparse saliency detection’. Proc. 16th IEEE Int. Conf. on Image Processing, Cairo, Egypt, 2009, pp. 3093–3096.
55. 55)
  - 30. Bouwmans, T., El Baf, F., Vachon, B.: ‘Background modeling using mixture of Gaussians for foreground detection – a survey’, Recent Pat. Comput. Sci., 2008, 1, (3), pp. 219–237 (doi: 10.2174/2213275910801030219).
56. 56)
  - 19. Itti, L.: ‘Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes’, Vis. Cogn., 2005, 12, pp. 1093–1123 (doi: 10.1080/13506280444000661).

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Particle filter framework for salient object detection in videos

References

Related content