Review of pedestrian detection techniques in automotive far-infrared video

Patrick Hurney; Peter Waldron; Fearghal Morgan; Edward Jones; Martin Glavin

Review of pedestrian detection techniques in automotive far-infrared video

View Fulltext

Author(s): Patrick Hurney ¹ ; Peter Waldron ² ; Fearghal Morgan ³ ; Edward Jones ¹ ; Martin Glavin ¹
- Affiliations: 1: Connaught Automotive Research Group, Electrical and Electronic Engineering, College of Engineering and Informatics, National University of Ireland, Galway, Ireland ;
  2: Intel Shannon, Dromore House, County Clare, Ireland ;
  3: Electrical and Electronic Engineering, College of Engineering and Informatics, National University of Ireland, Galway, Ireland
Source: Volume 9, Issue 8, October 2015, p. 824 – 832
DOI: 10.1049/iet-its.2014.0236 , Print ISSN 1751-956X, Online ISSN 1751-9578

Received 15/09/2014, Accepted 03/02/2015, Revised 02/01/2015, Published 22/04/2015

The use of advanced driver assistance systems is becoming increasingly common in road-going vehicles. One application of these driver assistance systems is in the automated detection of vulnerable road users, such as pedestrians, using automotive far-infrared imagery. Detection of pedestrians in infrared imagery can be quite difficult because of a number of factors such as the environment, pedestrian behaviour and also the physical limitations of currently available infrared sensors. This study presents a comprehensive review of the literature currently available in the area of pedestrian detection techniques in automotive infrared imagery. The challenges associated with automated detection of pedestrians in the automotive domain are first discussed. An overview of the general structure of pedestrian detection algorithms is then presented, followed by an in-depth analysis of existing literature in the area. Some proposals for future research in the area based on the methods described in this study are then offered.

References

1. 1)
  - 55. Benezeth, Y., Emile, B., Laurent, H., Rosenberger, C.: ‘A real time human detection system based on far infrared vision’, Image Signal Process., 2010, pp. 76–84.
2. 2)
  - 81. Bertozzi, M., Broggi, A., Grisleri, P., Tibaldi, A., Rose, M.: ‘A tool for vision based pedestrian detection performance evaluation’. IEEE Intelligent Vehicles Symp., 2004, June 2004, pp. 784–789.
3. 3)
  - 37. Krotosky, S.J., Trivedi, M.M.: ‘Mutual information based registration of multimodal stereo videos for person tracking’, Comput. Vis. Image Underst., 2007, 106, (2–3), pp. 270–287 (doi: 10.1016/j.cviu.2006.10.008).
4. 4)
  - 60. Wang, X., Zhenmin, T.: ‘Modified particle filter-based infrared pedestrian tracking’, Infrared Phys. Technol., 2010, 53, (4), pp. 280–287 (doi: 10.1016/j.infrared.2010.04.002).
5. 5)
  - 27. Bertozzi, M., Broggi, A., Grisleri, P., Graf, T., Meinecke, M.: ‘Pedestrian detection in infrared images’. IEEE Proc. of Intelligent Vehicles Symp 2003, June 2003, pp. 662–667.
6. 6)
  - C. Papageorgiou , T. Poggio . A trainable system for object detection. Int. J. Comput. Vis. , 1 , 15 - 33
7. 7)
  - 19. Meis, U., Ritter, W., Neumann, H.: ‘Detection and classification of obstacles in night vision traffic scenes based on infrared imagery’. IEEE Proc. of Intelligent Transportation Systems, 2003, October 2003, vol. 2, pp. 1140–1144.
8. 8)
  - T. Ojala , M. Pietikǎinen , T. Mǎenpǎǎ . Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. , 7 , 971 - 987
9. 9)
  - 52. Sun, H., Wang, C., Wang, B., El-Sheimy, N.: ‘Pyramid binary pattern features for real-time pedestrian detection from infrared videos’, Neurocomputing, 2011, 74, (5), pp. 797–804 (doi: 10.1016/j.neucom.2010.10.009).
10. 10)
  - 1. European New Car Assessment Program (Euro NCAP): http://www.euroncap.com/, accessedJanuary 2014.
11. 11)
  - 41. Tsuji, T., Hattori, H., Watanabe, M., Nagaoka, N.: ‘Development of night-vision system’, IEEE Trans. Intell. Transp. Syst., 2002, 3, (3), pp. 203–209 (doi: 10.1109/TITS.2002.802927).
12. 12)
  - M. Bertozzi , A. Broggi , A. Fascioli , T. Graf , M. Meinecke . Pedestrian detection for driver assistance using multiresolution infrared vision. IEEE Trans. Veh. Technol. , 6 , 1666 - 1678
13. 13)
  - R. O'Malley , E. Jones , M. Glavin . Detection of pedestrians in far-infrared automotive night vision using region-growing and clothing distortion compensation. Infrared Phys. Technol , 439 - 449
14. 14)
  - 49. Dalal, N., Triggs, B.: ‘Histograms of oriented gradients for human detection’. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, June 2004, no. 1, pp. 886–893.
15. 15)
  - 33. Besbes, B., Rogozan, A., Bensrhair, A.: ‘Pedestrian recognition based on hierarchical codebook of surf features in visible and infrared imagess’. IEEE Intelligent Vehicles Symp., June 2006, pp. 206–212.
16. 16)
  - 74. NVIDIA: ‘NVIDIA jetson pro’, May 2014. [Online]. Available at http://www.nvidia.com/object/jetsonautomotive-development-platform.html.
17. 17)
  - 24. Bertozzi, M., Broggi, A., Felisa, M., Vezzoni, G., Del Rose, M.: ‘Low-level pedestrian detection by means of visible and far infrared tetra-vision’. IEEE Intelligent Vehicles Symp., 2006, 2006, pp. 231–236.
18. 18)
  - 25. O'Malley, R., Glavin, M., Jones, E.: ‘An efficient region of interest generation technique for far-infrared pedestrian detection’. ICCE Int. Conf. on Consumer Electronics, January 2008, pp. 1–2.
19. 19)
  - 26. Bertozzi, M., Broggi, A., Caraffi, C., Rose, M.D., Felisa, M., Vezzoni, G.: ‘Pedestrian detection by means of far-infrared stereo vision’, Comput. Vis. Image Underst., 2007, 106, (23), pp. 194–204 (doi: 10.1016/j.cviu.2006.07.016).
20. 20)
  - 47. Zhao, L., Thorpe, C.: ‘Stereo and neural network based pedestrian detection’. IEEE/IEEJ/JSAI Int. Conf. on Intelligent Transportation Systems, 1999, Proc., 1999, 1999, pp. 298–303.
21. 21)
  - 31. Xia, D., Sun, H., Shen, Z.: ‘Real-time infrared pedestrian detection based on multi-block lbp’. 2010 Int. Conf. on Computer Application and System Modeling (ICCASM), October 2010, vol. 12, pp. 139–142.
22. 22)
  - 2. Hobbs, A.: ‘Euro NCAP/MORI survey on consumer buying interests (speech and presentation)’. Proc. Euro NCAP Conf.: Creating a Market for Safety – 10 years of Euro NCAP, Brussels, Belgium, November 2005.
23. 23)
  - 40. Bertozzi, M., Broggi, A., Del Rose, M., Felisa, M., Rakotomamonjy, A., Suard, F.: ‘A pedestrian detector using histograms of oriented gradients and a support vector machine classifier’. ITSC 2007, IEEE Intelligent Transportation Systems Conf., 2007, October 2007, pp. 143–148.
24. 24)
  - 39. Liu, X., Fujimura, K.: ‘Pedestrian detection using stereo night vision’, IEEE Trans. Veh. Technol., 2004, 53, (6), pp. 1657–1665 (doi: 10.1109/TVT.2004.834876).
25. 25)
  - 65. Chavan, A., Yogamani, S.: ‘Real-time DSP implementation of pedestrian detection algorithm using HOG features’. Twelfth Int. Conf. on ITS Telecommunications (ITST), 2012, November 2012, pp. 352–355.
26. 26)
  - 58. Kadota, R., Sugano, H., Hiromoto, M., Ochi, H., Miyamoto, R., Nakamura, Y.: ‘Hardware architecture for hog feature extraction’. IIHMSP 09, Fifth Int. Conf. on Intelligent Information Hiding and Multimedia Signal Processing, 2009, September 2009, pp. 1330–1333.
27. 27)
  - 53. Papageorgiou, C.P., Oren, M., Poggio, T.: ‘A general framework for object detection’. Sixth Int. Conf. on Computer Vision, 1998, January 1998, pp. 555–562.
28. 28)
  - 67. Gepperth, A., Ortiz, M., Heisele, B.: ‘Real-time pedestrian detection and pose classification on a gpu’. Sixteenth Int. IEEE Conf. on Intelligent Transportation Systems – (ITSC), 2013 , October 2013, pp. 348–353.
29. 29)
  - 79. Wojek, C., Walk, S., Schiele, B.: ‘Multi-cue onboard pedestrian detection’. CVPR 2009, IEEE Conf. on Computer Vision and Pattern Recognition, 2009, June 2009, pp. 794–801.
30. 30)
  - 32. Miron, A., Besbes, B., Rogozan, A., Ainouz, S., Bensrhair, A.: ‘Intensity self similarity features for pedestrian detection in far-infrared images’. IEEE Intelligent Vehicles Symp. (IV), 2012, June 2012, pp. 1120–1125.
31. 31)
  - 17. Fang, Y., Yamada, K., Ninomiya, Y., Horn, B., Masaki, I.: ‘A shape-independent method for pedestrian detection with far-infrared images’. IEEE Intelligent Vehicles Symp., June 2003, pp. 505–510.
32. 32)
  - 5. Chan, Y.M., Huang, S.S., Fu, L.C., Hsiao, P.Y., Lo, M.F.: ‘Vehicle detection and tracking under various lighting conditions using a particle filter’, IET Intell. Transp. Syst., 2012, 6, pp. 1–8 (doi: 10.1049/iet-its.2011.0019).
33. 33)
  - 12. Mahlke, S., Rösler, D., Seifert, K., Krems, J.F., Thüring, M.: ‘Evaluation of six night vision enhancement system: qualitative and quantitative support for intelligent image processing’, Hum. Factors Ergon. Manuf., 2007, 49, (3), pp. 518–531 (doi: 10.1518/001872007X200148).
34. 34)
  - 15. Fang, Y., Yamada, K., Ninomiya, Y., Horn, B., Masaki, I.: ‘Comparison between infrared-image-based and visible-image-based approaches for pedestrian detection’. IEEE Transactions on Vehicles Symp., Vehicular Technology, 2004, vol. 53, no. 6, pp. 1679–1697.
35. 35)
  - 11. Schenkman, B.N., Brunnström, K.: ‘Camera position and presentation scale for infrared night vision systems in cars’, Hum. Factors Ergon. Manuf., 2007, 17, (5), pp. 457–473 (doi: 10.1002/hfm.20086).
36. 36)
  - 14. Lim, J.H., Tsimhoni, O., Liu, Y.: ‘Investigation of driver performance with night vision and pedestrian detection systems 2014; part i: empirical study on visual clutter and glance behavior’, IEEE Trans. Intell. Transp. Syst., 2010, 11, (3), pp. 670–677 (doi: 10.1109/TITS.2010.2049843).
37. 37)
  - 44. Walczyk, R., Armitage, A., Binnie, T.D.: ‘Fpga implementation of hot spot detection in infrared video’. IET Signals and Systems Conf. (ISSC 2010), Irish, June 2010, pp. 233–238.
38. 38)
  - 3. European Road Safety Observatory: ‘Traffic Safety Basic Facts 2012 – Pedestrians’, March2012.
39. 39)
  - 16. Wiergersma, A.J.: ‘Real-time pedestrian detection in FIR and grayscale images’ (Bochum University, 2006).
40. 40)
  - 7. Unger, C., Wahl, E., Ilic, S.: ‘Parking assistance using dense motion-stereo’, Mach. Vis. Appl., 2011, 6, pp. 1–21.
41. 41)
  - 75. Altera: ‘Altera opencl specification’, May 2014. [Online]. Available at http://www.altera.com/products/software/opencl/openclindex.html.
42. 42)
  - 71. Shang, E., Li, J., An, X., He, H.: ‘A real-time lane departure warning system based on FPGA’. Fourteenth Int. IEEE Conf. on Intelligent Transportation Systems (ITSC), 2011, October 2011, pp. 1243–1248.
43. 43)
  - 35. Bertozzi, M., Broggi, A., Lasagni, A., Rose, M.: ‘Infrared stereo vision-based pedestrian detection’. IEEE Proc. of Intelligent Vehicles Symp., 2005, June 2005, pp. 24–29.
44. 44)
  - 18. Yasuno, M., Ryousuke, S., Yasuda, N., Aoki, M.: ‘Pedestrian detection and tracking in far infrared images’. IEEE Proc. of Intelligent Transportation Systems, 2005, September 2005, pp. 182–187.
45. 45)
  - 54. Sun, H., Wang, C., Wang, B.: ‘Night vision pedestrian detection using a forward-looking infrared camera’. Int. Workshop on Multi-Platform/Multi-Sensor Remote Sensing and Mapping (M2RSM), 2011, 2011, pp. 1–4.
46. 46)
  - 66. NVIDIA, NVIDIA cuda, May 2012. [Online]. Available at http://www.nvidia.com/object/cudahome new.html.
47. 47)
  - 34. Krotosky, S., Trivedi, M.: ‘A comparison of color and infrared stereo approaches to pedestrian detection’. IEEE Intelligent Vehicles Symp., 2007, June 2007, pp. 81–86.
48. 48)
  - 69. Cao, T.P., Deng, G., Mulligan, D.: ‘Implementation of real-time pedestrian detection on FPGA’. IVCNZ 2008, 23rd Int. Conf. on Image and Vision Computing New Zealand, 2008, November 2008, pp. 1–6.
49. 49)
  - 64. Douglas, A., Prasad, M., Sunil Gowtham, B., Kalyansundar, A., Swaminathan, V., Chattopadhyay, R.: ‘An efficient DSP implementation of real-time stationary vehicle detection by smart camera at outdoor conditions’. IEEE Int. Conf. on Image Processing, 2006, October 2006, pp. 3273–3276.
50. 50)
  - 45. Miyamoto, R., Sugano, H., Nakamura, Y.: ‘Pedestrian recognition suitable for night vision systems’, JCSNS Int. J. Comput. Sci. Netw. Secur., 2007, 7, (1), pp. 1–8.
51. 51)
  - 59. Welch, G., Bishop, G.: ‘An introduction to the Kalman filter’. Technical Report TR 95-041, Department of Computer Science, University of North Carolina at Chapel Hill, 2003.
52. 52)
  - 76. Griffin, G., Holub, A., Peronab, P.: ‘Caltech-256 object category data set’. Technical Report 7694, California Institute of Technology, 2007.
53. 53)
  - 36. Suard, F., Rakotomamonjy, A., Bensrhair, A., Broggi, A.: ‘Pedestrian detection using infrared images and histograms of oriented gradients’. IEEE Intelligent Vehicles Symp., 2006, 2006, pp. 206–212.
54. 54)
  - 61. Mählisch, M., Oberlander, M., Lohlein, O., Gavrila, D., Ritter, W.: ‘A multiple detector approach to low-resolution FIR pedestrian recognition’. IEEE Intelligent Vehicles Symp., 2005, June 2005, pp. 325–330.
55. 55)
  - 22. Bertozzi, M., Broggi, A., Ghidoni, S., Del Rose, M.: ‘Pedestrian shape extraction by means of active contours’, Field Serv. Robot., 2008, 42, pp. 265–274 (doi: 10.1007/978-3-540-75404-6_25).
56. 56)
  - 56. Leibe, B., Seemann, E., Schiele, B.: ‘Pedestrian detection in crowded scenes’. CVPR 2005, IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 2005, June 2005, vol. 1, pp. 878–885.
57. 57)
  - 13. Tsimhoni, J.B.O., Minoda, T., Flanagan, M.: ‘Pedestrian detection with near and far infrared night vision enhancement’, IEEE Trans. Intell. Transp. Syst., 2004, 6, (1), pp. 63–71.
58. 58)
  - 28. Brehar, R., Vancea, C., Nedevschi, S.: ‘Pedestrian detection in infrared images using aggregated channel features’. IEEE Int. Conf. on Intelligent Computer Communication and Processing (ICCP), 2014, September 2014, pp. 127–132.
59. 59)
  - D.G. Lowe . Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis , 2 , 91 - 110
60. 60)
  - 63. Fardi, B., Schuenert, U., Wanielik, G.: ‘Shape and motion-based pedestrian detection in infrared images: a multi sensor approach’. IEEE Proc. of Intelligent Vehicles Symp., 2005, June 2005, pp. 18–23.
61. 61)
  - S. Munder , D.M. Gavrila . An experimental study on pedestrian classification. IEEE Trans. Pattern Anal. Mach. Intell. , 11 , 1863 - 1868
62. 62)
  - 72. Walczyk, R., Armitage, A., Binnie, T.: ‘An embedded real-time pedestrian detection system using an infrared camera’. IET Signals and Systems Conf. (ISSC 2009), Irish, June 2009, pp. 1–6.
63. 63)
  - 30. Smith, S., Brady, J.: ‘Susan – a new approach to low level image processing’, IEEE Trans. Pattern Anal. Mach. Intell., 1997, 23, (1), pp. 45–78.
64. 64)
  - 70. Martelli, S., Marzotto, R., Colombari, A., Murino, V.: ‘FPGA-based robust ellipse estimation for circular road sign detection’. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, June 2010, pp. 53–60.
65. 65)
  - 73. Johnston, C.T., Bailey, D.G.: ‘FPGA implementation of a single pass connected components algorithm’. DELTA 2008, 4th IEEE Int. Symp. on Electronic Design, Test and Applications, 2008, January 2008, pp. 228–231.
66. 66)
  - 68. Benenson, R., Mathias, M., Timofte, R., Van Gool, L.: ‘Pedestrian detection at 100 frames per second’. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2012, June 2012, pp. 2903–2910.
67. 67)
  - 80. Davis, J., Keck, M.: ‘A two-stage approach to person detection in thermal imagery’. Proc. of Workshop on Applications of Computer Vision, January 2005.
68. 68)
  - 10. Hollnagel, E., Källhammer, J.: ‘Effects of a night vision enhancement system (NVES) on driving: results from a simulator study’. Proc. of the Second Int. Driving Symp. on Human Factors in Driver Assessment, Training and Vehicle Design, Park City, UT, 2003.
69. 69)
  - 21. Bauer, G., Homm, F., Walchshusl, L., Burschka, D.: ‘Multi spectral pedestrian detection and localization’. Advanced Microsystems for Automotive Applications 2008, 2008, pp. 21–35.
70. 70)
  - 46. Gavrila, D., Giebel, J.: ‘Shape-based pedestrian detection and tracking’. IEEE Intelligent Vehicle Symp., 2002, June 2002, vol. 1, pp. 8–14.
71. 71)
  - F. Xu , X. Liu , K. Fujimura . Pedestrian detection and tracking with night vision. IEEE Trans. Intell. Transp. Syst. , 63 - 71
72. 72)
  - 8. Tsimhoni, J.B.O., Minoda, T., Flanagan, M.J.: ‘Pedestrian detection with night vision systems enhanced by automatic warnings’. Proc. 50th Annual Conf. Human Factors Ergonomics Society, 2006, pp. 2443–2447.
73. 73)
  - 42. Hurney, P., Waldron, P., Morgan, F., Jones, E., Glavin, M.: ‘Night-time pedestrian classification with histograms of oriented gradients-local binary patterns vectors’, IET Intell. Transp. Syst., 2015, Accepted.
74. 74)
  - 38. Bertozzi, M., Binelli, E., Broggi, A., Rose, M.: ‘Stereo vision-based approaches for pedestrian detection’. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition – Workshop, 2005, CVPR Workshops., June 2005, pp. 16–16.
75. 75)
  - 2. Cualain, D.O., Hughes, C., Glavin, M., Jones, E.: ‘Automotive standards-grade lane departure warning system’, IET Intell. Transp. Syst., 2012, 6, (1), pp. 44–57 (doi: 10.1049/iet-its.2010.0043).
76. 76)
  - 9. Sullivan, J.M., Flanagan, M.J.: ‘Characteristics of pedestrian risk in darkness’. Technical Report UMTRI-2001-33, University of Michigan Transportation Research Institute, Ann Arbor, MI, 2001.
77. 77)
  - 43. Bertozzi, M., Broggi, A., Ghidoni, S., Meinecke, M.: ‘A night vision module for the detection of distant pedestrians’. IEEE Intelligent Vehicles Symp., 2007, June 2007, pp. 25–30.
78. 78)
  - 6. Kaparias, I., Bell, M.G.H.: ‘Testing a reliable in-vehicle navigation algorithm in the field’, IET Intell. Transp. Syst., 2012, 36, pp. 314–324.
79. 79)
  - 48. Nanda, H., Davis, L.: ‘Probabilistic template based pedestrian detection in infrared videos’. IEEE Intelligent Transportation Symp. 2002, June 2002, no. 1, pp. 15–20.
80. 80)
  - 57. Wang, X., Han, T., Yan, S.: ‘An HOG-LBP human detector with partial occlusion handling’. IEEE 12th Int. Conf. on Computer Vision, 2009, September 2009, pp. 32–39.
81. 81)
  - 62. Marron, M., Garcia, J.C., Sotelo, M.A., et al: ‘Comparing a kalman filter and a particle filter in a multiple objects tracking application’. IEEE Intelligent Vehicles Symp., 2005, October 2007, pp. 1–6.

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Review of pedestrian detection techniques in automotive far-infrared video

References

Related content