Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free Statistical evaluation of corner detectors: does the statistical test have an effect?

© The Institution of Engineering and Technology
Received 10/05/2017, Accepted 01/05/2018, Revised 28/03/2018, Published 03/05/2018

This study explores the use of several non-parametric statistical tests for evaluating the performances of computer vision algorithms, specifically corner detectors, as a more reliable alternative to the graphical approaches that have been commonly employed to date. Using synthetic images carrying corners of different internal angles and orientations and a carefully designed testing framework, a ranking of the performances of corner detectors was established. It was found that Harris & Stephens and SUSAN out-performed more modern detectors. These are one of the few examples where evaluation of vision operators independent of the application has predicted performance in a real-world problem. A similar exercise on real images of the same patterns produced similar results and the findings of a real-world application that uses corners to identify signage were also consistent. Together, all of the tests considered essentially perform pairwise comparisons of performance, so when many algorithms are involved it is important to take account of the potential for type I statistical errors. Several approaches were evaluated and none were found to affect the conclusions.

Inspec keywords: statistical testing; nonparametric statistics; graph theory; edge detection; computer vision

Other keywords: internal angles; nonparametric statistical tests; signage identification; computer vision algorithms; testing framework; vision operators; graphical approaches; corner detectors; synthetic images; type I statistical errors; performance evaluation; statistical evaluation

Subjects: Computer vision and image processing techniques; Image recognition; Other topics in statistics; Other topics in statistics

References

    1. 1)
      • 5. Durkalski, V.L., Palesch, Y.Y., Lipsitz, S.R., et al: ‘Analysis of clustered matched-pair data’, Stat. Med., 2003, 22, (15), pp. 2417–2428.
    2. 2)
      • 29. García, S., Fernández, A., Luengo, J., et al: ‘Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power’, Inf. Sci., 2010, 180, (10), pp. 2044–2064.
    3. 3)
      • 17. Harris, C., Stephens, M.: ‘A combined corner and edge detector’. Proc. Alvey Vision Conf., Manchester, UK, 1988, p. 50, vol. 15.
    4. 4)
      • 16. Pett, M.A.: ‘Nonparametric statistics for health care research: statistics for small samples and unusual distributions’ (Sage Publications, New York, NY, USA, 2015).
    5. 5)
      • 18. Rosten, E., Porter, R., Drummond, T.: ‘FASTER and better: a machine learning approach to corner detection’, IEEE Trans Pattern Anal. Mach. Intell., 2010, 32, pp. 105–119. Available at http://lanl.arXiv.org/pdf/0810.2434.
    6. 6)
      • 4. Provost, F.J., Fawcett, T., Kohavi, R.: ‘The case against accuracy estimation for comparing induction algorithms’. ICML'98. Proc. 15th Int. Conf. on Machine Learning, 1998, pp. 445–453.
    7. 7)
      • 28. Wellner, B., McCallum, A., Peng, F., et al: ‘An integrated, conditional model of information extraction and coreference with application to citation matching’. Proc. 20th Conf. on Uncertainty in Artificial Intelligence, 2004, pp. 593–601.
    8. 8)
      • 7. Barrow, M.: ‘Statistics for economics, accounting and business studies’ (Pearson Education, London, UK, 2009).
    9. 9)
      • 27. Uemura, N., Okamoto, S., Yamamoto, S., et al: ‘Helicobacter pylori infection and the development of gastric cancer’, N. Engl. J. Med., 2001, 345, (11), p. 784.
    10. 10)
      • 35. Li, J.D.: ‘A two-step rejection procedure for testing multiple hypotheses’, J. Stat. Plan. Inference, 2008, 138, (6), pp. 1521–1527.
    11. 11)
      • 40. Calonder, M., Lepetit, V., Strecha, C., et al: ‘Brief: binary robust independent elementary features’. Computer Vision (ECCV), 2010, pp. 778–792.
    12. 12)
      • 30. Perneger, T.V.: ‘What's wrong with Bonferroni adjustments’, Br. Med. J., 1998, 316, pp. 1236–1238. Available at http://www.bmj.com/cgi/content/full/316/7139/1236.
    13. 13)
      • 38. Kanwal, N., Ehsan, S., Bostanci, E., et al: ‘A statistical approach for comparing the performances of corner detectors’. IEEE Pacific Rim Conf. on Communications, Computers and Signal Processing (PacRim), 2011, pp. 321–326.
    14. 14)
      • 21. Shi, J., Tomasi, C.: ‘Good features to track’. Proc. Int. Conf. on Computer Vision and Pattern Recognition, 1994, pp. 593–600.
    15. 15)
      • 37. Kanwal, N., Ehsan, S., Bostanci, E., et al: ‘Evaluating the angular sensitivity of corner detectors’. IEEE Int. Conf. on Virtual Environments Human–Computer Interfaces and Measurement Systems (VECIMS), 2011, pp. 1–4.
    16. 16)
      • 33. Hochberg, Y.: ‘A sharper Bonferroni procedure for multiple tests of significance’, Biometrika, 1988, 75, (4), pp. 800–802.
    17. 17)
      • 6. Winer, B.J., Brown, D.R., Michels, K.M.: ‘Statistical principles in experimental design’ vol. 2 (McGraw-Hill, New York, 1971).
    18. 18)
      • 20. He, X.C., Yung, N.H.C.: ‘Corner detector based on global and local curvature properties’, Opt. Eng., 2008, 47, p. 057008.
    19. 19)
      • 24. Trawiński, B., Smętek, M., Telec, Z., et al: ‘Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms’, Int. J. Appl. Math. Comput. Sci., 2012, 22, (4), pp. 867–881.
    20. 20)
      • 31. Abdi, H.: ‘Bonferroni and Šidák corrections for multiple comparisons’, in Salkind, Neil (Ed.): ‘Encyclopedia of Measurement and Statistics’ (Sage, Thousand Oaks, California, USA, 2007), pp. 103–107.
    21. 21)
      • 9. Cornelis, N., Van.Gool, L.: ‘Fast scale invariant feature detection and matching on programmable graphics hardware’. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), 2008, pp. 1–8.
    22. 22)
      • 13. Saag, M.S., Powderly, W.G., Cloud, G.A., et al: ‘Comparison of amphotericin B with fluconazole in the treatment of acute AIDS-associated cryptococcal meningitis’, N. Engl. J. Med., 1992, 326, (2), pp. 83–89.
    23. 23)
      • 8. Galen, R.S., Gambino, S.R.: ‘Beyond normality: the predictive value and efficiency of medical diagnoses’ (John Wiley & Sons, New York, NY, USA, 1975).
    24. 24)
      • 36. ‘Webpage MRC, Cognition and Brian Science Unit, University of Cambridge’. Available at http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/SpssBonferroni accessed 6 December 2017.
    25. 25)
      • 22. Tomasi, C., Kanade, T.: ‘Detection and tracking of point features’, Int. J. Comput. Vis., 1991, 9(3), pp. 137–154.
    26. 26)
      • 3. Manning, C.D., Schütze, H.: ‘Foundations of statistical natural language processing’ vol. 999 (MIT Press, Cambridge, MA, USA, 1999).
    27. 27)
      • 15. Mack, G.A., Skillings, J.H.: ‘A Friedman-type rank test for main effects in a two-factor ANOVA’, J. Am. Stat. Assoc., 1980, 75, (372), pp. 947–951.
    28. 28)
      • 25. Demšar, J.: ‘Statistical comparisons of classifiers over multiple data sets’, J. Mach. Learn. Res., 2006, 7, pp. 1–30.
    29. 29)
      • 23. Thacker, N.A., Clark, A.F., Barron, J.L., et al: ‘Performance characterization in computer vision: a guide to best practices’, Comput. Vis. Image Underst., 2008, 109, pp. 305–334.
    30. 30)
      • 14. McNemar, Q.: ‘Note on the sampling error of the difference between correlated proportions or percentages’, Psychometrika, 1947, 12, (2), pp. 153–157.
    31. 31)
      • 32. Holm, S.: ‘A simple sequentially rejective multiple test procedure’, Scand. J Stat., 1979, pp. 65–70.
    32. 32)
      • 34. Finner, H.: ‘On a monotonicity problem in step-down multiple test procedures’, J. Am. Stat. Assoc., 1993, 88, (423), pp. 920–923.
    33. 33)
      • 39. Rosten, E., Drummond, T.: ‘Machine learning for high-speed corner detection’. Computer Vision–ECCV 2006, 2006, pp. 430–443.
    34. 34)
      • 1. Tuytelaars, T., Mikolajczyk, K.: ‘Local invariant feature detectors: a survey’, Found. Trends Comput. Graph. vision, 2008, 3, (3), pp. 177–280.
    35. 35)
      • 2. Lusted, L.B.: ‘Signal detectability and medical decision-making’, Science, 1971, 171, (3977), pp. 1217–1219.
    36. 36)
      • 10. Bay, H., Tuytelaars, T., Van.Gool, L.: ‘Surf: speeded up robust features’. European Conf. on computer vision, 2006, pp. 404–417.
    37. 37)
      • 12. Kanwal, N., Bostanci, E., Clark, A.F.: ‘Evaluation method, dataset size or dataset content: how to evaluate algorithms for image matching?’, J. Math. Imaging Vis., 2016, 55, (3), pp. 378–400.
    38. 38)
      • 11. Sokolova, M., Lapalme, G.: ‘A systematic analysis of performance measures for classification tasks’, Inf. Process. Manage., 2009, 45, (4), pp. 427–437.
    39. 39)
      • 26. Gönen, M., Panageas, K.S., Larson, S.M.: ‘Statistical issues in analysis of diagnostic imaging experiments with multiple observations per patient’, Radiology, 2001, 221, (3), pp. 763–767.
    40. 40)
      • 19. Smith, S.M., Brady, J.M.: ‘SUSAN: a new approach to low level image processing’, Int. J. Comput. Vis., 1997, 23, (1), pp. 45–78.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2017.0256
Loading

Related content

content/journals/10.1049/iet-cvi.2017.0256
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address