http://iet.metastore.ingenta.com
1887

Statistical evaluation of corner detectors: does the statistical test have an effect?

Statistical evaluation of corner detectors: does the statistical test have an effect?

For access to this article, please select a purchase option:

Buy article PDF
£12.50
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
IET Computer Vision — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

This study explores the use of several non-parametric statistical tests for evaluating the performances of computer vision algorithms, specifically corner detectors, as a more reliable alternative to the graphical approaches that have been commonly employed to date. Using synthetic images carrying corners of different internal angles and orientations and a carefully designed testing framework, a ranking of the performances of corner detectors was established. It was found that Harris & Stephens and SUSAN out-performed more modern detectors. These are one of the few examples where evaluation of vision operators independent of the application has predicted performance in a real-world problem. A similar exercise on real images of the same patterns produced similar results and the findings of a real-world application that uses corners to identify signage were also consistent. Together, all of the tests considered essentially perform pairwise comparisons of performance, so when many algorithms are involved it is important to take account of the potential for type I statistical errors. Several approaches were evaluated and none were found to affect the conclusions.

References

    1. 1)
      • 1. Tuytelaars, T., Mikolajczyk, K.: ‘Local invariant feature detectors: a survey’, Found. Trends Comput. Graph. vision, 2008, 3, (3), pp. 177280.
    2. 2)
      • 2. Lusted, L.B.: ‘Signal detectability and medical decision-making’, Science, 1971, 171, (3977), pp. 12171219.
    3. 3)
      • 3. Manning, C.D., Schütze, H.: ‘Foundations of statistical natural language processing’ vol. 999 (MIT Press, Cambridge, MA, USA, 1999).
    4. 4)
      • 4. Provost, F.J., Fawcett, T., Kohavi, R.: ‘The case against accuracy estimation for comparing induction algorithms’. ICML'98. Proc. 15th Int. Conf. on Machine Learning, 1998, pp. 445453.
    5. 5)
      • 5. Durkalski, V.L., Palesch, Y.Y., Lipsitz, S.R., et al: ‘Analysis of clustered matched-pair data’, Stat. Med., 2003, 22, (15), pp. 24172428.
    6. 6)
      • 6. Winer, B.J., Brown, D.R., Michels, K.M.: ‘Statistical principles in experimental design’ vol. 2 (McGraw-Hill, New York, 1971).
    7. 7)
      • 7. Barrow, M.: ‘Statistics for economics, accounting and business studies’ (Pearson Education, London, UK, 2009).
    8. 8)
      • 8. Galen, R.S., Gambino, S.R.: ‘Beyond normality: the predictive value and efficiency of medical diagnoses’ (John Wiley & Sons, New York, NY, USA, 1975).
    9. 9)
      • 9. Cornelis, N., Van.Gool, L.: ‘Fast scale invariant feature detection and matching on programmable graphics hardware’. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), 2008, pp. 18.
    10. 10)
      • 10. Bay, H., Tuytelaars, T., Van.Gool, L.: ‘Surf: speeded up robust features’. European Conf. on computer vision, 2006, pp. 404417.
    11. 11)
      • 11. Sokolova, M., Lapalme, G.: ‘A systematic analysis of performance measures for classification tasks’, Inf. Process. Manage., 2009, 45, (4), pp. 427437.
    12. 12)
      • 12. Kanwal, N., Bostanci, E., Clark, A.F.: ‘Evaluation method, dataset size or dataset content: how to evaluate algorithms for image matching?’, J. Math. Imaging Vis., 2016, 55, (3), pp. 378400.
    13. 13)
      • 13. Saag, M.S., Powderly, W.G., Cloud, G.A., et al: ‘Comparison of amphotericin B with fluconazole in the treatment of acute AIDS-associated cryptococcal meningitis’, N. Engl. J. Med., 1992, 326, (2), pp. 8389.
    14. 14)
      • 14. McNemar, Q.: ‘Note on the sampling error of the difference between correlated proportions or percentages’, Psychometrika, 1947, 12, (2), pp. 153157.
    15. 15)
      • 15. Mack, G.A., Skillings, J.H.: ‘A Friedman-type rank test for main effects in a two-factor ANOVA’, J. Am. Stat. Assoc., 1980, 75, (372), pp. 947951.
    16. 16)
      • 16. Pett, M.A.: ‘Nonparametric statistics for health care research: statistics for small samples and unusual distributions’ (Sage Publications, New York, NY, USA, 2015).
    17. 17)
      • 17. Harris, C., Stephens, M.: ‘A combined corner and edge detector’. Proc. Alvey Vision Conf., Manchester, UK, 1988, p. 50, vol. 15.
    18. 18)
      • 18. Rosten, E., Porter, R., Drummond, T.: ‘FASTER and better: a machine learning approach to corner detection’, IEEE Trans Pattern Anal. Mach. Intell., 2010, 32, pp. 105119. Available at http://lanl.arXiv.org/pdf/0810.2434.
    19. 19)
      • 19. Smith, S.M., Brady, J.M.: ‘SUSAN: a new approach to low level image processing’, Int. J. Comput. Vis., 1997, 23, (1), pp. 4578.
    20. 20)
      • 20. He, X.C., Yung, N.H.C.: ‘Corner detector based on global and local curvature properties’, Opt. Eng., 2008, 47, p. 057008.
    21. 21)
      • 21. Shi, J., Tomasi, C.: ‘Good features to track’. Proc. Int. Conf. on Computer Vision and Pattern Recognition, 1994, pp. 593600.
    22. 22)
      • 22. Tomasi, C., Kanade, T.: ‘Detection and tracking of point features’, Int. J. Comput. Vis., 1991, 9(3), pp. 137154.
    23. 23)
      • 23. Thacker, N.A., Clark, A.F., Barron, J.L., et al: ‘Performance characterization in computer vision: a guide to best practices’, Comput. Vis. Image Underst., 2008, 109, pp. 305334.
    24. 24)
      • 24. Trawiński, B., Smętek, M., Telec, Z., et al: ‘Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms’, Int. J. Appl. Math. Comput. Sci., 2012, 22, (4), pp. 867881.
    25. 25)
      • 25. Demšar, J.: ‘Statistical comparisons of classifiers over multiple data sets’, J. Mach. Learn. Res., 2006, 7, pp. 130.
    26. 26)
      • 26. Gönen, M., Panageas, K.S., Larson, S.M.: ‘Statistical issues in analysis of diagnostic imaging experiments with multiple observations per patient’, Radiology, 2001, 221, (3), pp. 763767.
    27. 27)
      • 27. Uemura, N., Okamoto, S., Yamamoto, S., et al: ‘Helicobacter pylori infection and the development of gastric cancer’, N. Engl. J. Med., 2001, 345, (11), p. 784.
    28. 28)
      • 28. Wellner, B., McCallum, A., Peng, F., et al: ‘An integrated, conditional model of information extraction and coreference with application to citation matching’. Proc. 20th Conf. on Uncertainty in Artificial Intelligence, 2004, pp. 593601.
    29. 29)
      • 29. García, S., Fernández, A., Luengo, J., et al: ‘Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power’, Inf. Sci., 2010, 180, (10), pp. 20442064.
    30. 30)
      • 30. Perneger, T.V.: ‘What's wrong with Bonferroni adjustments’, Br. Med. J., 1998, 316, pp. 12361238. Available at http://www.bmj.com/cgi/content/full/316/7139/1236.
    31. 31)
      • 31. Abdi, H.: ‘Bonferroni and Šidák corrections for multiple comparisons’, in Salkind, Neil (Ed.): ‘Encyclopedia of Measurement and Statistics’ (Sage, Thousand Oaks, California, USA, 2007), pp. 103107.
    32. 32)
      • 32. Holm, S.: ‘A simple sequentially rejective multiple test procedure’, Scand. J Stat., 1979, pp. 6570.
    33. 33)
      • 33. Hochberg, Y.: ‘A sharper Bonferroni procedure for multiple tests of significance’, Biometrika, 1988, 75, (4), pp. 800802.
    34. 34)
      • 34. Finner, H.: ‘On a monotonicity problem in step-down multiple test procedures’, J. Am. Stat. Assoc., 1993, 88, (423), pp. 920923.
    35. 35)
      • 35. Li, J.D.: ‘A two-step rejection procedure for testing multiple hypotheses’, J. Stat. Plan. Inference, 2008, 138, (6), pp. 15211527.
    36. 36)
      • 36. ‘Webpage MRC, Cognition and Brian Science Unit, University of Cambridge’. Available at http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/SpssBonferroni accessed 6 December 2017.
    37. 37)
      • 37. Kanwal, N., Ehsan, S., Bostanci, E., et al: ‘Evaluating the angular sensitivity of corner detectors’. IEEE Int. Conf. on Virtual Environments Human–Computer Interfaces and Measurement Systems (VECIMS), 2011, pp. 14.
    38. 38)
      • 38. Kanwal, N., Ehsan, S., Bostanci, E., et al: ‘A statistical approach for comparing the performances of corner detectors’. IEEE Pacific Rim Conf. on Communications, Computers and Signal Processing (PacRim), 2011, pp. 321326.
    39. 39)
      • 39. Rosten, E., Drummond, T.: ‘Machine learning for high-speed corner detection’. Computer Vision–ECCV 2006, 2006, pp. 430443.
    40. 40)
      • 40. Calonder, M., Lepetit, V., Strecha, C., et al: ‘Brief: binary robust independent elementary features’. Computer Vision (ECCV), 2010, pp. 778792.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2017.0256
Loading

Related content

content/journals/10.1049/iet-cvi.2017.0256
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address