The question of estimating uncertainty in measurement is fundamental to all scientific fields. In the field of automated human recognition, lack of repeatability and reproducibility of measurements has been noted since at least the 1970s. This study discusses current approaches to estimation of measurement uncertainty within the broader context of scientific philosophy and measurement science. The authors discuss the Duhem–Quine thesis on testing holism and international standards on estimating and reporting uncertainty in laboratory measurements, then apply these concepts to the estimation of uncertainty in technology, scenario and operational testing in biometrics. The authors advocate for moving beyond the calculation of ‘coverage’ intervals as defined in the ISO/IEC ‘guidelines for the expression of uncertainty in measurement’ to full application of the concepts of uncertainty assessment.

References

1. 1)
  - 49. Nuppeney, M.: ‘EasyPASS: evaluation of face recognition performance in an operational automated border control system’. Int. Biometric Performance Conf. 2010, Gaithersburg, MD, 2 March 2010. Available at: http://www.biometrics.nist.gov/cs_links/ibpc2010/pdfs/Nuppeney_ Marcus_IBPC2010_EasyPASS_Talk_Website.pdf.
2. 2)
  - 18. Ku, H.H. (Ed.): ‘Precision measurement and calibration: statistical concepts and Procedures. Nat. Bur. Stand. Spec. Pub. 300’ (US Government Printing Office, Washington, DC, 1969).
3. 3)
  - 47. Rodriguez, J.R., Bouchier, F., Ruehie, M.: ‘Performance evaluation of biometric identification devices: preliminary draft’. Sandia National Laboratories Report SAND93-1930, US Department of Energy, Albuquerque, NM, 1993.
4. 4)
  - 6. Mansfield, A.J., Kelly, G., Chandler, D., Kane, J.: ‘Biometric product testing final report’ (National Physical Laboratory, Teddington, UK, 2000), available at: http://www.cesg.gov.uk/policy_technologies/biometrics/media/biometrictestreportpt1.pdf.
5. 5)
  - 39. Philips, P.J., Martin, A., Wilson, C., Przybocki, M.: ‘An introduction to the evaluation of biometric systems’, IEEE Comput., 2000, 33, (2), pp. 56–63 (doi: 10.1109/2.820040).
6. 6)
  - 51. Chow, J.L.: ‘Science, ecological validity and experimentation’, J. Theory Soc. Behav., 1987, 17, (2), pp. 181–194 (doi: 10.1111/j.1468-5914.1987.tb00094.x).
7. 7)
  - 22. Eisenhart, C., Ku, H.H., Colle, R.: ‘Guidelines for the expression of uncertainty in measurement. ISO/IEC Guide 98, 1995. Available at: http://www.bipm.org/en/publications/guides/gum.html.
8. 8)
  - 10. Quine, W.V.: ‘Two dogmas of empiricism’, Phil. Rev., 1951, 60, pp. 20–43 (doi: 10.2307/2181906).
9. 9)
  - 4. Mill, J.S.: ‘System of logic ratiocinative and inductive: being a connected view of the principles of evidence and the methods of scientific investigation’ (University of Toronto Libraries, reprinted Toronto, 2011, 1843).
10. 10)
  - 3. Bacon, F.: ‘Novum organum. 1620’, In Fowler, T. (Ed.): ‘Novum organum’ (Clarendon Press, Oxford, 1878). Available at: http://www.books.google.com.
11. 11)
  - 28. Gleser, L.J.: ‘Assessing uncertainty in measurement’, Stat. Sci., 1998, 13, pp. 277–290 (doi: 10.1214/ss/1028905888).
12. 12)
  - 13. Roush, S.: ‘Tracking truth: knowledge, evidence, and science’ (Oxford University Press, Oxford, 2005).
13. 13)
  - 32. Kullback, S., Leibler, R.A.: ‘On information and sufficiency’, Annal. Math. Stats., 1951, 22, pp. 79–86 (doi: 10.1214/aoms/1177729694).
14. 14)
  - 35. Pato, J., Millett, L., (Eds.): ‘Biometric recognition: challenges and opportunities’ (National Academies Press, Washington, DC, 2010). (For a discussion on the issues of estimating prior probabilities in biometric systems, see Appendix B, ‘Watch-List Operational Performance and List Size: A First-Cut Analysis’).
15. 15)
  - 34. Auckenthaler, R., Carey, M., Lloyd-Thomas, H.: ‘Score normalization for text-independent speaker verification systems’, Dig. Sig. Proc., 2000, 10, pp. 42–54 (doi: 10.1006/dspr.1999.0360).
16. 16)
  - 53. Mook, D.G.: ‘In defense of external invalidity’, Am. Psychologist., 1983, 38, pp. 379–387 (doi: 10.1037/0003-066X.38.4.379).
17. 17)
  - 29. Mansfield, A.J., (ed.): ‘Information technology: biometric performance testing and reporting – part 1: principles and framework, ISO/IEC 19795-1:2006, 2006.
18. 18)
  - 27. NIST/SEMATECH: ‘e-Handbook of statistical methods’ (NIST, Gaithersburg, MD, 2010). Available at: http://www.itl.nist.gov/div898/handbook/.
19. 19)
  - 52. Lucas, J.W.: ‘Theory-testing, generalization, and the problem of external validity’, Soc. Theory, 2003, 21, (3), pp. 236–253 (doi: 10.1111/1467-9558.00187).
20. 20)
  - 26. Schervish, M.J.: ‘Theory of statistics’ (Springer, London, 1995).
21. 21)
  - 16. Eisenhart, C.: ‘Realistic evaluation of the precision and accuracy of instrument calibration systems’, J. Res. Natl. Bur. Stand., 1963, 67, (C), pp. 161–187.
22. 22)
  - 46. Holmes, J., Wright, L., Maxwell, R.: ‘A performance evaluation of biometric identification devices’. Sandia National Laboratories Report SAND91-0276, US Dept of Energy, Albuquerque, NM, 1991. Available at: http://www.prod.sandia.gov/techlib/access-control.cgi/1991/910276.pdf.
23. 23)
  - 7. Wayman, J.L.: ‘Evaluation of the INSPASS hand geometry data’, InWayman, J.L. (Ed.): ‘National biometric test center collected works: 1997-2000’ (San Jose State University, San Jose, CA, 2000). Available at: http://www.engr.sjsu.edu/biometrics/nbtccw.pdf.
24. 24)
  - 54. Greenwood, J.D.: ‘On the relation between laboratory experiments and social behaviour: causal explanation and generalization’, J. Theory Soc. Behav., 1982, 12, pp. 225–250 (doi: 10.1111/j.1468-5914.1982.tb00449.x).
25. 25)
  - 1. Joint Committee for Guides in Metrology: ‘International vocabulary of metrology: basic and general concepts and associated terms (VIM), JCGM 200:2008, 2008. Available at: http://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2008.pdf.
26. 26)
  - 30. Martin, A., Doddington, G., Kamm, T., Ordowski, M., Przybocki, M.: ‘The DET curve in assessment of detection task performance’. Proc. EuroSpeech, 1997, pp. 1895–1898, Available at: www.itl.nist.gov/iad/mig//publications/storage_paper/det.pdf.
27. 27)
  - 21. Eisenhart, C., Ku, H.H., Colle, R.: ‘Expression of the uncertainties of final measurement results. Special Publication 644’ (National Bureau of Standards, Washington, DC, 1983).
28. 28)
  - 38. Wayman, J.L.: ‘Error rate equations for the general biometric system’, IEEE Robot. Autom., 1999, 6, (1), pp. 35–48 (doi: 10.1109/100.755813).
29. 29)
  - 45. Draper, B., Beveridge, J.R., Lui, Y.M., Bolme, D., Givens, G.: ‘Quantifying how lighting and focus affect face recognition performance’. Proc. IEEE Comp. Vis. Patt. Rec. Workshop (CVPRW), 13 June 2010, pp. 74–81.
30. 30)
  - 2. Aristotle. Organon. (compiled around 400BC). Available at: http://archive.org/details/AristotleOrganon.
31. 31)
  - 12. Gilles, D.: ‘The Duhem thesis and the Quine thesis’, In Curd, M., Cover, J.A., (Eds.): ‘Philosophy of science: the central issues’ (WW Norton, New York, 1998).
32. 32)
  - 9. Duhem, P.: ‘The aim and structure of physical theory’ (Princeton University Press, Princeton, 1954).
33. 33)
  - 36. Dawid, A.P., Mortera, J.: ‘Coherent analysis of forensic identification evidence’, J. Roy. Stat Soc., Series B., 1996, 58, (2), pp. 425–443.
34. 34)
  - 44. Bowyer, K., Baker, S., Hentz, A., Hollingsworth, K., Peters, T., Flynn, P.: ‘Factors that degrade the match distribution in iris biometrics’, Ident. Inf., Soc., 2009, 2, (3), pp. 327–343. Available at: http://www.rd.springer.com/article/10.1007/s12394-009-0037-z (doi: 10.1007/s12394-009-0037-z).
35. 35)
  - 50. Fejfar, A., Myers, J.: ‘The testing of 3 automatic identity verification techniques for entry control’. Proc. Second Int. Conf. Crime Countermeasures, Oxford, 25 July 1977.
36. 36)
  - 42. Phillips, P.J., Grother, P., Micheals, R.J., Blackburn, D.M., Tabassi, E., Bone, J.M.: ‘Face recognition vendor test 2002: Evaluation report, NISTIR 6965’ (NIST, Gaithersburg, MD, 2003).
37. 37)
  - 11. Curd, M., Cover, J.A. (Eds.): ‘Philosophy of science: the central issues’ (WW Norton, New York, 1998).
38. 38)
  - 14. Moser, P.K., (Ed.): ‘The Oxford handbook of epistemology’ (Oxford University Press, Oxford, 2005).
39. 39)
  - 23. Taylor, B., Kuyatt, C.: ‘Guidelines for evaluating and expressing the uncertainty of NIST measurement results, NIST Technical Note 1297’ (National Institute of Standards and Technology, Gaithersburg, MD, 1994).
40. 40)
  - 37. Balding, D.J.: ‘The DNA database search controversy’, Biometrics, 2002, 58, (1), pp. 241–244 (doi: 10.1111/j.0006-341X.2002.00241.x).
41. 41)
  - 24. Joint Committee for Guides in Metrology: ‘Evaluation of measurement data: Supplement 1 to the Guide to the expression of uncertainty in measurement: Propagation of distributions using a Monte Carlo method’ JCGM 101:2008 (BIPM, 2008).
42. 42)
  - 25. Neyman, J.: ‘Outline of a theory of statistical estimation based on the classical theory of probability’, Phil. Trans. Roy. Soc. London A, 1937, 236, pp. 333–380 (doi: 10.1098/rsta.1937.0005).
43. 43)
  - 31. Finkelstein, M.O., Fairley, W.B.: ‘A Bayesian approach to identification evidence’, Harv. Law. Rev., 1970, 83, pp. 489–517 (doi: 10.2307/1339656).
44. 44)
  - 17. Ku, H.H.: ‘Expressions of imprecision, systematic error, and uncertainty associated with a reported value’, Meas. Data, 1968, 2, (4), pp. 72–77.
45. 45)
  - 20. Joint Committee for Guides in Metrology (JCGM): ‘GUM 1995 with minor corrections’. JCGM 100:2008, September 2008.
46. 46)
  - 48. Mansfield, A.J., Wayman, J.L.: ‘Best practices of testing and reporting biometric device performance: Version 2.01’. National Physical Laboratory Report CMSC 14/02, National Physical Laboratory, Teddington, UK, August 2002. Available at: http://www.npl.co.uk/upload/pdf/biometrics_bestprac_v2_1.pdf.
47. 47)
  - 19. Eisenhart, C.: ‘Expression of uncertainties in final results’, Science, 1968, 160, (3833), pp. 1201–1204 (doi: 10.1126/science.160.3833.1201).
48. 48)
  - 15. Shewhart, W.A.: ‘Statistical method from the viewpoint of quality control’ (US Department of Agriculture, Washington, DC, 1939).
49. 49)
  - 33. Kullback, S.: ‘Letter to the editor: the Kullback-Leibler distance’, Am. Stat., 1987, 41, (4), pp. 338–341 (doi: 10.1080/00031305.1987.10475510).
50. 50)
  - 43. Theofanos, M., Stanton, B., Wolfson, C.A.: ‘Usability and biometrics: ensuring successful biometric systems’ (NIST, Gaithersburg, MD, 2008), Available at: http://www.zing.ncsl.nist.gov/biousa/docs/Usability_and_Biometrics_final2.pdf.
51. 51)
  - 8. Wayman, J.L.: ‘Biometrics at the Sydney airport: evaluation of the SmartGate trial’. Proc, Fourth Int. Symp. Comput Media Stud Biom Authentication Symp., Kyoto, 2006, pp. 132–137.
52. 52)
  - 40. Watson, C., Wilson, C., Indovina, M., Cochran, B.: ‘Two finger matching with vendor SDK matchers. NISTIR 7249’ (NIST, Gaithersburg, MD, 2005), Available at: http://www.nvlpubs.nist.gov/nistpubs/ir/2005/ir7249.pdf.
53. 53)
  - 5. Feyerabend, P.: ‘Against method’ (Verso, London, 2010, 4th edn.).
54. 54)
  - 41. Wilson, C., Hicklin, A., Bone, M., et al: ‘Fingerprint vendor technology evaluation 2003: summary of results and analysis report. NISTIR 7123’ (NIST, Gaithersburg, MD, 2004) Available at: http://www.cse.msu.edu/~rossarun/BiometricsTextBook/Papers/Introduction/FpVTE2003Analysis.pdf.

Modern statistical and philosophical framework for uncertainty assessment in biometric performance testing

References

Related content