Evaluating Biometric System
An Introduction to Evaluating Biometric Systems
How and where biometric systems are deployed will depend on their performance. Knowing what to ask and how to decipher the answers can help you evaluate the performance of these emerging technologies.
P. Jonathon Phillips Alvin Martin C.L. Wilson Mark Przybocki
National Institute of Standards and Technology
O
n the basis of media hype alone, you might conclude that biometric passwords will soon replace their alphanumeric counterparts with versions that cannot be stolen, forgotten, lost, or given to another person. But what if the performance estimates of these systems are far more impressive than their actual performance? To measure the real-life performance of biometric systems—and to understand their strengths and weaknesses better—we must understand the elements that comprise an ideal biometric system. In an ideal system • all members of the population possess the characteristic that the biometric identifies, like irises or fingerprints; • each biometric signature differs from all others in the controlled population; • the biometric signatures don’t vary under the conditions in which they are collected; and • the system resists countermeasures. Biometric-system evaluation quantifies how well biometric systems accommodate these properties. Typically, biometric evaluations require that an independent party design the evaluation, collect the test data, administer the test, and analyze the results. We designed this article to provide you with sufficient information to know what questions to ask when evaluating a biometric system, and to assist you in determining if performance levels meet the requirements of your application. For example, if you plan to use a biometric to reduce—as opposed to eliminate— fraud, then a low-performance biometric system may be sufficient. On the other hand, completely replacing
an existing security system with a biometric-based one may require a high-performance biometric
How and where biometric systems are deployed will depend on their performance. Knowing what to ask and how to decipher the answers can help you evaluate the performance of these emerging technologies.
P. Jonathon Phillips Alvin Martin C.L. Wilson Mark Przybocki
National Institute of Standards and Technology
O
n the basis of media hype alone, you might conclude that biometric passwords will soon replace their alphanumeric counterparts with versions that cannot be stolen, forgotten, lost, or given to another person. But what if the performance estimates of these systems are far more impressive than their actual performance? To measure the real-life performance of biometric systems—and to understand their strengths and weaknesses better—we must understand the elements that comprise an ideal biometric system. In an ideal system • all members of the population possess the characteristic that the biometric identifies, like irises or fingerprints; • each biometric signature differs from all others in the controlled population; • the biometric signatures don’t vary under the conditions in which they are collected; and • the system resists countermeasures. Biometric-system evaluation quantifies how well biometric systems accommodate these properties. Typically, biometric evaluations require that an independent party design the evaluation, collect the test data, administer the test, and analyze the results. We designed this article to provide you with sufficient information to know what questions to ask when evaluating a biometric system, and to assist you in determining if performance levels meet the requirements of your application. For example, if you plan to use a biometric to reduce—as opposed to eliminate— fraud, then a low-performance biometric system may be sufficient. On the other hand, completely replacing
an existing security system with a biometric-based one may require a high-performance biometric
References: 1. J.P. Egan, Signal Detection Theory and ROC Analysis, Academic Press, New York, 1975. 2. A. Martin et al., “The DET Curve Assessment of Detection Task Performance,” Proc. EuroSpeech 97, IEEE CS Press, Los Alamitos, Calif., 1997, pp