|
FpVTE 2003 Evaluations Overview1 OverviewThe Fingerprint Vendor Technology Evaluation (FpVTE) 2003 was an independently administered technology evaluation of fingerprint matching, identification, and verification systems. FpVTE 2003 was conducted by the National Institute of Standards & Technology (NIST) on behalf of the Justice Management Division (JMD) of the U.S. Department of Justice. FpVTE was designed to assess the capability of fingerprint systems to meet requirements for both large-scale and small-scale real world applications. FpVTE 2003 consists of multiple tests performed with combinations of fingers (e.g., single fingers, two index fingers, four to ten fingers) and different types and qualities of operational fingerprints (e.g., flat livescan images from visa applicants, multi-finger slap livescan images from present-day booking or background check systems, or rolled and flat inked fingerprints from legacy criminal databases). FpVTE 2003 will serve as part of NIST's statutory mandate under section 403c of the USA PATRIOT Act to certify those biometric technologies that may be used in U.S. VISIT (U.S. Visitor and Immigrant Status Indication Technology, formerly known as the US entry-exit system). In addition, FpVTE results may form the basis for the design and acquisition of other large-scale fingerprint identification systems. FpVTE 2003 analysis and methodologies are built on the multi-agency Face Recognition Vendor Test (FRVT) 2002. See http://www.frvt.org FpVTE 2003 evaluations was conducted at the NIST facilities at Gaithersburg, MD, between September 29 and December 31, 2003. Interested parties should send contact information (name, email, telephone, address) to FpVTE@nist.gov . The deadline for registration for Participants was 12 August 2003. 2 PurposeFpVTE was intended to evaluate the state of the art / state of the practice in fingerprint matching systems. The Supporters and Sponsors of FpVTE are specifically interested in fingerprint matcher performance in these areas:
3 Who Should ParticipateMakers of commercially available fingerprint matching, verification, or identification (AFIS) systems were invited to participate in the Fingerprint Vendor Technology Evaluation 2003. In addition, companies, research organizations, and universities that have developed mature prototype or research fingerprint matching, verification, or identification systems are invited to participate. Makers of fingerprint screening or filtering systems were also encouraged to participate. FpVTE was not evaluating image acquisition devices (fingerprint scanners). 4 Sponsors and SupportersClick here to jump to the current list of Sponsors and Supporters. 5 Overview of EvaluationsFpVTE was composed of three separate tests, the Small-Scale Test (SST), Medium-Scale Test (MST), and the Large-Scale Test (LST). The SST and MST evaluated matching accuracy using individual fingerprint images. This contrasts with the LST, which evaluated matching accuracy using sets of fingerprint images. In general, the SST and MST was of interest to makers of single-stage matchers, verification or small-scale identification systems, or new algorithms. Makers of fingerprint screening or filtering systems were also encouraged to participate in the MST. In general, the LST would be of interest to AFIS vendors. The tests are designed so that the SST is a subset of the MST. LST participants were encouraged to participate in the MST. If space is available, FpVTE personnel may allow a participant to enter more than one system in the evaluation. A second system might be submitted to demonstrate capabilities that can be achieved only through distinct parameter settings, an alternate system configuration, or different algorithms. Click here to jump to the Test Plan for more information about SST, MST, or LST. 5.1 Throughput RequirementsTentative Participants were required to complete a System Throughput Questionnaire to qualify for the tests. This information was used to determine the final test design. The size and structure of the test was designed to optimize among competing analysis objectives, available resources, and the desire to include all qualified participants. The System Throughput Questionnaire is located in the Participant Area, which is only available to registered Participants and Tentative Participants. 5.2 Matcher ResultsThe results from each test are participant-defined measures of similarity, which for most systems means matcher scores. Note that the systems do not return Match and Non-match determinations. FpVTE analysis is based on distributions of similarity scores for mate vs. non-mate comparisons. Which fingerprints are mated is, of course, not revealed. For the SST and MST tests, matchers compare individual fingerprint images, and return a similarity score for each comparison. When comparing one dataset of fingerprint images against another, a fully-populated matrix of similarity scores is generated. This fully-populated similarity matrix is what will be used to evaluate SST and MST Participants. For the LST, matchers compare sets of fingerprint images (e.g. a 2-image, 4-image, or 10-image electronic "card" of fingerprint images). Since an AFIS generally filters out many non-mates, similarity scores may not be available for every comparison. Participants nevertheless will be expected to generate a fully-populated matrix of scores, but may, for instance, fill the majority of the matrix with a default value indicating no similarity. Participants are advised that the choice of true scores vs. default values will affect scoring, and are encouraged to fill the matrix as completely as possible with true scores. Click here to jump to the Test Plan for more information about similarity scores and matrices. 5.3 Operating PointsEvaluations were based on ROC (Receiver Operator Characteristic) or, equivalently, DET (Detection Error Trade-off) analysis based on similarity scores. Rank-based analysis will also be performed. For SST, evaluations focused on the relationship between TAR (True Accept Rate: TAR = 1 - FRR (False Reject Rate)) and FAR (False Accept Rate), when FAR is between 10-2 and 10-5. For MST, evaluations focused on the relationship between TAR and FAR when FAR is between 10-2 and 10-7. In addition, evaluations will focus on the relationship between TAR and FAR when TAR is between 98% and 99.99%. For LST, the relationship between TAR and FAR was analyzed, focusing solely on very low FAR values (down to about 10-8). Since some algorithms do not operate equally well at a broad range of operating points, participants could submit two algorithms or tunings of an algorithm for the SST or MST, one optimized for low FAR and another for high TAR. 6 Data Description and RestrictionsThe fingerprints have been collected from a range of U.S. Government sources. Some of the fingerprints are representative of current operational data, while others are representative of legacy data. 6.1 Fingerprint TypesThe fingerprints are of three types:
Note that the slap fingerprints have already been segmented into individual fingerprints. 6.2 Fingerprint SourcesIndividual flat (plain) fingerprints were collected using a variety of single-finger livescan devices. Slap fingerprints were collected using multi-finger livescan devices, as well as inked fingerprint cards scanned using flatbed scanners. Rolled fingerprints were collected using rolled livescan devices, as well as inked fingerprint cards scanned using flatbed scanners. The fingerprints were collected by the following devices. The listing of makes and models does not imply a recommendation by NIST or FpVTE personnel, but simply recognizes the actual devices used by the variety of Federal agencies that contributed data to FpVTE. Single-finger livescan devices:
Rolled and multi-finger livescan devices:
Inked cards were scanned using FBI EFTS Appendix F-certified flatbed scanners. The source (type of scanner) of each fingerprint was not provided in the tests. 6.3 Fingerprint File FormatEach fingerprint (SST or MST) or fingerprint set (LST) was contained in an ANSI/NIST format file. Each file contains
All images are WSQ compressed. Note: The FBI’s Electronic Fingerprint Transmission Specification (EFTS) is based on ANSI/NIST. Fingerprint files that are EFTS compliant are necessarily ANSI/NIST compliant. 6.4 Data RestrictionsThe FpVTE 2003 Evaluation Datasets will bear the legend “Notice: May contain Privacy Act or FOIA Protected Information” and, to the extent permitted by law, will be protected under the Freedom of Information Act (5 U.S.C 552) and the Privacy Act (5 U.S.C. 552a) as applicable. FpVTE 2003 Evaluation Datasets and data derived from the datasets shall not be retained in any way or form whatsoever by Participants after completion of the Evaluation. Systems being evaluated in FpVTE 2003 shall not be accessible from outside the room in which the evaluation is being conducted. No system entered in the test can have or use any wireless networking equipment, modems, or access to the Internet. 7 Test PreparationParticipants were expected to modify their systems as necessary to meet the following test requirements:
FpVTE 2003 Sample Datasets and Software was supplied to Participants to assist in preparing for FpVTE 2003. The FpVTE 2003 Sample Datasets are representative of the FpVTE 2003 Evaluation Datasets in format. Image quality, collection device and other characteristics may vary between the Sample and Evaluation Datasets. 8 Final ReportAfter completion of the evaluations, the Government combined all results into a Final Report. The FpVTE 2003 Final Report is expected to be released in early 2004. The FpVTE 2003 Final Report will contain, at a minimum, descriptive information concerning FpVTE 2003, descriptions of each experiment, evaluation results, and each Participant’s five-page system description document. The Final Report can be found on the FpVTE 2003 website (http://FpVTE.nist.gov). |
FpVTE LinksOther Links
|
