accuracy


On Biometric Suppliers Publishing Accuracy Figures 2

Of late there have been repeated calls on Twitter for biometric suppliers to publicly release statistics pertaining to the performance of their biometric algorithms, specifically False Accept Rates (FAR) and False Reject Rates (FRR).

Whilst not a response to those calls, this post is in part motivated by them.

Those repeatedly calling for the release of these figures know in advance that their calls will not be heeded. As they are already well-versed in the technology, they already understand the reasons why. Yet I believe they persist so they can cite the non-responsiveness of suppliers as “evidence” that the technology does not work.

Let’s examine why suppliers keep this information secret.

1. THEY DON’T

I have been involved in negotiating multiple contracts for deployment of biometric technology, ranging from large government infrastructure programmes, through to enterprise access control solutions. I can emphatically state that in every one of these instances, the customer has been absolutely fully aware of the performance metrics of the technology they are deploying, from accuracy through to HW requirements. In fact, before securing any contract, it is very common for the supplier to have to benchmark their technology on customer supplied data, and often the adherence to pre-defined accuracy SLAs is written into contract, with penalties for non-performance.

2. There is no single correct answer

Anybody versed in biometrics knows that the answer is almost always “It depends”. The accuracy is dependent upon multiple factors, many of which will be under the control of the customer, not just the supplier, such as:

      – Quality of the data being matched against
      – Representative population
        – Environmental conditions

     

          – Performance required
          – Budget

      Again, required levels of accuracy will often be pre-agreed with the client, and it is often down to a matter of how much budget the client has available. Faster and / or more accurate will require more computing power, and the determination is often down to a cost benefit analysis.

      3. It is Competitive Confidential Information

      Accuracy of biometric technology can pose a strong competitive advantage, and suppliers often don’t want this information to be in the public domain (or more specifically, available to their competitors). Though the release of this information is often required, for example to prospective clients, it will almost always be under a non-disclosure agreement.

      4. There is no Commercial Reason to do so

      Suppliers, like anybody, don’t like having their time wasted. They’ll apply their resources to those who wish to engage with them seriously, and as mentioned above, they will have no problem in releasing the information as required. A car salesman will spend his or her resources on the individual who wants to buy a car, and ignore the tire kickers.

      My Point

      To only ever argue the facts on one side of a debate to follow a predefined agenda generally results in a loss of credibility. The irony is that people who do so often have valid concerns or issues that quite rightly should be aired and considered, but end up falling by the wayside.

      These are my own personal opinions, and not necessarily the opinions of any suppliers I may happen to work with.


      Biometric Trends Improving Performance

      Iris Biometrics

      Major improvements have been realised in the capture capability, enabling Iris capture on the move or from a distance. While this is not an improvement in the SDK matching per say, it has a significant influence on the matching and usability of the system.

      Face Biometrics

      There have been significant and drastic improvements in the quality and accuracy of matching performance in a very short period of time in the last few years. This has been demonstrated by recent NIST tests, as well as other independent testing. It is not anticipated this rate of improvement will level out any time soon; expect in the coming years further drastic improvements.

      Fingerprint Biometrics

      Accuracy is still continually improving, though not at the same drastic rate as face recognition, as this is a much older technology. However, areas where there are major improvements are in the automated processing of latent prints (both in automated ridge, minutiae identification, feature extraction, and in automated 10-print to latent matching). This has the potential to enable enhanced functionality at verification points, such as border crossings, by implementing functionality such as real-time watchlist checking against latent watchlists.

      Multi-Biometric Record Level Fusion

      Another area where developments are aiding in accuracy improvements is multi-biometric fusion, occurring at the record level. Rather than merging multiple candidate lists from multiple biometrics post search, fusing biometrics and biographics in-record has the potential to provide multi-biometric record-level scores. However, this has more of an impact in very large scale identification systems, as opposed to verification systems, or small scale databases, such as watchlist checking.

      Biometric Matching as a Service

      Supported by trends such as cloud computing, data center consolidation, shared infrastructure and virtualisation.
      See here for more.