On Biometric Suppliers Publishing Accuracy Figures 2


Of late there have been repeated calls on Twitter for biometric suppliers to publicly release statistics pertaining to the performance of their biometric algorithms, specifically False Accept Rates (FAR) and False Reject Rates (FRR).

Whilst not a response to those calls, this post is in part motivated by them.

Those repeatedly calling for the release of these figures know in advance that their calls will not be heeded. As they are already well-versed in the technology, they already understand the reasons why. Yet I believe they persist so they can cite the non-responsiveness of suppliers as “evidence” that the technology does not work.

Let’s examine why suppliers keep this information secret.

1. THEY DON’T

I have been involved in negotiating multiple contracts for deployment of biometric technology, ranging from large government infrastructure programmes, through to enterprise access control solutions. I can emphatically state that in every one of these instances, the customer has been absolutely fully aware of the performance metrics of the technology they are deploying, from accuracy through to HW requirements. In fact, before securing any contract, it is very common for the supplier to have to benchmark their technology on customer supplied data, and often the adherence to pre-defined accuracy SLAs is written into contract, with penalties for non-performance.

2. There is no single correct answer

Anybody versed in biometrics knows that the answer is almost always “It depends”. The accuracy is dependent upon multiple factors, many of which will be under the control of the customer, not just the supplier, such as:

      – Quality of the data being matched against
      – Representative population
        – Environmental conditions

     

          – Performance required
          – Budget

      Again, required levels of accuracy will often be pre-agreed with the client, and it is often down to a matter of how much budget the client has available. Faster and / or more accurate will require more computing power, and the determination is often down to a cost benefit analysis.

      3. It is Competitive Confidential Information

      Accuracy of biometric technology can pose a strong competitive advantage, and suppliers often don’t want this information to be in the public domain (or more specifically, available to their competitors). Though the release of this information is often required, for example to prospective clients, it will almost always be under a non-disclosure agreement.

      4. There is no Commercial Reason to do so

      Suppliers, like anybody, don’t like having their time wasted. They’ll apply their resources to those who wish to engage with them seriously, and as mentioned above, they will have no problem in releasing the information as required. A car salesman will spend his or her resources on the individual who wants to buy a car, and ignore the tire kickers.

      My Point

      To only ever argue the facts on one side of a debate to follow a predefined agenda generally results in a loss of credibility. The irony is that people who do so often have valid concerns or issues that quite rightly should be aired and considered, but end up falling by the wayside.

      These are my own personal opinions, and not necessarily the opinions of any suppliers I may happen to work with.


      Leave a comment

      Your e-mail address will not be published. Required fields are marked *

      2 thoughts on “On Biometric Suppliers Publishing Accuracy Figures

      • Stephen Wilson

        Serious security is all about transparency. The biometrics industry is conspicuously unusual in this respect. There is no security in obscurity.

        It’s a worry that researchers looking for information are branded as “wasting time”. When you say vendors will “apply their resources to those who wish to engage with them seriously” you mean those who wish to hand over money. Terrific. But most security companies doing cutting edge R&D traditionally engage with the research and consultant communities as well as commercial customers, because transparency, evidence and rigor are important. [The notable exception in biometrics is iris recognition, which has sound theoretical foundations for its entropy and trait stability, and there’s a history of published academic research.]

        So biometrics test results depend on lots of factors? Welcome to the real world. Your anxiety about qualifying test results only shows how poor is the state of standardisation. Vendor specifications (even when they are quoted) are chronically optimistic because they almost always use the Zero Effort Imposter assumption. Biometrics test results are always peculiar to the test environment, and as the FBI says, lab results do not translate to the real world.

        And since when was publishing specifications giving up competitive advantage? Only when a spec is lousy, and then what you have is competitive disadvantage.

      • allevate Post author

        Thanks, Stephen, great rebuttal.

        RE: “Serious security is all about transparency.”, please refer to my point 1 as to why suppliers keep the info secret.