On Biometric Suppliers Publishing Accuracy Figures 2

Of late there have been repeated calls on Twitter for biometric suppliers to publicly release statistics pertaining to the performance of their biometric algorithms, specifically False Accept Rates (FAR) and False Reject Rates (FRR).

Whilst not a response to those calls, this post is in part motivated by them.

Those repeatedly calling for the release of these figures know in advance that their calls will not be heeded. As they are already well-versed in the technology, they already understand the reasons why. Yet I believe they persist so they can cite the non-responsiveness of suppliers as “evidence” that the technology does not work.

Let’s examine why suppliers keep this information secret.

1. THEY DON’T

I have been involved in negotiating multiple contracts for deployment of biometric technology, ranging from large government infrastructure programmes, through to enterprise access control solutions. I can emphatically state that in every one of these instances, the customer has been absolutely fully aware of the performance metrics of the technology they are deploying, from accuracy through to HW requirements. In fact, before securing any contract, it is very common for the supplier to have to benchmark their technology on customer supplied data, and often the adherence to pre-defined accuracy SLAs is written into contract, with penalties for non-performance.

2. There is no single correct answer

Anybody versed in biometrics knows that the answer is almost always “It depends”. The accuracy is dependent upon multiple factors, many of which will be under the control of the customer, not just the supplier, such as:

– Quality of the data being matched against

– Representative population

– Environmental conditions

– Performance required

  – Budget

Again, required levels of accuracy will often be pre-agreed with the client, and it is often down to a matter of how much budget the client has available. Faster and / or more accurate will require more computing power, and the determination is often down to a cost benefit analysis.

3. It is Competitive Confidential Information

Accuracy of biometric technology can pose a strong competitive advantage, and suppliers often don’t want this information to be in the public domain (or more specifically, available to their competitors). Though the release of this information is often required, for example to prospective clients, it will almost always be under a non-disclosure agreement.

4. There is no Commercial Reason to do so

Suppliers, like anybody, don’t like having their time wasted. They’ll apply their resources to those who wish to engage with them seriously, and as mentioned above, they will have no problem in releasing the information as required. A car salesman will spend his or her resources on the individual who wants to buy a car, and ignore the tire kickers.

My Point

To only ever argue the facts on one side of a debate to follow a predefined agenda generally results in a loss of credibility. The irony is that people who do so often have valid concerns or issues that quite rightly should be aired and considered, but end up falling by the wayside.

These are my own personal opinions, and not necessarily the opinions of any suppliers I may happen to work with.