Sunday, June 24, 2018

Uncalled rant on testing metrics

Uncalled because we all already know this right? As great people such as Cem Kaner already told us about it a long time ago

Uncalled as I haven't used these in years.

Written because I was just asked by management to report product fault statistics.

Using any sort of defect/bug/fault count related metrics in sw development is harmful.

Bug counts fall apart already as they are trying to quantify something that is inherently qualitative. Additionally they make people focus on reporting problems rather than solving them. And really bug counts tell nothing about the product that involved people wouldn't already know.

The only thing good in bug statistics on sw development is that it gives test managers a very easy way to provide meaningful and professional looking but totally hollow metrics. 

And that is not good.

Using any sort of test case count related metric in sw development is harmful. 

Test case is not a specific thing. A test case can be good, bad, incomplete, false, useless, misleading, dishonest, etc. A test case may be small, large, expensive, cheap, etc. And no amount of test cases will ever be "all the test cases". Counting these together gives you a sum that means nothing about the coverage of the tests on product under test. 

A passing test may mean that the test, the tester, the system under test, or the circumstances in the test were wrong/right. A failing test means the same thing. Counting these means nothing on the quality of the product under test.  

The only good thing about test case counts is that it gives test managers a very easy way to give meaningful and professional looking measures on the progress of testing, that actually have no substance.

And that is not good.

Want to get information about the quality of your product and process?

Ask the developers, testers, sales people, customers, and end users. Investigate the root causes of problems. Hold retrospectives. Analyse logs and usage of the system.

Do the work. Don't settle for defect and test case counts just because those are easy.

No comments:

Post a Comment