Skip to main content

Uncalled rant on testing metrics



Uncalled because we all already know this right? As great people such as Cem Kaner already told us about it a long time ago http://kaner.com/pdfs/PracticalApproachToSoftwareMetrics.pdf

 Uncalled as I haven't used these in years.

 Written because I was just asked by management to report product fault statistics.


Using any sort of defect/bug/fault count related metrics in sw development is harmful.

Bug counts fall apart already as they are trying to quantify something that is inherently qualitative. Additionally they make people focus on reporting problems rather than solving them. And really bug counts tell nothing about the product that involved people wouldn't already know.

The only thing good in bug statistics on sw development is that it gives test managers a very easy way to provide meaningful and professional looking but totally hollow metrics. 

And that is not good.



Using any sort of test case count related metric in sw development is harmful. 

Test case is not a specific thing. A test case can be good, bad, incomplete, false, useless, misleading, dishonest, etc. A test case may be small, large, expensive, cheap, etc. And no amount of test cases will ever be "all the test cases". Counting these together gives you a sum that means nothing about the coverage of the tests on product under test. 

A passing test may mean that the test, the tester, the system under test, or the circumstances in the test were wrong/right. A failing test means the same thing. Counting these means nothing on the quality of the product under test.  

The only good thing about test case counts is that it gives test managers a very easy way to give meaningful and professional looking measures on the progress of testing, that actually have no substance.

And that is not good.



Want to get information about the quality of your product and process?

Ask the developers, testers, sales people, customers, and end users. Investigate the root causes of problems. Hold retrospectives. Analyse logs and usage of the system.

Do the work. Don't settle for defect and test case counts just because those are easy.

Comments

Post a Comment

Popular posts from this blog

Testers doing test automation - is that the most important thing for you to do right now?

I've been thinking quite a lot about tester's moving to do test automation.  Lately beause of these three things: 1.  European testing conference , a great testing conference I attended couple of weeks ago. It is very cool due to many things; the way the speakers get compensated, the focus on the conferring side of conferences making it very easy for people to discuss stuff, the way the talks are chosen, and because there are also a lot of developers joining. So anyway when I was talking with several of the attendees, it was a bit strange how it was easier for me to talk about the product development as a whole with the developers, where as with the testers it more naturally moved into talk of automation and tools. Also on the open space, I think the majority of the topics pitched was automation or tool related. And quite little on the process or on the customer facing side of product development.  2. In my company there are a lot of different products and...
3 comments
Read more

Testing drunk

(My first blog writing ever.) I've been thinking a long time that it's funny how many bugs I find by accident. Try to do something, make a mistake and boom - a bug is found.  Making the mistakes intentionally doesn't quite work - that's why they are called accidents I guess.. So I've thought of ways to make myself more prone to accidents, coming up with an apparent one; testing drunk. TUI (testing under the influence). So this I gotta try. More to come on that later.
3 comments
Read more

Six reasons why testers should do code reviews

I have had quite a lot of discussions about code reviews. Quite many also with testers, by which I have understood that many do not do those. I will not start arguing here on whether code reviews are good/important or not. But I will list a few things why I think testers would benefit of doing them. 1. Code is the only documentation that is up to date . If you really want to know how something really functions, you want to be able to see and read the code. 2. Knowing more about the thing done enables you to do better testing . You can spot things that you should definitely test, and things that you probably don't need to test that much. Like extra things added by coder, usage&modifications of existing functions, data types, etc. And to arguments thinking that one loses their "independence" as a tester by knowing too much, I would worry a lot less about that than about testing stuff that you have no idea on how it has been built. 3. Improve logging . I obse...
2 comments
Read more