Skip to main content

Uncalled rant on testing metrics



Uncalled because we all already know this right? As great people such as Cem Kaner already told us about it a long time ago http://kaner.com/pdfs/PracticalApproachToSoftwareMetrics.pdf

 Uncalled as I haven't used these in years.

 Written because I was just asked by management to report product fault statistics.


Using any sort of defect/bug/fault count related metrics in sw development is harmful.

Bug counts fall apart already as they are trying to quantify something that is inherently qualitative. Additionally they make people focus on reporting problems rather than solving them. And really bug counts tell nothing about the product that involved people wouldn't already know.

The only thing good in bug statistics on sw development is that it gives test managers a very easy way to provide meaningful and professional looking but totally hollow metrics. 

And that is not good.



Using any sort of test case count related metric in sw development is harmful. 

Test case is not a specific thing. A test case can be good, bad, incomplete, false, useless, misleading, dishonest, etc. A test case may be small, large, expensive, cheap, etc. And no amount of test cases will ever be "all the test cases". Counting these together gives you a sum that means nothing about the coverage of the tests on product under test. 

A passing test may mean that the test, the tester, the system under test, or the circumstances in the test were wrong/right. A failing test means the same thing. Counting these means nothing on the quality of the product under test.  

The only good thing about test case counts is that it gives test managers a very easy way to give meaningful and professional looking measures on the progress of testing, that actually have no substance.

And that is not good.



Want to get information about the quality of your product and process?

Ask the developers, testers, sales people, customers, and end users. Investigate the root causes of problems. Hold retrospectives. Analyse logs and usage of the system.

Do the work. Don't settle for defect and test case counts just because those are easy.

Comments

Post a Comment

Popular posts from this blog

Periodical retrospectives are lame

  "You got nothing, not a single thing?! Well lets just end this here then." I remember well when I said this, being very frustrated. About ten years ago I had been working as a Scrum master for a team some months, and putting quite a lot of effort into planning our scrum teams sprint retrospectives. Lot of work also because I felt we were not getting too much out from them; not very good discussions, very few actions, and even the few actions we did come up with did not stick.  And then it happened: a retro where none of the participants came up with anything to say about the sprint. Regardless of the retro topic boxes, reading of books on retrospectives, getting inspiration from tools like retromat.org, having them in different places, using all kinds of different formats and rainbow coloured post-it notes. Not a single thing. Blank.  So then I said the words, out of frustration, mainly to myself. Why couldn't I get this thing everyone is so hyped about to work? After t
Post a Comment
Read more

I don't report bugs

I don't report bugs . Bug is such a loaded word that people understand very differently, that instead of using it and explaining what I mean by it I rather just use other words. Like observations, thoughts, surprises, ideas, alternatives, or something similar. (And no I don't use fault, defect, or error either). Bug has also quite a negative connotation. "Reporting a bug" is kind of like telling someone that they've been served. And as we are actually giving away the gift of information, why wrap it in such a nasty package? And maybe more importantly it is very likely that whatever you might have to say is wrong. If not plain wrong, then at least incomplete. So I like to approach the kind of situations with the assumption that I am probably wrong. Cutting off anything that might sound arrogant makes stuff quite a lot easier. Especially after you realise later on that you have been wrong. I leave plenty of observations unreported . I don't want to waste
Post a Comment
Read more

Testing drunk

(My first blog writing ever.) I've been thinking a long time that it's funny how many bugs I find by accident. Try to do something, make a mistake and boom - a bug is found.  Making the mistakes intentionally doesn't quite work - that's why they are called accidents I guess.. So I've thought of ways to make myself more prone to accidents, coming up with an apparent one; testing drunk. TUI (testing under the influence). So this I gotta try. More to come on that later.
3 comments
Read more