Skip to main content

Uncalled rant on testing metrics



Uncalled because we all already know this right? As great people such as Cem Kaner already told us about it a long time ago http://kaner.com/pdfs/PracticalApproachToSoftwareMetrics.pdf

 Uncalled as I haven't used these in years.

 Written because I was just asked by management to report product fault statistics.


Using any sort of defect/bug/fault count related metrics in sw development is harmful.

Bug counts fall apart already as they are trying to quantify something that is inherently qualitative. Additionally they make people focus on reporting problems rather than solving them. And really bug counts tell nothing about the product that involved people wouldn't already know.

The only thing good in bug statistics on sw development is that it gives test managers a very easy way to provide meaningful and professional looking but totally hollow metrics. 

And that is not good.



Using any sort of test case count related metric in sw development is harmful. 

Test case is not a specific thing. A test case can be good, bad, incomplete, false, useless, misleading, dishonest, etc. A test case may be small, large, expensive, cheap, etc. And no amount of test cases will ever be "all the test cases". Counting these together gives you a sum that means nothing about the coverage of the tests on product under test. 

A passing test may mean that the test, the tester, the system under test, or the circumstances in the test were wrong/right. A failing test means the same thing. Counting these means nothing on the quality of the product under test.  

The only good thing about test case counts is that it gives test managers a very easy way to give meaningful and professional looking measures on the progress of testing, that actually have no substance.

And that is not good.



Want to get information about the quality of your product and process?

Ask the developers, testers, sales people, customers, and end users. Investigate the root causes of problems. Hold retrospectives. Analyse logs and usage of the system.

Do the work. Don't settle for defect and test case counts just because those are easy.

Comments

Post a Comment

Popular posts from this blog

Periodical retrospectives are lame

  "You got nothing, not a single thing?! Well lets just end this here then." I remember well when I said this, being very frustrated. About ten years ago I had been working as a Scrum master for a team some months, and putting quite a lot of effort into planning our scrum teams sprint retrospectives. Lot of work also because I felt we were not getting too much out from them; not very good discussions, very few actions, and even the few actions we did come up with did not stick.  And then it happened: a retro where none of the participants came up with anything to say about the sprint. Regardless of the retro topic boxes, reading of books on retrospectives, getting inspiration from tools like retromat.org, having them in different places, using all kinds of different formats and rainbow coloured post-it notes. Not a single thing. Blank.  So then I said the words, out of frustration, mainly to myself. Why couldn't I get this thing everyone is so hyped about to work? Af...
Post a Comment
Read more

Testers doing test automation - is that the most important thing for you to do right now?

I've been thinking quite a lot about tester's moving to do test automation.  Lately beause of these three things: 1.  European testing conference , a great testing conference I attended couple of weeks ago. It is very cool due to many things; the way the speakers get compensated, the focus on the conferring side of conferences making it very easy for people to discuss stuff, the way the talks are chosen, and because there are also a lot of developers joining. So anyway when I was talking with several of the attendees, it was a bit strange how it was easier for me to talk about the product development as a whole with the developers, where as with the testers it more naturally moved into talk of automation and tools. Also on the open space, I think the majority of the topics pitched was automation or tool related. And quite little on the process or on the customer facing side of product development.  2. In my company there are a lot of different products and...
3 comments
Read more

Testing drunk

(My first blog writing ever.) I've been thinking a long time that it's funny how many bugs I find by accident. Try to do something, make a mistake and boom - a bug is found.  Making the mistakes intentionally doesn't quite work - that's why they are called accidents I guess.. So I've thought of ways to make myself more prone to accidents, coming up with an apparent one; testing drunk. TUI (testing under the influence). So this I gotta try. More to come on that later.
3 comments
Read more