Thursday, February 19, 2015
Regulations on medical software development kill more than save
After a two years as a sw testing specialist in the medical device industry, I decided to quit my job. Reasons were many, but perhaps the biggest one is the insane situation where regulations have taken the industry.
As a summary, I have come under the belief that these regulations kill more people than save. This is because of:
- the unbelievable opportunity cost
- the delay in time to market
- killing of innovation
- restricting of competition
and because of the fact that regardless of all those regulations, you still can fake it all.
Actually it is easier to fake than follow.
The main idea in the regulations is that everything needs to be planned in detail, executed accordingly, and documented to be done this way. And by planned in detail, they mean that there needs to be a documented plan, in which all the actions taken needs to be specified. If you do something that was not in the plan, you are in trouble. If you don't do something that was in the plan - boy are you in trouble. The only way to create a plan that has everything you will do and nothing you will not, well documented, is a plan that is written afterwards. So the only way to do audit proof plans is to create them after stuff has already been done. Smart.
Everything that is done, also needs to be done following well established processes, of course. And you need to also prove that processes have been followed. This way, the audit proof way of creating processes is to add stuff into them that can be easily proven to have happened, and that do not generate too much load in documentation side when proving to have happened, This means that the processes are written for the auditors, and not the people who should follow them. So the manageable way to do the process dance is to have the written process generating the documentation, and an actual process generating the product. Efficient.
And when the auditors pop by, they always find issues (probably the easiest job in the world to find them), which need to be rectified, by including something new into the process, which again grows the part done for the auditors. If you actually find something that you should improve on, you must not write it to a process cause then you need to prove that you are following it, which means arbitrary metrics and more documentation. And also by including stuff into the process, you hang yourself on not being able to improve and change it efficiently, so better not to include it. Real atmosphere for continuous improvement.
You may at this point think that what is so bad in writing a documentation. I'll tell you what is. First, anything you write into a document needs to be carefully considered on how the auditors would interpret them, which is really burdening. Secondly, all documents need to be reviewed and authorized after EVERY change you do. Which is horrible. Thirdly, there is a 99% chance that no-one will ever read those documents, which is not the most motivating thing either.
I could rant about so many things, but as this is a testing blog I'll switch into that now.
Nothing, absolutely nothing is so badly misunderstood by the auditors than testing is. Testing for them means verification of requirements, in a preplanned, pre-reviewed, extremely detailed test cases having descriptions on what to do, and what is the expected result. The tests must be run exactly as detailed, and objective evidence needs to be provided on each result. So if you go and execute a test case, and everything works perfectly but some step cannot exactly be followed the way it is written, you need to rewrite, re-review, and rerun the test. The only way to be able to survive this is to write the test case while executing the readily built system, and writing the step by step instructions. And even then, it is really hard to write, and really burdening to maintain. The tests are not designed or executed to find problems, but to prove that those tests can be run. And because of the huge amount of work put into them, you generally like to keep the amount of new tests as minimal as possible. As the actual tests quality or coverage is not that important either.
This has nothing to do with sw testing, but this is what auditors require. As a funny anecdote, while I was trying to push for more high level tests and more skills for the testers, we got a warning from the auditors stating we need more detailed test cases, detailed on the level that anyone could run them. What probably makes it even more painful is, that as in the medical domain there are a lot of scientists, there seems to be a general understanding that good testing means and requires exploration and investigation. But that is not at all what the auditors expect. And it is hard to keep on pushing and pushing to do a good job in testing, while what is expected from you is just a demonstration of bad checking.
Funnily enough, working as a sw tester I have nowhere else tested so little as I have tested here. Or perhaps funny is not the right word here.
Talk about the hell of a tester.
I must raise my hat to my fellow colleagues who keep on trying to do best possible work under these conditions. Who try to do everything keeping in mind the customer, while trying to please the auditors. I could not, and will now try to do my bit in changing the way auditing is being done on the medical domain from the outside. I hope that I can return again to the domain, either as a stronger person, or preferably into a domain where the regulations are actually working.
If any auditor by any chance would happen to read this, lets talk, please. What we need in this domain is more communication and collaboration, and focus on what it is we really are trying to do. Provide safe, working products to people who should be able to rely on them.