In testing we can be bombarded with the notion that test automation is best. I recently read an article saying all manual testing can be replaced with a new test automation tool – what craziness! I can’t wait for someone to say all developing and coding can be automated too!
Test automation does have its place though. When a tester is repeating a step or task while testing, it’s worth investigating if that can be automated. Sometimes, writing quick and dirty automated scripts can save a lot of time. For example, when testing a web app, I often use a record and play back tool to log into the app, once logged in, I can carry on with manual testing. I can re-do these steps as many times as needed. Once the testing is finished, I throw away the automated script. At other times, for a large testing project, it’s good to invest in a test automation framework if you expect to run the same checks over and over again as regression tests.
Yet test automation always has flaws. One major flaw is the fact the automation can only check for what it is programmed to check. For example, if it is checking the login screen is displayed, it will check for this but won’t check whereabouts on the screen it is displayed, what colours, what font it is using or whether anything else is displayed on the screen.
Another flaw is that test automation can be brittle. A developer changing the layout of a screen may generate a false positive in the test automation. The tester will need to spend some time investigating why it failed and then update the script. All this adds to the maintenance costs.
From my experience, the biggest flaw is that test automation doesn’t find many bugs, and the bugs it finds don’t tend to be critical bugs compared to those found through manual testing. The main reason for this is that test automation is typically used for regression. The application has been manually tested in the past and most of the bugs have been identified, a test automation script has been developed to give confidence that the application has not regressed. One positive use for using automation is that regression testing is still required but can be and is boring to most testers – as it’s repeating the same tests executed previously without much hope of finding new bugs.
I recently read this article from Adam Goucher about test automation heuristics. It’s a good read :)
Watch out for next cartoon from Raimond Sinivee which has a similar theme.
Thanks for this post. I struggle with test automation because I haven't been in a context where it seemed to be viable or at least where investment was made it in. I have similar experience to you: I used automation tools to record "setup" tasks for a test, then I would go modify things for a particular test. I made the test scripts brittle, so that I would not feel averse to throwing them away later. They saved me time, and they supercharged my manual/exploratory abilities.
ReplyDeleteI am now at a company where I am dreading the day I will do regression tests, precisely because I don't want to do repetitive tasks, without creativity, and with little hope of finding bugs. That's what sucks the fun out of testing for me; what's the solution?
Hi Kimball,
ReplyDeleteThanks for your comment.
The only thing I would suggest is to raise your issue with your team/manager. I don't know the culture at your work or much of the context, but hopefully an issue with someone in the team is an issue for the whole team to resolve - How can we reduce the need to perform regression tests? How can we automate some of these checks? - More and more people are realising that developers tend to be the best people to write the test automation as it is mainly a programming task.