Today, our developers and testers were asked to justify the use of test automation - a surprising question after we’ve invested 5 years in writing automated test cases.

The challenge was to prove the value in continuing to automate our test cases, on the basis that it should be up to scrutiny if the value really does exist.

So we tried…

Automated tests are repeatable and consistent
Automated tests and testing platforms can be easily scaled as the code base grows
Automated tests can be executed concurrently against many environments
Automated tests can provide rapid feedback on system/code changes

Of course, the same isn’t true for manual testing

Manual tests can be unpredictable
- different testers may produce different results
- testers may use workarounds to avoid some bugs
Manual testing isn’t scalable - employing more people is the only option
An individual tester can only (sensibly) test one environment at a time
Manual tests are slow - feedback might take days, weeks or months

But these weren’t the right answers.

We couldn’t understand why.

So, lets have a closer look at automation…

What is automation?

Automation, by definition, is:

the technique, method, or system of operating or controlling a process by highly automatic means, as by electronic devices, reducing human intervention to a minimum

Originally used circa 1940, the word was an irregular formation combining “automatic” and “action”, but process automation had become a well established practice long before then.

Everything from manufacturing and agriculture to construction and transportation - in modern history, humans have automated nearly every aspect of their lives.

On an industrial scale the benefits are immediately obvious. A farmer wouldn’t employ manual labour to plough fields any more than a car manufacturer would employ manual labour to assemble cars.

The return on investment (ROI) for the automation of these processes is clear - though of course it wasn’t in the decades leading up to the industrial revolution.

Why do we automate things?

There are many reasons to automate processes - some are purely economic while others are psychological.

Simple or boring tasks (paying bills)
Time consuming tasks (washing dishes)
Beyond our physical capability (lifting shipping containers)
To reduce cost (human labour)
To reduce risk (bomb disposal robots)

In all cases, our ability to automate something is limited by our mental capacity to perform that task. We can only automate the things we understand, that are simple enough and repeatable enough. We can’t easily automate tasks requiring creativity or emotion.

For example, we can easily automate opening a shop door (we could do it manually), but we would find it difficult to automate brain surgery (most humans couldn’t do it even manually) or software engineering (many have tried).

But sometimes, even when a process can be automated, we decide not to.

Why do we choose not to automate things?

Just as there are some processes we would like to automate, but can’t - there are some processes we could automate, but don’t:

Things we enjoy doing
Economic cost, e.g. R&D investment is too high or unpredictable
Social cost, e.g. unemployment and poverty

Like our ability to automate is limited by our mental capacity to perform a task, our ability to choose not to automate is equally limited by our physical limitations in performing a task (we wouldn’t even consider using human labour to lift a shipping container).

How we decide what to automate

Deciding whether we automate a process comes down to a cost-benefit analysis, determining if the investment required (whether an economic, physical or psychological investment) is worth the benefit we get in return.

As with all cost-benefit analysis, the time-frame over which we calculate the costs and benefits can have a considerable impact on the ROI.

For example, if Ford had only planned to make 1000 cars over a 2 year time frame, then it would be obvious that the ROI on designing, building, testing and deploying an automated car manufacturing process would be terrible, and would probably result in a net loss (or bankruptcy) for the company.

But if Ford wanted to continue producing cars - maybe another 350 million cars over a 109 years - then the ROI becomes far more appealing.

Although the up-front investment in research and development is high, the long-term benefit of this is exponentially higher, ultimately making Ford one of the world’s leading car manufacturers and forging the modern automotive industry.

Why software testing is no different

Just like agriculture and manufacturing, automating software testing comes with a high initial (and sometimes on-going) cost:

Developers and testers need to learn how to write automated tests
Test suites need to be written and maintained
An automated testing platform must be created

And just like agriculture and manufacturing, some of it doesn’t need automation (or can’t be automated):

If it’s throwaway/one-use code
Exploratory testing which requires creativity
Visual testing (does it look/feel right)

But in most cases, well written automated tests provide a level of confidence unmatched by manual testing:

Entire system components can be updated or replaced efficiently
Codebases can be safely refactored
Integration and release can be automated
Fixed defects can’t regress
More platforms can be tested (desktop, web, mobile, etc)

By developing an automated testing suite, testing resources can then be reallocated to more productive work:

Improving test coverage
Collaborating with developers
Exploratory and visual testing
Accessibility testing

So, what was the answer?

It certainly won’t be “because we should” or “it’s the right thing to do”, or even “it’ll reduce defects” or “it’ll improve code quality”.

It will come down to proving, through cost-benefit analysis, that the investment in automated testing provides a strong enough ROI. This will largely depend on the time frame used for the ROI calculation.

If the focus is short-term (“we want a great product now”) then any further investment in test automation will yield no value, and manual testing is the only choice.

But if the focus is long-term (“we still want a great product in 5 years”) then test automation is invaluable (supplemented with appropriate manual testing), and provably so in any cost-benefit analysis.

Is there a middle ground?

The middle ground does seem attractive:

Manual testing to get a quick delivery
Automate tests longer-term

It seems to promise a good short and long term ROI. We get our quick delivery, to an acceptable standard. We also get our test automation. And eventually we get a high quality product.

But until the test automation happens, developers are constrained by the existing codebase:

refactoring becomes difficult or impossible
updating components carries significant risk
minor changes, bug fixes and features take an inordinate amount of time to develop, test and deploy

This has a substantial consequence for the product or service being delivered:

If test automation never happens (no time is made available), the entire product will suffer and eventually adding new features or fixing bugs will become impossible
If test automation happens (quickly), new features will be held up while test suites are automated, delaying the creation of business value

Either way, the middle ground eventually becomes technical debt, and the short term business value gained through a reduction in the initial investment must eventually be repaid (through reduced longer-term value).

A cost-benefit analysis

Many cost-benefit analysis of test automation have already been carried out, so I’m not going to write “Yet Another Cost-Benefit Analysis” - but here’s a few links instead:

Conclusion

Given the historical importance of process automation throughout the industrial revolution, the rapid improvement to standards of living that we’re still benefiting from today, and the significant expansion of the human race as a result of the earliest technological automation, it seems counter-intuitive to even question the value in automating software testing.

Though I agree that some tests shouldn’t be automated (or can’t be), when products or services are expected to have a long “shelf-life”, test automation becomes the only sensible solution.

It’s also important to consider the human element in any cost-benefit analysis.

Testers and developers, like anyone else, get bored easily when tasked with simple and repetitive work. If we have the opportunity to automate this work, we leave humans with the more complex and creative work - the stuff we’re really good at, the stuff that we can’t automate, and the stuff that’s just more satisfying to do.

The automation what-for