“Never send a human to do a computer’s job”
Unknown, circa 1990
During the past 20 years, software engineering has benefited from tremendous improvements. We have more powerful tools than ever for writing and testing software. The training materials and books are better than ever. Frequently, solutions on a wide variety of subjects are readily discoverable online. All these improvements have made us software engineers – including SDET-s (Software Design Engineer in Test – the folks who write automation to validate the software works as expected) – more productive by orders of magnitude compared to 20 years ago.
And yet, we continue to create software test automation in the same way we did it earlier – manually. We read specs, we design and implement layers of code, we combine them into test cases that encode the operations and the expected outcomes, and we manage them in test-case management systems. As any manual work, this process is a laborious, error-prone, expensive and slow.
In order to make the next jump in software development productivity, we need to transcend this manual process.
The opportunities are clearly there – instead of developing tests manually, we could build and use systems with which will do the work for us. As SDETs, we will switch from manually writing test automation to defining what the valid configurations are under which the software must run, how the software should behavior, and what outputs and other side effects it should produce.
Consider what a test case does – on a particular configuration of the external environment (for example a particular version of the OS) the test case configures the system under test (SUT) in a particular state. Then it provides some input to the SUT. Finally, it confirms that the output produced by the SUT, the change to its internal state, and the change the SUT effected on the environment all match the expectations of the test case.
Therefore, if a software system has the capabilities 1) to describe for an SUT the supported environments and expected behaviors based on the inputs and prior state, and 2) to deploy the environment and SUT, invoke the input, and then observe the output, and 3) be extensible, intuitive, and easy enough to use by the general population of software engineers, then that system will be able to replace the automation work done by SDETs.
The benefits of such systems could be significant. Every SDET who is freed from his or her current work on automating tests that otherwise could be trivially automated could focus on higher-value automation work or join the software development team – in either case improving the velocity of software production.
The numbers are staggering – by some estimates, the cost of SDETs in the US alone may be as much as $30 billion dollars or more per year. Even a small reduction of that cost could represent significant savings – enough to justify the initial investments.
Creating these systems won’t be easy, and the systems won’t be great at first. Whenever a new technologies has replaced an old one, the process usually has started with a solution that handled the more straightforward aspects of a job and as it became more sophisticated, it continuously took over the more advanced aspects.
We should expect the creation of these systems to follow a similar path. The initial investments that will be needed to get such systems off the ground won’t be trivial, but the big players – Amazon, IBM, Apple, Google, and the like – employ enough SDETs to make those investments profitable within a reasonable timespan.
In the next several posts I’ll go in some detail on how we can build these systems and some good problems to tackle first in order to achieve initial success which will motivate subsequent investments. For now, just remember – we currently have all the tools and knowledge required build those systems that can replace cost-effectively the majority of SDET work currently done by humans. These systems are not a futuristic dream – they are a tractable problem waiting to be solved.
Relevant additional information