you're reading...

Automate away the SDET role – English optional in specs

The previous post “The last automation – let software take over the SDET work” introduced the idea that software systems could take a large and increasing amount of the work on testing software that is currently done by people. The current post will describe at a high level what the feature spec work would look like in the new way of doing things as compared to the traditional specification process, and what the new job of the SDETs would be.

Consider a project for which the business requirements have been understood, and the Program Manager (PM) is starting to write the functional specification – which will describe in details the features, the use cases, and the non-functional requirements.

A traditional specification versus a computer-readable specification

Today, the PM would write the functional specification in a natural language such as English. The tool of choice would be a word processor that creates a humanly readable document in a Word, PDF, or HTML format. The stakeholders would review this document, provide feedback, and sign-off on the humanly-readable version of that document. The spec may include lengthy and detailed formal specifications of expected behavior, but ultimately a human would need to understand the details and test the software – either through creating test automation or manually. The written text in the specification is the specification.

“In a traditional specification, the text in the specification is the specification.”


In the proposed new state of affairs, the specification would be written in a very different tool. Rather than a word processor, the tool would resemble much more a rule-based system in which the PM will define the behaviors and constraints of the software. Since this new specification can be parsed by a computer, we’ll call it a computer-readable spec, or crspec for short.

A typical example

Consider this English sentence which is typical of any functional specification: “The application works on Windows Server 2008 (amd64 only) and on Windows Server 2008 R2 (amd64 only)”. In a crspec the PM would not write this sentence. Rather, assuming a reasonable GUI interface, the PM would just select the two items from a list of supported operating systems:

Currently: Natural language text Proposed: Computer-readable spec (crspec)
The application works on:

  • Windows Server 2008 (amd64 only)
  • Windows Server 2008 R2 (amd64 only)

Similarly, consider this paragraph from what would be the typical specification for installing the product: “Prerequisites A, B, and C must be installed on the system. To install the application, execute ‘setup.exe /arg1’ on a supported platform. After setup completes, the application will be installed in Program Files and a shortcut will be placed on the desktop…”.

In the envisioned tool for creating and editing crspecs, this requirement would be described as a rule that specifies:

  1. The ID of the initial configuration (e.g. “Supported_OSs_with_prerequisites_A_B_C”)
  2. The action (which would be another rule that encodes how it is executed)
  3. The post-action validation (or post-condition) which describes how to validate that the operation was successful
  4. The ID of the post-action configuration, so that it can be used in subsequent rules (e.g. “App_successfully_installed”)
Currently: Natural language text Proposed: Computer-readable spec (crspec)
Use Case: Installing the application:

  • Use a supported platform
  • Prerequisites A, B, and C are installed
  • Run “setup.exe /arg1”
  • Result:
    • The application is installed in Program Files
    • A shortcut is created on the desktop

In this new tool, English could still be used for the purpose of adding notes or providing context of why certain decisions were made. However, the English content is not the specification. The specification is the set of rules and requirements defined in the tool.

“In a crspec, the text is not the specification. The rules and requirements defined in the crspec tool are the specification.”

Once these rules and requirements are defined in the tool, a computer can execute them and find out whether the application that was build meets the encoded requirements. For example, in the case of the above rule “Installing the application”, the computer would:

  1. Obtain the initial configuration – which can be either cached, or built on demand based on the rules
  2. On that configuration invoke the specified action – which would run setup.exe
  3. On that configuration, run the code for the post-action validation – which will report whether the  expected behaviors have occurred
  4. On success, set the state of the new configuration as “App_successfully_installed”, and possibly cache this configuration for efficient future retrieval for other rules that need this configuration as they starting one.
  5. On failure, report the problem and take any other actions that may be defined

In the next posts we’ll delve into more complex examples to demonstrate how more complex expectations and behaviors can be encoded. For now, just consider how different this approach is.

In the traditional approach, the SDET would:

  • Read the specification and then write the automation to test that the product can be installed on all  supported platforms
  • If the automated tests fail, investigate the failures, investigate the code, collect traces and other information and log bugs
  • If bugs are found in the field, debug the product to understand the bug and where it is in the code

In the proposed approach, the PM defines the rules in the crspec et voila – the software tools can test the product without a human intervention and can file the bugs when the behavior does not meet the encoded expectations. The work of the SDET here would be to:

  • Read the crspec rules and then write only the piece of code for which the configuration and action  databases don’t have a default implementation yet (for example the post-action validation)
  • If bugs are found by the automated testing system, do nothing – the developers will have all the information from the crspec and the output of the testing system
  • If bugs are found in the field, debug the rules in the crspec to understand why the problem was not found by the automated system

In any event, the total PM and SDET work will be much less than in the traditional approach, hence increasing the productivity of all the disciplines engaged in the software creation process.

But wait, this example assumes too much!

Yes. This example assumes that the crspec writing tool is connected to a database for configurations and another database for actions, and that there is another system that knows how to read the rules from the crspec and how to execute all the actions.

These assumptions are neither farfetched, nor unpractical.

Consider the configurations database – creating one that is publicly available to any third party and comes with some support statements would be good business for both the author of such a database, and for the users. Since this database does not exist today, every SDET today who works on automating install-and-deploy test cases must create her own solution for provisioning the configurations. Likewise, any enterprise who needs to install software has to do the work independently. As a result, this works gets repeated untold number of times across the software industry, and across enterprises that need to deploy software. If such a system was created, it would significantly reduce the overhead across the industry since the encoded deployment rules would be able to satisfy the vast majority of needs without any or with only slight extensions.

Without going into the obvious justification, creating an actions database can have likewise similar benefits to both the owner and the users of such a database.

In summary

Some will say “But wait, we have model-based testing and other formal approaches, what you are describing here already exists.” Here are some simple questions that will help you determine if you are using the traditional approach or the new approach:

Question Answer in the traditional approach Answer in the crspec approach
What constitutes  the feature specification? A textual document in a natural language (such as English) that contains descriptions of the software and expected functionality to be built. The natural language content and any other written formalisms are the specification. A computer readable specification encoding a set rules and requirements that describes the software that will be built. Natural-language content such as English is optional and serve only for additional notes.
What is the role of the Program Manager? Writes in a natural language and keeps up-to-date the document that describes the  software to be built. Creates and keeps up-to-date the rules that describe in a computer-readable manner the  expectations and constraints of the software to be built.
What is the role of the SDET?

Writes code for automated test cases and/or creates manual test cases.

Manages these test cases in Test Case Management (TCM) systems.

When automated tests fail, debugs the test output and the code to determine the source of the failure, files bugs with logs, repro steps, and other information.

When customers report problems (i.e. when bugs were missed by the testing), reviews specifications, emails, and other communication to determine why the problems were not found through the test process.

Writes code for actions and validations for which defaults don’t exist yet.

Augments the configurations and/or actions database to prevent the need to write that same code a second time, ever.

When automated tests fail, does nothing – the system has all the knowledge to report the expected and actual behavior.

When customers report problems (i.e. when bugs were missed by the testing), debugs the rules in the crspec to determine why the problems were not found through the test process.

How does the product team answer the question “Is scenario ABC supported on configuration XYZ?” Manually – by reading the specifications and any related documents and communications. Automatically – the automated testing system answers that question based on the encoded rules and constraints.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: