Removing humans from the task of testing software will benefit from expert systems that encode domain expertise for various types of software. Software engineers will be able to reference from their computer-readable specs (crspec) various modules in those expert systems, and write a small amount of code to encode the specifics for their cases. The system will do the rest of the testing – functionality, performance, stress, localization, security, and so on.
Suppose an SDET has to test a RESTful service implemented as a WCF service on IIS. A domain expert system could have a module that encodes the knowledge of what an WCF IIS RESTful service is. The SDET could just “drag and drop” the module into the crspec, and could write a little code or markup to define the inputs and outputs of the service. The system will then do all the remaining work of testing the service across the myriads of configurations and inputs, such as:
- Deployment of the service across different but compatible IIS configurations and operating systems
- Functionality of the service across all the equivalence classes defined by the RESTful module and the SDET definitions
- Longevity, performance, robustness, and scalability of the service – under normal or stress conditions
To understand how these testing capabilities could be derived automatically from the expert system and the configuration done by the SDET, consider the longevity, performance, and stress testing. The definitions by the SDET for the operations supported by the service can be used to automatically generate code that will request those operations from the service.
From there, requesting those operations in various modes will provide the testing, for example:
- Longevity testing – requesting the operations millions of times from one or a small number of clients in a manner that gives enough time to the service to respond, and validating that the memory consumption of the service stays constant
- Performance testing – requesting the operations a number of times from an increasing number of simultaneous clients, and measuring and reporting the time the service takes to handle the requests
- Stress testing – requesting the operations as quickly as possible from a number of clients significantly exceeding the expected capability of the service to handle them simultaneously, and validating that the service doesn’t crash, and that after the stress has stopped, the service continues to operate adequately
Properties of a reusable domain expertise system
Based on the example above, a reusable domain expertise system would need the following properties:
- Extensible – new modules can be added for domains which the system does not currently support; existing modules can be updated with more capabilities
- Searchable – customers can search for and discover what modules the system supports
- Descriptive – customers can review and understand what the available modules will and will not cover
- Integrated – the expert system will have to play well with the other components in a testing system, such as the crspec or the configuration database (see previous post)
- Comprehensive – the more modules and validations the system can handle, the higher the value it provides
- Trusted – customers will have to trust that the system will accomplish the promised validations and testing
The business justification
To see why anyone would take the time to create this domain expertise system, consider RESTful services as a class (not specific to just WCF/IIS). While it’s hard to know for certain how many such services have been written, it’s likely that there are more than a million of them in production at the time of writing. After all, just in the smart phone space, it would be fair to assume that half of the two million applications available for iPhone and Android smart phones talk to a proprietary back end service.
And each of these services needs to be tested for correct functionality, performance, stress, security, and more. A number of them may not have been tested extensively due to the current high cost of testing them the old “manual” way – which could be prohibitively expensive for small companies and startups. Or, they could have been tested partially (for example for correct functionality), only to discover in production that as the product becomes successful, the performance cannot meet the increasing customer demand.
The proposed reusable domain expertise system could reduce dramatically the cost of testing for both big and small companies by reducing the amount of human labor that needs to go into the testing.
Relevant additional information