- Summarize the potential benefits and risks of test automation and tool support for testing. (K2)
- Recognize that test execution tools can have different scripting techniques, including data-driven and keyword-driven. (K1)
The reason for acquiring tools to support testing is to gain benefits, by using a software program to do certain tasks that are better done by a computer than by a person.
Advice on introducing tools into an organization can be found in web articles, magazines and books such as [Dustin et al., 1999], [Siteur, 2005] and [Fewster and Graham, 1999].
6.2.1 Potential benefits of using tools
There are many benefits that can be gained by using tools to support testing, whatever the specific type of tool. Benefits include:
- reduction of repetitive work;
- greater consistency and repeatability;
- objective assessment;
- ease of access to information about tests or testing.
Repetitive work is tedious to do manually. People become bored and make mistakes when doing the same task over and over. Examples of this type of repetitive work include running regression tests, entering the same test data over and over again (both of which can be done by a test execution tool), checking against coding standards (which can be done by a static analysis tool) or creating a specific test database (which can be done by a test data preparation tool).
People tend to do the same task in a slightly different way even when they think they are repeating something exactly. A tool will exactly reproduce what it did before, so each time it is run the result is consistent. Examples of where this aspect is beneficial include checking to confirm the correctness of a fix to a defect (which can be done by a debugging tool or test execution tool), entering test inputs (which can be done by a test execution tool) and generating tests from requirements (which can be done by a test design tool or possibly a requirements management tool).
If a person calculates a value from the software or incident reports, they may inadvertently omit something, or their own subjective prejudices may lead them to interpret that data incorrectly. Using a tool means that subjective bias is removed, and the assessment is more repeatable and consistently calculated.
Examples include assessing the cyclomatic complexity or nesting levels of a component (which can be done by a static analysis tool), coverage (coverage measurement tool), system behavior (monitoring tools) and incident statistics (test management tool).
Having lots of data doesn’t mean that information is communicated. Information presented visually is much easier for the human mind to take in and interpret. For example, a chart or graph is a better way to show information than a long list of numbers – this is why charts and graphs in spreadsheets are so useful. Special purpose tools give these features directly for the information they process. Examples include statistics and graphs about test progress (test execution or test management tool), incident rates (incident management or test management tool) and performance (performance testing tool).
In addition to these general benefits, each type of tool has specific benefits relating to the aspect of testing that the particular tool supports. These benefits are normally prominently featured in the sales information available for the type of tool. It is worth investigating a number of different tools to get a general view of the benefits.
6.2.2 Risks of using tools
Although there are significant benefits that can be achieved using tools to support testing activities, there are many organizations that have not achieved the benefits they expected.
Simply purchasing a tool is no guarantee of achieving benefits, just as buying membership in a gym does not guarantee that you will be fitter. Each type of tool requires investment of effort and time in order to achieve the potential benefits.
There are many risks that are present when tool support for testing is introduced and used, whatever the specific type of tool. Risks include:
- unrealistic expectations for the tool;
- underestimating the time, cost and effort for the initial introduction of a tool;
- underestimating the time and effort needed to achieve significant and continuing benefits from the tool;
- underestimating the effort required to maintain the test assets generated by the tool;
- over-reliance on the tool.
Unrealistic expectations may be one of the greatest risks to success with tools. The tools are only software, and we all know that there are many problems with any kind of software! It is important to have clear objectives for what the tool can do and that those objectives are realistic.
Introducing something new into an organization is seldom straightforward. Having purchased a tool, you will want to move from opening the box to having a number of people being able to use the tool in a way that will bring benefits.
There will be technical problems to overcome, but there will also be resistance from other people – both need to be addressed in order to succeed in introducing a tool. Think back to the last time you did something new for the very first time (learning to drive, riding a bike, skiing). Your first attempts were unlikely to be very good but with more experience you became much better. Using a testing tool for the first time will not be your best use of the tool either. It takes time to develop ways of using the tool in order to achieve what is possible.
Fortunately, there are some short-cuts (e.g., reading books and articles about other people’s experiences and learning from them). See also Section 6.3 for more detail on introducing a tool into an organization.
Insufficient planning for maintenance of the assets that the tool produces is a strong contributor to tools that end up as “shelf-ware”, along with the previously listed risks. Although particularly relevant for test execution tools, planning for maintenance is also a factor with other types of tool.
Tools are definitely not magic! They can do very well what they have been designed to do (at least a good quality tool can), but they cannot do everything.
A tool can certainly help, but it does not replace the intelligence needed to know how best to use it, and how to evaluate current and future uses of the tool.
For example, a test execution tool does not replace the need for good test design and should not be used for every test – some tests are still better executed manually. A test that takes a very long time to automate and will not be run very often is better done manually.
This list of risks is not exhaustive. Two other important factors are:
- the skill needed to create good tests;
- the skill needed to use the tools well, depending on the type of tool.
The skills of a tester are not the same as the skills of the tool user. The tester concentrates on what should be tested, what the test cases should be and how to prioritize the testing. The tool user concentrates on how best to get the tool to do its job effectively and how to give increasing benefit from tool use.
