6.1.2 Tool support for management of testing and tests
What does “test management” mean? It could be “the management of tests” or it could be “managing the testing process”. The tools in this broad category provide support for either or both of these. The management of testing applies over the whole of the software development life cycle, so a test management tool could be among the first to be used in a project. A test management tool may also manage the tests, which would begin early in the project and would then continue to be used throughout the project and also after the system had been released. In practice, test management tools are typically used by specialist testers or test managers at system or acceptance test level.
Test management tools
The features provided by test management tools include those listed below. Some tools will provide all of these features; others may provide one or more of the features, however such tools would still be classified as test management tools.
Features or characteristics of test management tools include support for:
- management of tests (e.g., keeping track of the associated data for a given set of tests, knowing which tests need to run in a common environment, number of tests planned, written, run, passed or failed);
- scheduling of tests to be executed (manually or by a test execution tool);
- management of testing activities (time spent in test design, test execution, whether we are on schedule or on budget);
- interfaces to other tools, such as:
- test execution tools (test running tools);
- incident management tools;
- requirement management tools;
- configuration management tools;
- traceability of tests, test results and defects to requirements or other sources;
- logging test results (note that the test management tool does not run tests, but could summarize results from test execution tools that the test management tool interfaces with);
- preparing progress reports based on metrics (quantitative analysis), such as:
- tests run and tests passed;
- incidents raised, defects fixed and outstanding.
This information can be used to monitor the testing process and decide what actions to take (test control), as described in Chapter 5. The tool also gives information about the component or system being tested (the test object). Test management tools help to gather, organize and communicate information about the testing on a project.
Requirements management tools
Are requirements management tools really testing tools? Some people may say they are not, but they do provide some features that are very helpful to testing.
Because tests are based on requirements, the better the quality of the requirements, the easier it will be to write tests from them. It is also important to be able to trace tests to requirements and requirements to tests, as we saw in Chapter 2.
Some requirements management tools are able to find defects in the requirements, for example by checking for ambiguous or forbidden words, such as “might”, “and/or”, “as needed” or “(to be decided)”.
Features or characteristics of requirements management tools include support for:
- storing requirement statements;
- information about requirement attributes;
- checking consistency of requirements;
- identifying undefined, missing or ‘to be defined later’ requirements;
- prioritizing requirements for testing purposes;
- traceability of requirements to tests and tests to requirements, functions or features;
- traceability through levels of requirements;
- interfacing to test management tools;
- coverage of requirements by a set of tests (sometimes).
Incident management tools
This type of tool is also known as a defect-tracking tool, a defect-management tool, a bug-tracking tool or a bug-management tool. However, “incident management tool” is probably a better name for it because not all of the things tracked are actually defects or bugs; incidents may also be perceived problems, anomalies (that aren’t necessarily defects) or enhancement requests. Also, what is normally recorded is information about the failure (not the defect) that was generated during testing – information about the defect that caused that failure would come to light when someone (e.g., a developer) begins to investigate the failure.
Incident reports go through a number of stages from initial identification and recording of the details, through analysis, classification, assignment for fixing, fixed, re-tested and closed, as described in Chapter 5. Incident management tools make it much easier to keep track of the incidents over time.
Features or characteristics of incident management tools include support for:
- storing information about the attributes of incidents (e.g., severity);
- storing attachments (e.g., a screen shot);
- prioritizing incidents;
- assigning actions to people (fix, confirmation test, etc.);
- status (e.g., open, rejected, duplicate, deferred, ready for confirmation test, closed);
- reporting of statistics/metrics about incidents (e.g., average time open, number of incidents with each status, total number raised, open or closed).
Incident management tool functionality may be included in commercial test management tools.
Configuration management tools
An example: A test group began testing the software, expecting to find the usual fairly high number of problems. But to their surprise, the software seemed to be much better than usual this time – very few defects were found. Before they celebrated the great quality of this release, they just made an additional check to see if they had the right version and discovered that they were actually testing the version from two months earlier (which had been debugged) with the tests for that earlier version. It was nice to know that this was still OK, but they weren’t actually testing what they thought they were testing or what they should have been testing.
Configuration management tools are not strictly testing tools either, but good configuration management is critical for controlled testing, as was described in Chapter 5. We need to know exactly what it is that we are supposed to test, such as the exact version of all of the things that belong in a system. It is possible to perform configuration management activities without the use of tools, but the tools make life a lot easier, especially in complex environments.
Testware needs to be under configuration management and the same tool may be able to be used for testware as well as for software items. Testware also has different versions and is changed over time. It is important to run the correct version of the tests as well, as our earlier example shows.
Features or characteristics of configuration management tools include support for:
- storing information about versions and builds of the software and testware;
- traceability between software and testware and different versions or variants;
- keeping track of which versions belong with which configurations (e.g., operating systems, libraries, browsers);
- build and release management;
- baselining (e.g., all the configuration items that make up a specific release);
- access control (checking in and out).
