5.2.2 What to do with your brain while planning tests
Writing a good test plan is easier than writing a novel, but both tasks require an organized approach and careful thought. In fact, since a good test plan is kept short and focused, unlike some novels, some might argue that it’s harder to write a good test plan. Let’s look at some of the planning tasks you need to carry out.
At a high level, you need to consider the purpose served by the testing work. In terms of the overall organizational needs, this purpose is referred to variously as the test team’s mission or the organization’s testing policy. In terms of the specific project, understanding the purpose of testing means knowing the answers to questions such as:
- What is in scope and what is out of scope for this testing effort?
- What are the test objectives?
- What are the important project and product risks? (more on risks in Section 5.5).
- What constraints affect testing (e.g., budget limitations, hard deadlines, etc.)?
- What is most critical for this product and project?
- Which aspects of the product are more (or less) testable?
- What should be the overall test execution schedule and how should we decide the order in which to run specific tests? (Product and planning risks, discussed later in this chapter, will influence the answers to these questions.)
You should then select strategies which are appropriate to the purpose of testing (more on the topic of selecting strategies in a few pages).
In addition, you need to decide how to split the testing work into various levels, as discussed in Chapter 2 (e.g., component, integration, system and acceptance). If that decision has already been made, you need to decide how to best fit your testing work in the level you are responsible for with the testing work done in those other test levels. During the analysis and design of tests, you’ll want to reduce gaps and overlap between levels, and, during test execution, you’ll want to coordinate between the levels. Such details dealing with inter-level coordination are often addressed in the master test plan.
In addition to integrating and coordinating between test levels, you should also plan to integrate and coordinate all the testing work to be done with the rest of the project. For example, what items must be acquired for the testing?
Are there on-going supply issues, such as with imitation bills (i.e., simulated banknotes) for a financial application such as an ATM? When will the programmers complete work on the system under test? What operations support is required for the test environment? What kind of information must be delivered to the maintenance team at the end of testing?
Moving down into the details, what makes a plan a plan – rather than a statement of principles, a laundry list of good ideas or a collection of suggestions – is that the author specifies in it who will do what when and (at least in a general way) how. Resources are required to carry out the work. These are often hard decisions that require careful consideration and building a consensus across the team, including with the project manager.
The entire testing process – from planning through to closure – produces information, some of which you will need to document. How precisely should testers write the test designs, cases and procedures? How much should they leave to the judgment of the tester during test execution, and what are the reproducibility issues associated with this decision? What kinds of templates can testers use for the various documents they’ll produce? How do those documents relate to one another? If you intend to use tools for tasks such as test design and execution, as discussed in Chapter 6, you’ll need to understand how those tools capture documentation while working on the plan.
Some information you’ll need to gather in the form of raw data and then distill. What metrics to do you intend to use to monitor, control and manage the testing? Which of those metrics – and perhaps other metrics – will you use to report your results? We’ll look more closely at possible answers to those questions in Section 5.3, but a good test plan provides answers early in the project.
Finally, moving back up to a higher level, think about what would be true about the project when the project was ready to start executing tests. What would be true about the project when the project was ready to declare test execution done? At what point can you safely start a particular test level or phase, test suite or test target? When can you finish it? The factors to consider in such decisions are often called “entry criteria” and “exit criteria”.
For such criteria, typical factors are:
- Acquisition and supply: the availability of staff, tools, systems and other materials required.
- Test items: the state that the items to be tested must be in to start and to finish testing.
- Defects: the number known to be present, the arrival rate, the number predicted to remain, and the number resolved.
- Tests: the number run, passed, failed, blocked, skipped, and so forth.
- Coverage: the portions of the test basis, the software code or both that have been tested and which have not.
- Quality: the status of the important quality characteristics for the system.
- Money: the cost of finding the next defect in the current level of testing compared to the cost of finding it in the next level of testing (or in production).
- Risk: the undesirable outcomes that could result from shipping too early (such as latent defects or untested areas) – or too late (such as loss of market share).
When writing exit criteria, we try to remember that a successful project is a balance of quality, budget, schedule and feature considerations. This is even more important when applying exit criteria at the end of the project.
