CHAPTER 5 – 5.2 Test Plans, Estimates and Strategies – Part 1/5

1. Recognize the different levels and objectives of test planning. (K1)
2. Summarize the purpose and content of the test plan, test design specification and test procedure documents according to [IEEE 829]. (K2)
3. Recall typical factors that influence the effort related to testing. (K1)
4. Differentiate between two conceptually different estimation approaches: the metrics-based approach and the expert-based approach. (K2)
5. Differentiate between the subject of test planning for a project, for individual test levels (e.g. system test) or specific test targets (e.g. usability test), and for test execution. (K2)
6. List test preparation and execution tasks that need planning. (K1)
7. Recognize and justify adequate exit criteria for specific test levels and groups of test cases (e.g., for integration testing, acceptance testing or usability testing). (K2)

In this section, let’s talk about a complicated trio of test topics: plans, estimates and strategies. Plans, estimates and strategies depend on a number of factors, including the level, targets and objectives of the testing we’re setting out to do. Writing a plan, preparing an estimate and selecting test strategies tend to happen concurrently and ideally during the planning period for the overall project, though we must be ready to revise them as the project proceeds and we gain more information.

Let’s look closely at how to prepare a test plan, examining issues related to planning for a project, for a test level or phase, for a specific test type and for test execution. We’ll examine typical factors that influence the effort related to testing and see two different estimation approaches: metrics-based and expert-based. We’ll discuss selecting test strategies and ways to establish adequate exit criteria for testing. In addition, we’ll look at various tasks related to test preparation and execution that need planning.

As you read, keep your eyes open for the glossary terms entry criteria, exit criteria, exploratory testing, test approach, test level, test plan, test procedure and test strategy.

5.2.1 The purpose and substance of test plans

While people tend to have different definitions of what goes in a test plan, for us a test plan is the project plan for the testing work to be done. It is not a test design specification, a collection of test cases or a set of test procedures; in fact, most of our test plans do not address that level of detail.

Why do we write test plans? We have three main reasons.

First, writing a test plan guides our thinking. We find that if we can explain something in words, we understand it. If not, there’s a good chance we don’t.

Writing a test plan forces us to confront the challenges that await us and focus our thinking on important topics. In Chapter 2 of Fred Brooks’ brilliant and essential book on software engineering management, The Mythical Man-Month, he explains the importance of careful estimation and planning for testing as follows:

Failure to allow enough time for system test, in particular, is peculiarly disastrous. Since the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the delivery date [and] delay at this point has unusually severe … financial repercussions. The project is fully staffed, and cost-perday is maximum [as are the associated opportunity costs]. It is therefore very important to allow enough system test time in the original schedule.

[Brooks, 1995]

We find that using a template when writing test plans helps us remember the important challenges. You can use the IEEE 829 test plan template shown in this chapter, use someone else’s template, or create your own template over time.

The test planning process and the plan itself serve as vehicles for communicating with other members of the project team, testers, peers, managers and other stakeholders. This communication allows the test plan to influence the project team and the project team to influence the test plan, especially in the areas of organization-wide testing policies and motivations; test scope, objectives and critical areas to test; project and product risks, resource considerations and constraints; and the testability of the item under test.

You can accomplish this communication through circulation of one or two test plan drafts and through review meetings. Such a draft will include many notes such as the following:

[To Be Determined: Jennifer: Please tell me what the plan is for releasing the test items into the test lab for each cycle of system test execution?] [Dave – please let me know which version of the test tool will be used for the regression tests of the previous increments.]

As you document the answers to these kinds of questions, the test plan becomes a record of previous discussions and agreements between the testers and the rest of the project team.

The test plan also helps us manage change. During early phases of the project, as we gather more information, we revise our plans. As the project evolves and situations change, we adapt our plans. Written test plans give us a baseline against which to measure such revisions and changes. Furthermore, updating the plan at major milestones helps keep testing aligned with project needs. As we run the tests, we make final adjustments to our plans based on the results. You might not have the time – or the energy – to update your test plans every time a variance occurs, as some projects can be quite dynamic. In Chapter 6 [Black, 2001], we describe a simple approach for documenting variances from the test plan that you can implement using a database or spreadsheet. You can include these change records in a periodic test plan update, as part of a test
status report, or as part as an end-of-project test summary.

We’ve found that it’s better to write multiple test plans in some situations. For example, when we manage both integration and system test levels, those two test execution periods occur at different points in time and have different objectives. For some systems projects, a hardware test plan and a software test plan will address different techniques and tools as well as different audiences.

However, since there might be overlap between these test plans, a master test plan that addresses the common elements can reduce the amount of redundant documentation.