1.1.6 How much testing is enough?
Testing Principle – Exhaustive testing is impossible.
Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, we use risks and priorities to focus testing efforts
We’ve seen that testing helps us find defects and improve software quality. How much testing should we do? We have a choice: test everything, test nothing or test some of the software. Now, your immediate response to that may well be to say, “Everything must be tested”. We don’t want to use software that has not been completely tested, do we? This implies that we must exercise every aspect of a software system during testing. What we need to consider is whether we must, or even can, test completely.
Let’s look at how much testing we’d need to do to be able to test exhaustively. How many tests would you need to do to completely test a one-digit numeric field? The immediate question is, “What you mean by test completely?”
There are 10 possible valid numeric values but as well as the valid values we need to ensure that all the invalid values are rejected. There are 26 uppercase alpha characters, 26 lower case, at least 6 special and punctuation characters as well as a blank value. So there would be at least 68 tests for this example of a one-digit field.
This problem just gets worse as we look at more realistic examples. In practice, systems have more than one input field with the fields being of varying sizes. These tests would be alongside others such as running the tests in different environments. If we take an example where one screen has 15 input fields, each having 5 possible values, then to test all of the valid input value combinations you would need 30 517 578 125 (515) tests! It is unlikely that the project timescales would allow for this number of tests.
Testing our one-digit field with values 2, 3 and 4 makes our tests more thorough, but it does not give us more information than if we had just tested with the value 3.
Pressures on a project include time and budget as well as pressure to deliver a technical solution that meets the customers’ needs. Customers and project managers will want to spend an amount on testing that provides a return on investment for them. This return on investment includes preventing failures after release that are costly. Testing completely – even if that is what customers and project managers ask for – is simply not what they can afford.
Instead we need a test approach which provides the right amount of testing for this project, these customers (and other stakeholders) and this software. We do this by aligning the testing we do with the risks for the customers, the stakeholders, the project and the software. Assessing and managing risk is one of the most important activities in any project, and is a key activity and reason for testing. Deciding how much testing is enough should take account of the level of risk, including technical and business risks related to the product and project constraints such as time and budget.
We carry out a risk assessment to decide how much testing to do. We can then vary the testing effort based on the level of risk in different areas.
Additionally, testing should provide sufficient information to stakeholders to make informed decisions about the release of the software or system we’re testing, for the next development step or handover to customers. The effort put into the quality assurance and testing activities needs to be tailored to the risks and costs associated with the project. Because of the limits in the budget, the time, and in testing we need to decide how we will focus our testing, based on the risks. We’ll look at risk assessment in Chapter 5.
