Designing test cases
Having identified the conditions that you wish to test, in this case by using equivalence partitioning and boundary value analysis, the next step is to design the test cases. The more test conditions that can be covered in a single test case, the fewer test cases will be needed in order to cover all the conditions. This is usually the best approach to take for positive tests and for tests that you are reasonably confident will pass.
However, if a test fails, then we need to find out why it failed – which test condition was handled incorrectly? We need to get a good balance between covering too many and too few test conditions in our tests.
Let’s look at how one test case can cover one or more test conditions. Using the bank balance example, our first test could be of a new customer with a balance of $500. This would cover a balance in the partition from $100.01 to $999.99 and an output partition of a 5% interest rate. We would also be covering other partitions that we haven’t discussed yet. for example, a valid customer, a new customer, a customer with only one account, etc. All of the partitions covered in this test are valid partitions. When we come to test invalid partitions, the safest option is probably to try to cover only one invalid test condition per test case. This is because programs may stop processing input as soon as they encounter the first problem. So, if you have an invalid customer name, invalid address, and invalid balance, you may get an error message saying, “invalid input” and you don’t know whether the test has detected only one invalid input or all of them. (This is also why specific error messages are much better than general ones!)
However, if it is known that the software under test is required to process all input regardless of its validity, then it is sensible to continue as before and design test cases that cover as many invalid conditions in one go as possible. For example, if every invalid field in a form has some red text above or below the field saying that this field is invalid and why, then you know that each field has been checked, so you have tested all of the error processing in one test case. In either case, there should be separate test cases covering valid and invalid conditions.
To cover the boundary test cases, it may be possible to combine all of the minimum valid boundaries for a group of fields into one test case and also the maximum boundary values. The invalid boundaries could be tested together if the validation is done on every field; otherwise, they should be tested separately, as with the invalid partitions.
Why do both equivalence partitioning and boundary value analysis?
Technically, because every boundary is in some partition, if you did only boundary value analysis you would also have tested every equivalence partition. However, this approach may cause problems if that value fails – was it only the boundary value that failed or did the whole partition fail? Also, by testing only boundaries we would probably not give the users much confidence as we are using extreme values rather than normal values. The boundaries may be more difficult (and therefore more costly) to set up as well.
For example, in the printer copies example described earlier we identified the following boundary values:
Suppose we test only the valid boundary values 1 and 99 and nothing in between. If both tests pass, this seems to indicate that all the values in between should also work. However, suppose that one-page prints correctly, but 99 pages do not. Now we don’t know whether any set of more than one page works, so the first thing we would do would be to test for say 10 pages, i.e., a value from the equivalence partition.
We recommend that you test the partitions separately from boundaries – this means choosing partition values that are NOT boundary values.
However, if you use the three-value boundary value approach, then you would have valid boundary values of 1, 2, 98 and 99, so having a separate equivalence value in addition to the extra two boundary values would not give much additional benefit. But notice that one equivalence value, e.g., 10, replaces both of the extra two boundary values (2 and 98). This is why equivalence partitioning with two-value boundary value analysis is more efficient than three-value boundary value analysis.
Which partitions and boundaries you decide to exercise (you don’t need to test them all), and which ones you decide to test first, depends on your test objectives. If your goal is the most thorough approach, then follow the procedure of testing valid partitions first, then invalid partitions, then valid boundaries and finally invalid boundaries. However, if you are under time pressure and cannot test everything (and who isn’t?), then your test objectives will help you decide what to test. If you are after user confidence of typical transactions with a minimum number of tests, you may do valid partitions only. If you want to find as many defects as possible as quickly as possible, you may start with boundary values, both valid and invalid. If you want confidence that the system will handle bad inputs correctly, you may do mainly invalid partitions and boundaries. Your previous experience of types of defects found can help you find similar defects; for example, if there are typically a number of boundary defects, then you would start by testing boundaries.
Equivalence partitioning and boundary value analysis are described in most testing books, including [Myers, 1979] and [Copeland, 2003]. Examples of types of equivalence classes to look out for are given in [Kaner et al., 1993] Equivalence partitioning and boundary value analysis are described in BS7925-2, including designing tests and measuring coverage.
