4.4.2 Statement coverage and statement testing
Statement coverage is calculated by:
Studies and experience in the industry have indicated that what is considered reasonably thorough black-box testing may actually achieve only 60% to 75% statement coverage. Typical ad hoc testing is likely to be around 30% – these leaves 70% of the statements untested.
Different coverage tools may work in slightly different ways, so they may give different coverage figures for the same set of tests on the same code, although at 100% coverage they should be the same.
We will illustrate the principles of coverage on code. In order to simplify our examples, we will use a basic pseudo-code – this is not any specific programming language, but should be readable and understandable to you, even if you have not done any programming yourself.
For example, consider code sample 4.1.
To achieve 100% statement coverage of this code segment just one test case is required, one which ensures that variable A contains a value that is greater than the value of variable B, for example, A = 12 and B = 10. Note that here we are doing structural test design first, since we are choosing our input values in order ensure statement coverage.
Let’s look at an example where we measure coverage first. In order to simplify the example, we will regard each line as a statement. (Different tools and methods may count different things as statements, but the basic principle is the same however they are counted.) A statement may be on a single line, or it may be spread over several lines. One line may contain more than one statement, just one statement, or only part of a statement. Some statements can contain other statements inside them. In code sample 4.2, we have two read statements, one assignment statement, and then one IF statement on three lines, but the IF statement contains another statement (print) as part of it.
Although it isn’t completely correct, we have numbered each line and will regard each line as a statement. (Some tools may group statements that would always be executed together in a basic block which is regarded as a single statement.) However, we will just use numbered lines to illustrate the principles of coverage of statements (lines). Let’s analyze the coverage of a set of tests on our six-statement program:
TEST SET 1
Test 1_1: A = 2, B = 3
Test 1_2: A = 0, B = 25
Test 1_3: A = 47, B = 1
Which statements have we covered?
- In Test 1_1, the value of C will be 8, so we will cover the statements on lines 1 to 4 and line 6.
- In Test 1_2, the value of C will be 50, so we will cover exactly the same state ments as Test 1_1.
- In Test 1_3, the value of C will be 49, so again we will cover the same statements.
Since we have covered five out of six statements, we have 83% statement coverage (with three tests). What test would we need in order to cover statement 5, the one statement that we haven’t exercised yet? How about this one: Test 1_4: A = 20, B = 25
This time the value of C is 70, so we will print ‘Large C and we will have exercised all six of the statements, so now statement coverage = 100%. Notice that we measured coverage first, and then designed a test to cover the statement that we had not yet covered.
Note that Test 1_4 on its own is more effective- (towards our goal of achieving 100% statement coverage) than the first three tests together. Just taking
Test 1_4 on its own is also more efficient than the set of four tests, since it has
used only one test instead of four. Being more effective and more efficient is
the mark of a good test technique.
4.4.3 Decision coverage and decision testing
A decision is an IF statement, a loop control statement (e.g. DO-WHILE or REPEAT-UNTIL), or a CASE statement, where there are two or more possible exits or outcomes from the statement. With an IF statement, the exit can either be TRUE or FALSE, depending on the value of the logical condition that comes after IF. With a loop control statement, the outcome is either to perform the code within the loop or not – again a True or False exit. Decision coverage is calculated by:
What feels like reasonably thorough functional testing may achieve only 40% to 60% decision coverage. Typical ad hoc testing may cover only 20% of the decisions, leaving 80% of the possible outcomes untested. Even if your testing seems reasonably thorough from a functional or specification-based perspective, you may have only covered two-thirds or three-quarters of the decisions. Decision coverage is stronger than statement coverage. It ‘subsumes’ statement coverage – this means that 100% decision coverage always guarantees 100% statement coverage. Any stronger coverage measure may require more test cases to achieve 100% coverage. For example, consider code sample 4.1 again.
We saw earlier that just one test case was required to achieve 100% statement coverage. However, decision coverage requires each decision to have had both a True and False outcome. Therefore, to achieve 100% decision coverage, a second test case is necessary where A is less than or equal to B. This will ensure that the decision statement ‘IF A > B’ has a False outcome. So, one test is sufficient for 100% statement coverage, but two tests are needed for 100% decision coverage. Note that 100% decision coverage guarantees 100% statement coverage, but not the other way around!
Let’s suppose that we already have the following test, which gives us 100% statement coverage for code sample 4.3.
TEST SET 2
Test 2_1: A = 20, B = 15
Which decision outcomes have we exercised with our test? The value of C is -10, so the condition “C < 0” is True, so we will print “C negative” and we have exercised the True outcome from that decision statement. But we have not exercised the decision outcome of False. What other test would we need to exercise the False outcome and to achieve 100% decision coverage?
Before we answer that question, let’s have a look at another way to represent this code. Sometimes the decision structure is easier to see in a control flow diagram (see Figure 4.4).
The dotted line shows where Test 2_1 has gone and clearly shows that we haven’t yet had a test that takes the False exit from the IF statement.
Let’s modify our existing test set by adding another test:
TEST SET 2
Test 2_1: A = 20, B = 15
Test 2_2: A = 10, B = 2
This now covers both of the decision outcomes, True (with Test 2_1) and False (with Test 2_2). If we were to draw the path taken by Test 2_2, it would be a straight line from the read statement down the False exit and through the ENDIF. Note that we could have chosen other numbers to achieve either the True or False outcomes.
4.4.4 Other structure-based techniques
There are other structure-based techniques that can be used to achieve testing to different degrees of thoroughness. Some techniques are stronger (require more tests to achieve 100% coverage and therefore, have a greater chance of detecting defects) and others are weaker.
For example, branch coverage is closely related to decision coverage and at 100% coverage they give exactly the same results. Decision coverage measures the coverage of conditional branches; branch coverage measures the coverage of both conditional and unconditional branches. The Syllabus uses decision coverage, as it is the source of the branches. Some coverage measurement tools may talk about branch coverage when they actually mean decision coverage.
Other control-flow code-coverage measures include linear code sequence and jump (LCSAJ) coverage, condition coverage, multiple condition coverage (also known as condition combination coverage) and condition determination coverage (also known as multiple condition decision coverage or modified condition decision coverage, MCDC). This technique requires the coverage of all conditions that can affect or determine the decision outcome.
Another popular, but often misunderstood, code-coverage measure is path coverage. Sometimes any structure-based technique is called “path testing” [Patton, 2001]. However, strictly speaking, for any code that contains a loop, path coverage is impossible since a path that travels round the loop three times is different from the path that travels round the same loop four times. This is true even if the rest of the paths are identical. So, if it is possible to travel round the loop an unlimited number of times then there are an unlimited number of paths through that piece of code. For this reason, it is more correct to talk about “independent path segment coverage” though the shorter term “path coverage” is frequently used.
Structure-based measures and related test design techniques are described in [BS7925-2]. Structure-based techniques are also discussed in [Copeland. 2003] and [Myers, 1979]. A good description of the graph theory behind structural testing can be found in [Jorgensen, 1995] and [Hetzel, 1988] also shows a structural approach. [Pol et al, 2001] describes a structure-based approach called an algorithm test.
