4.1.4 Test design: specifying test cases
Test conditions can be rather vague, covering quite a large range of possibilities as we saw with our mobile phone company example (e.g., a teenager in the midwest), or a test condition may be more specific (e.g. a particular male customer on pay-as-you-go with less than $10 credit). However, when we come to make a test case, we are required to be very specific; in fact we now need exact and detailed specific inputs, not general descriptions (e.g. Jim Green, age 17, living in Grand Rapids, Michigan, with credit of $8.64, expected result: add to Q4 marketing campaign). Note that one test case covers a number of conditions (teenager, male, mid-west area, pay-as-you-go, and credit of less than $10).
For a test condition of ‘an existing customer’, the test case input needs to be “Jim Green” where Jim Green already exists on the customer database, or part of this test would be to set up a database record for Jim Green.
A test case needs to have input values, of course, but just having some values to input to the system is not a test! If you don’t know what the system is supposed to do with the inputs, you can’t tell whether your test has passed or failed.
Should these detailed test cases be written down? They can be formally documented, as we will describe below. However, it is possible to test without documenting at the test-case level. If you give an experienced user acceptance tester with a strong business background a list of high-level test conditions, they could probably do a good job of testing. But if you gave the same list to a new starter who didn’t know the system at all, they would probably be lost, so they would benefit from having more detailed test cases.
Test cases can be documented as described in the IEEE 829 Standard for Test Documentation. Note that the contents described in the standard don’t all have to be separate physical documents. But the standard’s list of what needs to be kept track of is a good starting point, even if the test conditions and test cases for a given functionality or feature are all kept in one physical document.
One of the most important aspects of a test is that it assesses that the system does what it is supposed to do. Copeland says “At its core, testing is the process of comparing “what is” with “what ought to be” “. [Copeland, 2003] If we simply put in some inputs and think “that was fun, I guess the system is probably OK because it didn’t crash”, then are we actually testing it? We don’t think so. You have observed that the system does what the system does – this is not a test.
Boris Beizer refers to this as “kiddie testing” [Beizer, 1990]. We may not know what the right answer is in detail every time, and we can still get some benefit from this approach at times, but it isn’t really testing.
In order to know what the system should do, we need to have a source of information about the correct behavior of the system – this is called an ‘oracle’ or a test oracle.
This has nothing to do with databases or companies that make them. It comes from the ancient Greek Oracle at Delphi, who supposedly could predict the future with unerring accuracy. Actually her answers were so vague that people interpreted them in whatever way they wanted – perhaps a bit like requirements specifications!
Once a given input value has been chosen, the tester needs to determine what the expected result of entering that input would be and document it as part of the test case. Expected results include information displayed on a screen in response to an input, but they also include changes to data and/or states, and any other consequences of the test (e.g., a letter to be printed overnight).
What if we don’t decide on the expected results before we run a test? We can still look at what the system produces and would probably notice if something was wildly wrong. However, we would probably not notice small differences in calculations, or results that seemed to look OK (i.e. are plausible). So we would conclude that the test had passed, when in fact the software has not given the correct result. Small differences in one calculation can add up to something very major later on, for example if results are multiplied by a large factor.
Ideally expected results should be predicted before the test is run – then your assessment of whether or not the software did the right thing will be more objective. For a few applications it may not be possible to predict or know exactly what an expected result should be; we can only do a “reasonableness check”. In this case we have a ‘partial oracle’ – we know when something is very wrong, but would probably have to accept something that looked reasonable. An example is when a system has been written to calculate something where it may not be possible to manually produce expected results in a reasonable timescale because the calculations are so complex.
In addition to the expected results, the test case also specifies the environment and other things that must be in place before the test can be run (the preconditions) and any things that should apply after the test completes (the postconditions).
The test case should also say why it exists – i.e. the objective of the test it is part of or the test conditions that it is exercising (traceability). Test cases can now be prioritized so that the most important test cases are executed first, and low priority test cases are executed later, or even not executed at all. This may reflect the priorities already established for test conditions or the priority may be determined by other factors related to the specific test cases, such as a specific input value that has proved troublesome in the past, the risk associated with the test, or the most sensible sequence of running the tests. Chapter 5 gives more detail of risk-based testing.
Test cases need to be detailed so that we can accurately check the results and know that we have exactly the right response from the system. If tests are to be automated, the testing tool needs to know exactly what to compare the system output to.
4.1.5 Test implementation: specifying test procedures or scripts
The next step is to group the test cases in a sensible way for executing them and to specify the sequential steps that need to be done to run the test. For example, a set of simple tests that cover the breadth of the system may form a regression suite, or all of the tests that explore the working of a given functionality or feature in depth may be grouped to be run together.
Some test cases may need to be run in a particular sequence. For example, a test may create a new customer record, amend that newly created record and then delete it. These tests need to be run in the correct order, or they won’t test what they are meant to test.
The document that describes the steps to be taken in running a set of tests (and specifies the executable order of the tests) is called a test procedure in IEEE 829 and is often also referred to as a test script. It could be called a manual test script for tests that are intended to be run manually rather than using a test execution tool. Test script is also used to describe the instructions to a test execution tool. An automation script is written in a programming language that the tool can interpret. (This is an automated test procedure.) See Chapter 6 for more information on this and other types of testing tools.
The test procedures, or test scripts, are then formed into a test execution schedule that specifies which procedures are to be run first – a kind of superscript. The test schedule would say when a given script should be run and by whom. The schedule could vary depending on newly perceived risks affecting the priority of a script that addresses that risk, for example. The logical and technical dependencies between the scripts would also be taken into account when scheduling the scripts. For example, a regression script may always be the first to be run when a new release of the software arrives, as a smoke test or sanity check.
Returning to our example of the mobile phone company’s marketing campaign, we may have some tests to set up customers of different types on the database. It may be sensible to run all of the setup for a group of tests first. So our first test procedure would entail setting up a number of customers, including Jim Green, on the database
We may then have another test procedure to do with the marketing campaign:
Writing the test procedure is another opportunity to prioritize the tests, to ensure that the best testing is done in the time available. A good rule of thumb is “Find the scary stuff first”. However, the definition of what is “scary” depends on the business, system or project. For example, is it worse to raise Bob Founders’ credit limit when he is not a good credit risk (he may not pay for the credit he asked for) or to refuse to raise his credit limit when he is a good credit risk (he may go elsewhere for his phone service, and we lose the opportunity of lots of income from him).
