Chapter 1 – 1.2 – What is testing? – Part 1/3

Syllabus learning objectives for 1.2 What is testing?

1. Recall the common objectives of testing. (K1)
2. Describe the purpose of testing in software development, maintenance and operations as a means to find defects, provide confidence and information, and prevent defects. (K2)

In this section, we will review the common objectives of testing. We’ll explain how testing helps us to find defects, provide confidence and information, and prevent defects. We will also introduce additional fundamental principles of testing.

As you read this section, you’ll encounter the terms code, debugging, development of software, requirement, review, test basis, test case, testing and test objective.

1.2.1 The driving test – an analogy for software testing

We have spent some time describing why we need to test, but we have not discussed what testing is. What do we mean by the word testing? We use the words test and testing in everyday life and earlier we said testing could be described as “checking the software is OK”. That is not a detailed enough definition to help us understand software testing. Let’s use an analogy to help us: driving tests. In a driving test, the examiner critically assesses the candidate’s driving, noting every mistake, large or small, made by the driver under test.

The examiner takes the driver through a route which tests many possible driving activities, such as road junctions of different types, control and maneuvering of the car, ability to stop safely in an emergency, and awareness of the road, other road users and hazards. Some of the activities must be tested. For example, in the UK, an emergency stop test is always carried out, with the examiner simulating the moment of emergency by hitting the dashboard at which point the driver must stop the car quickly, safely and without loss of control. At the end of the test, the examiner makes a judgment about the driver’s performance. Has the driver
passed the test or failed? The examiner bases the judgment on the number and severity of the failures identified, and also whether the driver has been able to meet the driving requirements. A single severe fault is enough to fail the whole test, but a small number of minor faults might still mean the test is passed.

Many minor faults would reduce the confidence of the examiner in the quality – of the driving to the point where the driver cannot pass. The format of the driving test and the conduct of the examiner are worth considering:

• The test is planned and prepared for. In advance of the test, the examiner has planned a series of routes which cover the key driving activities to allow a thorough assessment of the driver’s performance. The drivers under test do not know what route they will be asked to take in advance, although they know the requirements of the test.
• The test has known goals – assessing whether the driver is sufficiently safe to be allowed to drive by themselves without an instructor, without endangering themselves or others. There are clear pass/fail criteria, based on the number and severity of faults, but the confidence of the examiner in the driving is also taken into account.
• The test is therefore carried out to show that the driver satisfies the requirements for driving and to demonstrate that they are fit to drive. The examiner looks for faults in the driving. The time for the test is limited, so it is not a complete test of the driver’s abilities, but it is representative and allows the
  examiner to make a risk-based decision about the driver. All the drivers are tested in an equivalent way and the examiner is neutral and objective. The examiner will log factual observations which enable a risk assessment to be made about the driving. Based on this, a driver who passes will be given a form enabling him to apply for a full driving license. A driver who fails will get a report with a list of faults and areas to improve before retaking the test.
• As well as observing the driver actually driving, the examiner will ask questions or the driver will take a written exam to check their understanding of the rules of the road, road signs, and what to do in various traffic situations.

1.2.2 Defining software testing.

With that analogy in mind, let’s look at the ISTQB definition of software testing. Let’s break the definition down into parts; the definition has some key phrases to remember. The definition starts with a description of testing as a process and then lists some objectives of the test process. First, let’s look at testing as a process:

• Process – Testing is a process rather than a single activity – there are a series of activities involved.
• All life cycle activities – Chapter 2 looks at testing as a process that takes place throughout the software development life cycle. We saw earlier that the later in the life cycle we find bugs, the more expensive they are to fix. If we can find and fix requirements defects at the requirements stage, that must make commercial sense. We’ll build the right software, correctly and at a lower cost overall. So, the thought process of designing tests early in the life cycle can help to prevent defects from being introduced into code. We sometimes refer to this as “verifying the test basis via the test design”. The test basis includes documents such as the requirements and design specifications. You’ll see how to do this in Chapter 4.
• Both static and dynamic – We’ll see in Chapter 3 that as well as tests where the software code is executed to demonstrate the results of running tests (often called dynamic testing) we can also test and find defects without executing code. This is called static testing. This testing includes reviewing of
  documents (including source code) and static analysis. This is a useful and cost effective way of testing.
• Planning – Activities take place before and after test execution. We need to manage the testing; for example, we plan what we want to do; we control the test activities; we report on testing progress and the status of the software under test; and we finalize or close testing when a phase completes. Chapter
  5 covers these test management activities.
• Preparation – We need to choose what testing we’ll do, by selecting test conditions and designing test cases. Chapter 4 covers the test design activities.
• Evaluation – As well as executing the tests, we must check the results and evaluate the software under test and the completion criteria, which help us decide whether we have finished testing and whether the software product has passed the tests.
• Software products and related work products – We don’t just test code. We test the requirements and design specifications, and we test related documents such as operation, user and training material. Static and dynamic testing are both needed to cover the range of products we need to test.

The second part of the definition covers the some of the objectives for testing – the reasons why we do it:

• Determine that (software products) satisfy specified requirements – Some of the testing we do is focused on checking products against the specification for the product; for example we review the design to see if it meets requirements, and then we might execute the code to check that it meets the design. If the product meets its specification, we can provide that information to help stakeholders judge the quality of the product and decide whether it is ready for use.
• Demonstrate that (software products) are fit for purpose – This is slightly different to the point above; after all the specified requirements might be wrong or incomplete. “Fit for purpose” looks at whether the software does enough to help the users to carry out their tasks; we look at whether the software does what the user might reasonably expect. For example, we might look at who might purchase or use the software, and check that it does do what they expect; this might lead us to add a review of the marketing material to our static tests, to check that expectations of the software are properly set. One way of judging the quality of a product is by how fit it is for its purpose.
• Detect defects – We most often think of software testing as a means of detecting faults or defects that in operational use will cause failures. Finding the defects helps us understand the risks associated with putting the software into operational use, and fixing the defects improves the quality of the products. However, identifying defects has another benefit. With root cause analysis, they also help us improve the development processes and make fewer mistakes in future work.

This is a suitable definition of testing for any level of testing, from component testing through to acceptance testing, provided that we remember to take the varying objectives of these different levels of testing into account. (In Chapter 2 we’ll cover the different test levels, their objectives, and how they fit into the software development life cycle.)

We can clearly see now why the common perception of testing (that it only consists of running tests, i.e. executing the software) is not complete. This is one of the testing activities, but not all of the testing process.