- Recognize the content of the [IEEE 829] incident report. (K1)
- Write an incident report covering the observation of a failure during testing. (K3)
Let’s wind down this chapter on test management with an important subject: how we can document and manage the incidents that occur during test execution. We’ll look at what topics we should cover when reporting incidents and defects. At the end of this section, you’ll be ready to write a thorough incident report.
Keep your eyes open for the only Syllabus term in this section, incident logging.
5.6.1 What are incident reports for and how do I write good ones?
When running a test, you might observe actual results that vary from expected results. This is not a bad thing – one of the major goals of testing is to find problems. Different organizations have different names to describe such situations. Commonly, they’re called incidents, bugs, defects, problems or issues.
To be precise, we sometimes draw a distinction between incidents on the one hand and defects or bugs on the other. An incident is any situation where the system exhibits questionable behavior, but often we refer to an incident as a defect only when the root cause is some problem in the item we’re testing.
Other causes of incidents include misconfiguration or failure of the test environment, corrupted test data, bad tests, invalid expected results and tester mistakes. (However, in some cases the policy is to classify as a defect any incident that arises from a test design, the test environment or anything else which is under formal configuration management.) We talk about incidents to indicate the possibility that a questionable behavior is not necessarily a true defect. We log these incidents so that we have a record of what we observed and can follow up the incident and track what is done to correct it.
While it is most common to find incident logging or defect reporting processes and tools in use during formal, independent test phases, you can also log, report, track, and manage incidents found during development and
reviews. In fact, this is a good idea, because it gives useful information on the extent to which early – and cheaper – defect detection and removal activities are happening.
Of course, we also need some way of reporting, tracking, and managing incidents that occur in the field or after deployment of the system. While many of these incidents will be user error or some other behavior not related to a defect, some percentages of defects do escape from quality assurance and testing activities. The defect detection percentage, which compares field defects with test defects, is an important metric of the effectiveness of the test process.
Here is an example of a DDP formula that would apply for calculating DDP for the last level of testing prior to release to the field:
It is most common to find defects reported against the code or the system itself. However, we have also seen cases where defects are reported against requirements and design specifications, user and operator guide and tests.
Often, it aids the effectiveness and efficiency of reporting, tracking and managing defects when the defect-tracking tool provides an ability to vary some of the information captured depending on what the defect was reported against.
In some projects, a very large number of defects are found. Even on smaller projects where 100 or fewer defects are found, you can easily lose track of them unless you have a process for reporting, classifying, assigning and managing the defects from discovery to final resolution.
An incident report contains a description of the misbehavior that was observed and classification of that misbehavior. As with any written communication, it helps to have clear goals in mind when writing. One common goal for such reports is to provide programmers, managers and others with detailed information about the behavior observed and the defect. Another is to support the analysis of trends in aggregate defect data, either for understanding more about a particular set of problems or tests or for understanding and reporting the overall level of system quality. Finally, defect reports, when analyzed over a project and even across projects, give information that can lead to development and test process improvements.
When writing an incident, it helps to have the readers in mind, too. The programmers need the information in the report to find and fix the defects. Before that happens, though, managers should review and prioritize the defects so that scarce testing and developer resources are spent fixing and confirmation testing the most important defects. Since some defects may be deferred – perhaps to be fixed later or perhaps, ultimately, not to be fixed at all – we should include workarounds and other helpful information for help desk or technical support teams. Finally, testers often need to know what their colleagues are finding so that they can watch for similar behavior elsewhere and avoid trying to run tests that will be blocked.
A good incident report is a technical document. In addition to being clear for its goals and audience, any good report grows out of a careful approach to researching and writing the report. We have some rules of thumb that can help you write a better incident report.
First, use a careful, attentive approach to running your tests. You never know when you’re going to find a problem. If you’re pounding on the keyboard while gossiping with office mates or daydreaming about a movie you just saw, you might not notice strange behaviors. Even if you see the incident, how much do you really know about it? What can you write in your incident report?
Intermittent or sporadic symptoms are a fact of life for some defects and it’s always discouraging to have an incident report bounced back as ‘irreproducible’. So, it’s a good idea to try to reproduce symptoms when you see them, and we have found three times to be a good rule of thumb. If a defect has intermittent symptoms, we will still report it, but we would be sure to include as much information as possible, especially how many times we tried to reproduce it and how many times it did in fact occur.
You should also try to isolate the defect by making carefully chosen changes to the steps used to reproduce it. In isolating the defect, you help guide the programmer to the problematic part of the system. You also increase your own knowledge of how the system works – and how it fails.
Some test cases focus on boundary conditions, which may make it appear that a defect is not likely to happen frequently in practice. We have found that it’s a good idea to look for more generalized conditions that cause the failure to occur, rather than simply relying on the test case. This helps prevent the infamous incident report rejoinder, “No real user is ever going to do that”. It also cuts down on the number of duplicate reports that get filed.
As there is often a lot of testing going on with the system during a test period, there are lots of other test results available. Comparing an observed problem against other test results and known defects found is a good way to find and document additional information that the programmer is likely to find very useful. For example, you might check for similar symptoms observed with other defects, the same symptom observed with defects that were fixed in previous versions or similar (or different) results seen in tests that cover similar parts of the system.
Many readers of incident reports, managers especially, will need to understand the priority and severity of the defect. So, the impact of the problem on the users, customers and other stakeholders is important. Most defect-tracking systems have a title or summary field, and that field should mention the impact, too.
Choice of words definitely matters in incident reports. You should be clear and unambiguous. You should also be neutral, fact-focused and impartial, keeping in mind the testing-related interpersonal issues discussed in Chapter 1 and earlier in this chapter. Finally, keeping the report concise helps keep people’s attention and avoids the problem of losing them in the details.
As a last rule of thumb for incident reports, we recommend that you use a review process for all reports filed. It works if you have the lead tester review reports and we have also allowed testers – at least experienced ones – to review other testers’ reports. Reviews are proven quality assurance techniques and incident reports are important project deliverables.
