- Compare four software test types (functional, non-functional, structural and change-related) by example. (K2)
- Recognize that functional and structural tests occur at any test level. (K1)
- Identify and describe non-functional test types based on nonfunctional requirements. (K2)
- Identify and describe test types based on the analysis of a software system’s structure or architecture. (K2)
- Describe the purpose of confirmation testing and regression testing. (K2)
Test types are introduced as a means of clearly defining the objective of a certain test level for a programme or project. We need to think about different types of testing because testing the functionality of the component or system may not be sufficient at each level to meet the overall test objectives.
Focusing the testing on a specific test objective and, therefore, selecting the appropriate type of test helps making and communicating decisions against test objectives easier.
A test type is focused on a particular test objective, which could be the testing of a function to be performed by the component or system; a nonfunctional quality characteristic, such as reliability or usability; the structure or architecture of the component or system; or related to changes, i.e. confirming that defects have been fixed (confirmation testing, or re-testing) and looking for unintended changes (regression testing). Depending on its objectives, testing will be organized differently. For example, component testing aimed at performance would be quite different to component testing aimed at achieving decision coverage.
2.3.1 Testing of function (functional testing)
The function of a system (or component) is “what it does”. This is typically described in a requirements specification, a functional specification, or in use cases. There may be some functions that are ‘assumed’ to be provided that are not documented that are also part of the requirement for a system, though it is difficult to test against undocumented and implicit requirements.
Functional tests are based on these functions, described in documents or understood by the testers and may be performed at all test levels (e.g. test for components may be based on a component specification).
Functional testing considers the specified behavior and is often also referred to as black-box testing. This is not entirely true, since black-box testing also includes non-functional testing (see Section 2.3.2).
Function (or functionality) testing can, based upon ISO 9126, be done focusing on suitability, interoperability, security, accuracy and compliance. Security testing, for example, investigates the functions (e.g., a firewall) relating to detection of threats, such as viruses, from malicious outsiders.
Testing functionality can be done from two perspectives: requirements-based or business-process-based.
Requirements-based testing uses a specification of the functional requirements for the system as the basis for designing tests. A good way to start is to use the table of contents of the requirements specification as an initial test inventory or list of items to test (or not to test). We should also prioritize the requirements based on risk criteria (if this is not already done in the specification) and use this to prioritize the tests. This will ensure that the most important and most critical tests are included in the testing effort.
Business-process-based testing uses knowledge of the business processes. Business processes describe the scenarios involved in the day-to-day business use of the system. For example, a personnel and payroll system may have a business process along the lines of: someone joins the company, he or she is paid on a regular basis, and he or she finally leaves the company. Use cases originate from object-oriented development but are nowadays popular in many development life cycles. They also take the business processes as a starting point, although they start from tasks to be performed by users. Use cases are a very useful basis for test cases from a business perspective.
The techniques used for functional testing are often specification-based, but experienced-based techniques can also be used (see Chapter 4 for more on test techniques). Test conditions and test cases are derived from the functionality of the component or system. As part of test designing, a model may be developed, such as a process model, state transition model or a plain-language specification.
2.3.2 Testing of software product characteristics (non-functional testing)
A second target for testing is the testing of the quality characteristics, or nonfunctional attributes of the system (or component or integration group). Here we are interested in how well or how fast something is done. We are testing something that we need to measure on a scale of measurement, for example time to respond.
Non-functional testing, as functional testing, is performed at all test levels. Non-functional testing includes, but is not limited to, performance testing, load testing, stress testing, usability testing, maintainability testing, reliability testing and portability testing. It is the testing of ‘how well’ the system works.
Many have tried to capture software quality in a collection of characteristics and related sub-characteristics. In these models some elementary characteristics keep on reappearing, although their place in the hierarchy can differ. The International Organization for Standardization (ISO) has defined a set of quality characteristics [ISO/IEC 9126, 2001]. This set reflects a major step towards consensus in the IT industry and thereby addresses the general notion of software quality. The ISO 9126 standard defines six quality characteristics and the subdivision of each quality characteristic into a number of sub-characteristics. This standard is getting more and more recognition in the industry, enabling development, testing and their stakeholders to use a common terminology for quality characteristics and thereby for non-functional testing.
The characteristics and their sub-characteristics are, respectively:
- functionality, which consists of five sub-characteristics: suitability, accuracy, security, interoperability and compliance; this characteristic deals with functional testing as described in Section 2.3.1;
- reliability, which is defined further into the sub-characteristics maturity (robustness), fault-tolerance, recoverability and compliance;
- usability, which is divided into the sub-characteristics understandability, learnability, operability, attractiveness and compliance;
- efficiency, which is divided into time behavior (performance), resource utilization and compliance;
- maintainability, which consists of five sub-characteristics: analyzability, changeability, stability, testability and compliance;
- portability, which also consists of five sub-characteristics: adaptability, installability, co-existence, replaceability and compliance.
