6.1.6 Tool support for performance and monitoring
The tools described in this section support testing that can be carried out on a system when it is operational, i.e. while it is running. This can be during testing or could be after a system is released into live operation.
Dynamic analysis tools (D)
Dynamic analysis tools are “dynamic” because they require the code to be running. They are “analysis” rather than ‘testing’ tools because they analyze what is happening ‘behind the scenes’ while the software is running (whether being executed with test cases or being used in operation).
An analogy with a car may be useful here. If you go to look at a car to buy, you might sit in it to see if is comfortable and see what sound the doors make – this would be static analysis because the car is not being driven. If you take a test drive, then you would check that the car performs as you expect (e.g., turns right when you turn the steering wheel clockwise) – this would be a test. While the car is running, if you were to check the oil pressure or the brake fluid, this would be dynamic analysis – it can only be done while the engine is running, but it isn’t a test case.
When your PC’s response time gets slower and slower over time, but is much improved after you re-boot it, this may well be due to a “memory leak”, where the programs do not correctly release blocks of memory back to the operating system. Eventually the system will run out of memory completely and stop. Rebooting restores all of the memory that was lost, so the performance of the system is now restored to its normal state.
Features or characteristics of dynamic analysis tools include support for:
- detecting memory leaks;
- identifying pointer arithmetic errors such as null pointers;
- identifying time dependencies.
These tools would typically be used by developers in component testing and component integration testing, e.g., when testing middleware, when testing security or when looking for robustness defects.
Another form of dynamic analysis for websites is to check whether each link does actually link to something else (this type of tool may be called a “web spider”). The tool doesn’t know if you have linked to the correct page, but at least it can find dead links, which may be helpful.
Performance-testing, load-testing and stress-testing tools Performance testing tools are concerned with testing at system level to see whether or not the system will stand up to a high volume of usage. A “load” test checks that the system can cope with its expected number of transactions. A “volume” test checks that the system can cope with a large amount of data, e.g., many fields in a record, many records in a file, etc. A “stress” test is one that goes beyond the normal expected usage of the system (to see what would happen outside its design expectations), with respect to load or volume.
In performance testing, many test inputs may be sent to the software or system where the individual results may not be checked in detail. The purpose of the test is to measure characteristics, such as response times, throughput or the mean time between failures (for reliability testing).
In order to assess performance, the tool needs to generate some kind of activity on the system, and this can be done in different ways. At a very simple level the same transaction could be repeated many times, but this is not realistic. There are many levels of realism that could be set, depending on the tool, such as different user profiles, different types of activity, timing delays and other parameters. Adequately replicating the end-user environments or user profiles is usually key to realistic results.
Analyzing the output of a performance-testing tool is not always straightforward and it requires time and expertise. If the performance is not up to the standard expected, then some analysis needs to be performed to see where the problem is and to know what can be done to improve the performance.
Features or characteristics of performance-testing tools include support for:
- generating a load on the system to be tested;
- measuring the timing of specific transactions as the load on the system varies;
- measuring average response times;
- producing graphs or charts of responses over time.
Monitoring tools
Monitoring tools are used to continuously keep track of the status of the system in use, in order to have the earliest warning of problems and to improve service.
There are monitoring tools for servers, networks, databases, security, performance, website and internet usage, and applications.
Features or characteristics of monitoring tools include support for:
- identifying problems and sending an alert message to the administrator (e.g., network administrator);
- logging real-time and historical information;
- finding optimal settings;
- monitoring the number of users on a network;
- monitoring network traffic (either in real time or covering a given length of time of operation with the analysis performed afterwards).
6.1.7 Tool support for specific application areas (K1)
In this chapter, we have described tools according to their general functional classifications. There are also further specializations of tools within these classifications. For example, there are web-based performance-testing tools as well as performance-testing tools for back-office systems. There are static analysis tools for specific development platforms and programming languages, since each programming language and every platform has distinct characteristics.
There are dynamic analysis tools that focus on security issues, as well as dynamic analysis tools for embedded systems.
Commercial tool sets may be bundled for specific application areas such as web-based or embedded systems.
6.1.8 Tool support using other tools
The tools described in this chapter are not the only tools that a tester can make use of. You may not normally think of a word processor or a spreadsheet as a testing tool, but they are often used to store test designs, test scripts or test data.
Testers may also use SQL to set up and query databases containing test data. Tools used by developers when debugging, to help localize defects and check their fixes, are also testing tools.
Developers use debugging tools when identifying and fixing defects. The debugging tools enable them to run individual and localized tests to ensure that they have correctly identified the cause of a defect and to confirm that their change to the code will indeed fix the defect.
It is a good idea to look at any type of tool available to you for ways it could be used to help support any of the testing activities. For example, testers can use Perl scripts to help compare test results.
