6.2 Effective Use of Tools
6.2.1 Main Principles for Tool Selection
The main considerations in selecting a tool for an organization include:
- Assessment of the maturity of the own organization, its strengths and weaknesses
- Identification of opportunities for an improved test process supported by tools
- Understanding of the technologies used by the test object(s), in order to select a tool that is compatible with that technology
- Understanding the build and continuous integration tools already in use within the organization, in order to ensure tool compatibility and integration
- Evaluation of the tool against clear requirements and objective criteria
- Consideration of whether or not the tool is available for a free trial period (and for how long)
- Evaluation of the vendor (including training, support and commercial aspects) or support for noncommercial (e.g., open source) tools
- Identification of internal requirements for coaching and mentoring in the use of the tool
- Evaluation of training needs, considering the testing (and test automation) skills of those who will be working directly with the tool(s)
- Consideration of pros and cons of various licensing models (e.g., commercial or open source)
- Estimation of a cost-benefit ratio based on a concrete business case (if required)
As a final step, a proof-of-concept evaluation should be done to establish whether the tool performs effectively with the software under test and within the current infrastructure or, if necessary, to identify changes needed to that infrastructure to use the tool effectively.
6.2.2 Pilot Projects for Introducing a Tool into an Organization
After completing the tool selection and a successful proof-of-concept, introducing the selected tool into an organization generally starts with a pilot project, which has the following objectives:
- Gaining in-depth knowledge about the tool, understanding both its strengths and weaknesses
- Evaluating how the tool fits with existing processes and practices, and determining what would need to change
- Deciding on standard ways of using, managing, storing, and maintaining the tool and the test work products (e.g., deciding on naming conventions for files and tests, selecting coding standards, creating libraries and defining the modularity of test suites)
- Assessing whether the benefits will be achieved at reasonable cost
- Understanding the metrics that you wish the tool to collect and report, and configuring the tool to ensure these metrics can be captured and reported
6.2.3 Success Factors for Tools
Success factors for evaluation, implementation, deployment, and on-going support of tools within an organization include:
- Rolling out the tool to the rest of the organization incrementally
- Adapting and improving processes to fit with the use of the tool
- Providing training, coaching, and mentoring for tool users
- Defining guidelines for the use of the tool (e.g., internal standards for automation)
- Implementing a way to gather usage information from the actual use of the tool
- Monitoring tool use and benefits
- Providing support to the users of a given tool
- Gathering lessons learned from all users
It is also important to ensure that the tool is technically and organizationally integrated into the software development lifecycle, which may involve separate organizations responsible for operations and/or third party suppliers. See Graham 2012 for experiences and advice about using test execution tools.
