8.1.1 Purpose
The purpose of Measure Solution Performance is to define performance measures and use the data collected to evaluate the effectiveness of a solution in relation to the value it brings
8.1.2 Description
Performance measures determine the value of a newly deployed or existing solution. The measures used depend on the solution itself, the context, and how the organization defines value. When solutions do not have built-in performance measures, the business analyst works with stakeholders to determine and collect the measures that will best reflect the performance of a solution. Performance may be assessed through key performance indicators (KPIs) aligned with enterprise measures, goals and objectives for a project, process performance targets, or tests for a software application.
8.1.3 Inputs
- Business Objectives: the measurable results that the enterprise wants to achieve. Provides a benchmark against which solution performance can be assessed.
- Implemented Solution (external): a solution (or component of a solution) that exists in some form. It may be an operating solution, a prototype, or a pilot or beta solution.
8.1.4 Elements
.1 Define Solution Performance Measures
When measuring solution performance, business analysts determine if current measures exist, or if methods for capturing them are in place. Business analysts ensure that any existing performance measures are accurate, relevant and elicit any additional performance measures identified by stakeholders.
Business goals, objectives, and business processes are common sources of measures. Performance measures may be influenced or imposed by third parties such as solution vendors, government bodies, or other regulatory organizations.
The type and nature of the measurements are considered when choosing the elicitation method. Solution performance measures may be quantitative, qualitative, or both, depending on the value being measured.
- Quantitative Measures: are numerical, countable, or finite, usually involving amounts, quantities, or rates.
- Qualitative Measures: are subjective and can include attitudes, perceptions, and any other subjective response. Customers, users, and others involved in the operation of a solution have perceptions of how well the solution is meeting the need.
.2 Validate Performance Measures
Validating performance measures helps to ensure that the assessment of solution performance is useful. Business analysts validate the performance measures and any influencing criteria with stakeholders. Specific performance measures should align with any higher-level measures that exist within the context affecting the
solution. Decisions about which measures are used to evaluate solution performance often reside with the sponsor, but may be made by any stakeholder with decision-making authority.
.3 Collect Performance Measures
When defining performance measures, business analysts may employ basic statistical sampling concepts.
When collecting performance measures, business analysts consider:
- Volume or Sample Size: a volume or sample size appropriate for the initiative is selected. A sample size that is too small might skew the results and lead to inaccurate conclusions. Larger sample sizes may be more desirable, but may not be practical to obtain.
- Frequency and Timing: the frequency and timing with which measurements are taken may have an effect on the outcome.
- Currency: measurements taken more recently tend to be more representative than older data.
Using qualitative measures, business analysts can facilitate discussions to estimate the value realized by a solution. Stakeholders knowledgeable about the operation and use of the solution reach a consensus based on facts and reasonable assumptions, as perceived by them.
8.1.5 Guidelines and Tools
- Change Strategy: the change strategy used or in use to implement the potential value.
- Future State Description: boundaries of the proposed new, removed, or modified components of the enterprise, and the potential value expected from the future state.
- Requirements (validated): a set of requirements that have been analyzed and appraised to determine their value.
- Solution Scope: the solution boundaries to measure and evaluate.
8.1.6 Techniques
- Acceptance and Evaluation Criteria: used to define acceptable solution performance.
- Benchmarking and Market Analysis: used to define measures and their acceptable levels.
- Business Cases: used to define business objectives and performance measures for a proposed solution.
- Data Mining: used to collect and analyze large amounts of data regarding solution performance.
- Decision Analysis: used to assist stakeholders in deciding on suitable ways to measure solution performance and acceptable levels of performance.
- Focus Groups: used to provide subjective assessments, insights, and impressions of a solution’s performance.
- Metrics and Key Performance Indicators (KPIs): used to measure solution performance.
- Non-Functional Requirements Analysis: used to define expected characteristics of a solution.
- Observation: used either to provide feedback on perceptions of solution performance or to reconcile contradictory results.
- Prototyping: used to simulate a new solution so that performance measures can be determined and collected.
- Survey or Questionnaire: used to gather opinions and attitudes about solution performance. Surveys and questionnaires can be effective when large or disparate groups need to be polled.
- Use Cases and Scenarios: used to define the expected outcomes of a solution.
- Vendor Assessment: used to assess which of the vendor’s performance measures should be included in the solution’s performance assessment.
8.1.7 Stakeholders
- Customer: may be consulted to provide feedback on solution performance.
- Domain Subject Matter Expert: a person familiar with the domain who can be consulted to provide potential measurements.
- End User: contributes to the actual value realized by the solution in terms of solution performance. They may be consulted to provide reviews and feedback on areas such as workload and job satisfaction.
- Project Manager: responsible for managing the schedule and tasks to perform the solution measurement. For solutions already in operation, this role may not be required.
- Sponsor: responsible for approving the measures used to determine solution performance. May also provide performance expectations.
- Regulator: an external or internal group that may dictate or prescribe constraints and guidelines that must be incorporated into solution performance measures.
8.1.8 Outputs
- Solution Performance Measures: measures that provide information on how well the solution is performing or potentially could perform.
