Agile test metrics and KPIs are measurements that enable Agile teams to assess how effective their software tests are and whether their testing efforts help achieve specific business objectives.
An important aim of Agile software testing is to get testing up to speed with development—metrics and KPIs can assess the efficacy of the Agile testing team in this context.
In this article, you'll find out about the important difference between a metric and a KPI, and why you need to distinguish between the two. You'll also get some examples of KPIs and metrics relevant to Agile software testing efforts. Finally, you'll learn exactly how KPIs and metrics can lead to overall improvement in software testing within an Agile framework in addition to understanding some important measuring pitfalls.
What’s The Difference between a Metric and a KPI?
A metric is simply a measure of something. Test metrics are measurements that give insight on software testing efforts and efficacy. They can help diagnose specific problems with software testing processes that lead to poor quality software.
A KPI (Key Performance Indicator) is a particular type of metric that closely aligns with critical business objectives. In the context of software testing in an Agile project, KPIs measure whether the current testing processes result in faster testing efforts, higher quality software, more frequent releases, and, most importantly, a positive ROI.
Agile Testing KPIs
When an Agile team chooses its KPIs, it's vital each time to ask the same question—does this metric tell us anything about the business objectives of software testing? If the answer is yes, then the metric is a valid KPI.
The following examples of Agile metrics are valid KPIs.
Automation Progress
Since software testing in an Agile team must not become a bottleneck if the team is to release high-quality software regularly. Therefore, automation is an important part of speeding up testing, so that testing occurs at a similar pace to development.
Automation progress serves as an indicator that determines how fast the testing efforts are for each build. Generally, the more automation achieved out of all possible automatable tests, the faster a team can test each release candidate.
Defect Resolution Time
This KPI helps measure whether testing teams are helping to achieve the business aim of high-quality software released frequently. The longer it takes to fix errors for a given build, the longer it takes to release the software. Agile testing teams must aim to fix errors as quickly as possible.
Covered Requirements
An important marker of software quality is that the application does exactly what the user needs it to do. Testing teams must create test cases that verify the functioning of all software requirements if companies are to release high-quality software. This measurement simply tracks the percentage of requirements covered by at least one test, and it serves as an indicator of both software quality and testing efficiency.
Agile Testing Metrics
Metrics within any software testing framework should always serve an actionable purpose. It is important for teams to avoid vanity metrics that don't help diagnose testing issues or evaluate testing processes.
Escaped Defects
A marker of an effective software testing effort is minimal defects escaping into production. Fixing defects post-release is much costlier than fixing them earlier in the development cycle. By counting the defects per release that are found after the release date, you can get a clear picture of your Agile teams' testing efficacy. Teams can capture this over time, per release, or per sprint.
Velocity
Velocity is an Agile specific measurement useful within a testing context. Velocity measures how much work a team completes during a certain interval, which is typically a sprint in Agile terms.
Each team estimates its velocity by committing certain numbers of story points or hours to each sprint. By comparing actual velocity with the committed figures, you get an idea of how fast the team develops working software. Since testing is often the slowest part of the development cycle, higher velocity can mean faster progression with software testing.
Cumulative Flow Diagram
A cumulative flow diagram (CFD) provides a graphical depiction of summary information for a project. A CFD is a stacked chart displaying the Agile workflow over time. Since testing is part of the Agile workflow, you can use a CFD to measure the progression of software testing.
In particular, CFDs are useful for assessing whether testing is a bottleneck in the Agile workflow. Bottleneck processes in a CFD vertically widen over time as in the below image.
Will Metrics and KPIs Improve Your Testing Process?
These indicators serve little use if they don't lead to tangible benefits for the Agile team. Any metric has the potential to be misused—it's important to remember that the cross-functional Agile approach calls for new assessments that reflect this integrated unit.
Measurements that focus on the individual and promote excessive competition among software testers are bad for the Agile team as a whole. By their very definition, individual metrics are distinctly non-Agile.
Managers that continue to measure software testing at the individual level discourage improvement in software quality. For example, counting test cases says nothing about the quality of software—individual testers can easily fudge the metric by creating tons of test cases that don't test anything important.
Furthermore, some companies claim they have transitioned to using an Agile approach, but the metrics and KPIs they use have been simply ported over from the old waterfall model, without critically evaluating the benefit of them within an Agile context.
However, despite these pitfalls, when chosen suitably and used with the Agile methodology in mind, metrics and KPIs improve both the speed of software delivery and software quality. The bottom line is that the potential for metrics and KPIs to improve the testing process comes down to tracking measurements which are conducive to continuous improvement.
Lastly, it's imperative to understand that no single dimension provides all the answers you need—a holistic approach is required to get the full picture on software quality.
Closing Thoughts
Before diving in and measuring a bunch of metrics and KPIs in your team, consider which values the Agile methodology promotes and what it aims to achieve.
Choosing wisely is half the battle with any measurement—a contextual approach weeds out those that provide little use to Agile teams and focuses on the ones that tell you:
- Whether the testing team as a whole is doing its job efficiently and effectively
- What issues there are with current testing efforts
- If enough tests are being automated to get testing up to speed with development
- Whether Agile teams are releasing high-quality software with sufficient frequency
The caveat with all test metrics is that viewing measurements in isolation can give a misleading picture of the testing efforts. Agile teams need a holistic approach that collates their useful test metrics and KPIs to provide a unified view on software quality.
