Beginner's Guide to Software Delivery Metrics

What Are Software Delivery Metrics?

Software delivery metrics give us quantitative ways of measuring our team’s progress and performance. 

They help us understand how we deliver and what’s getting in the way. They give us a tool to iterate/improve our delivery while helping us communicate progress to other business stakeholders.

In this article we’ll give you an easy framework to think about delivery metrics and how they might fit into your team.

But first..

Why are delivery metrics useful? 

When you search for "engineering delivery metrics" the results are filled with the same half-baked “Top 10” list of mishmashed metrics.

Fortunately, Dr. Nicole Forsgren and her team have done all the heavy lifting and summarized their findings in a book called Accelerate. It’s by far the most comprehensive view of high performing engineering teams with research across thousands of companies.

In Accelerate, Dr. Forsgren and her team identified four key metrics that are proven indicators of software delivery performance. Their research across thousands of organizations showed that if software teams can excel at delivery (across these 4 metrics) then their companies perform better as well. 

Elite performers (across the 4 key delivery metrics) not only delivered 100x faster, more frequent deployments but their companies showed higher rates of profitability, market share, and customer satisfaction.

Choosing the right delivery metrics

There are many things we can measure, but very few things we should.

Choosing the right metrics always starts with ‘why’. The most important reason to use metrics is to fuel continuous improvement. 

Want to ship faster?

Improve quality?

Remove inefficiency?

Metrics give us a tool to do that.

Rather than taking a stab in the dark, metrics give us a strategic way to approach improvement. Allowing us to experiment, measure the results, and improve at a much faster rate. 

Here’s a great story of delivery metrics in action from Splice’s VP of Engineering.

For the remainder of the article, we’ll be discussing types of delivery metrics and help you decide which metrics matter for your team.

‍

Types of Delivery Metrics: Leading vs. Lagging Indicators

A leading indicator is a predictive measurement, for example; the percentage of unit test coverage is a leading quality indicator. Leading indicators are a tool to influence team behavior and improve on lagging metrics.

A lagging indicator is an outcome measurement, for example; the number of escaped defects to production (# of bugs) is a lagging quality indicator. Lagging indicators are used to gauge team health and identify large scale issues.

Lagging Indicators - “North Star” Metrics

Lagging Indicators give you a north star goal to align your teams. 

Lagging Indicators should always strike a balance. You never want to sacrifice one for the other so it’s important that these lagging indicators (or North Star Metrics) enable us to set a balance of what we want to achieve.

Examples of Lagging Metrics:

The four examples we give below give a comprehensive view of speed, throughput, quality, and stability. 

Speed - Cycle Time is the time from first commit to pull request merged. This helps us understand how quickly changes are being completed. 

Throughput - Deployment Frequency is the frequency of production deployments.

Quality - Change Failure Rate is the percentage of deployments that require subsequent fixes (hotfixes, bugs, etc). This helps us understand the quality of deployments.

Stability - MTTR is the mean time it takes to restore service. This helps us understand how stable our system is and how quickly we recover from downtime.

Leading Indicators - “Influencer” Metrics

Leading Indicators help us influence outcomes (or Lagging Metrics). 

You can think of these as ‘habits’. Just like eating vegetables (leading indicator) helps us achieve better health (lagging indicator), leading indicators help us influence our high level goals.

Never use lead indicators alone as this can lead to incorrect behavior. 

Example: Using unit test coverage alone incentivizes the team to create many tests but does not necessarily ensure quality. The goal is not 100% code coverage, rather to improve lagging indicators (e.g. escaped defects) and enable teams to deploy quickly, with confidence (deployment frequency).

Examples of Leading Metrics:

While this is not an exhaustive list of leading metrics, these are the most effective leading metrics we’ve identified after working with 200+ engineering leaders.

Review Time - Review time is the time it takes on average to review a pull request. Decreasing review time allows teams to increase iteration speed, feedback on features, and (since pull requests inevitably become smaller) higher quality releases.

Rework Time - Rework time is the time spent ‘reworking’ or ‘rewriting’ code during review. Decreasing rework time allows teams to increase speed and reduce waste (since teams put more focus on design and quality during the development process)

Work in Progress - Work in Progress is the amount of active concurrent projects per developer. Decreasing work in progress allows teams to increase speed and reduce the cost of context switching - resulting in faster, higher quality deployments.

Batch Size - Batch size is the average size of work completed. Decreasing batch size allows teams to increase speed, reduce the cost of mistakes, and deploy more frequently.

How To Use Delivery Metrics

Delivery metrics help instill a culture of improvement. It gives us a tool to test, experiment, and make progress towards better, faster delivery.

1) Define ‘North Star’ Metrics - Decide which metrics matter for you. Remember to only select metrics which your team can control (i.e. delivery metrics) and limit yourself to few key  ‘North Star’ metrics to maintain focus. 

2) Establish Baselines - Start to measure across each core metric. Establish how your team performs across these metrics to give a point of reference on improvements.

3) Identity Opportunities - Your North Star metrics should help you highlight where your team can improve. Whether it’s speed, quality, throughput, or stability - use your metrics to determine the biggest opportunity to improve. 

4) Make Improvements - Invest in the opportunities you just identified. Take concrete, actionable steps to improve on your baselines.

5) Measure Results - Circle back to your metrics to see if your changes had a meaningful impact. If not, re-evaluate your approach and find other ways to improve.

6) Repeat - You should never stop improving. Continue this process to ensure your team is keeping up with elite performers. Netflix, Google, and Amazon deploy over a thousand times per day and still continue to improve.

Just like we iterate and improve our product, we should iterate on how we deliver. The best teams never stop improving and delivery metrics give us a scientific approach to improving how we deliver. 

Avoiding Common Pitfalls

While the ancient days of ‘lines of code’ are well behind us, it’s important to discuss how to avoid the mistakes of the past.

Backed by years of research across thousands of teams, we’ve compiled a list of ways to avoid misuse of metrics:

1) Measure Teams, Not Individuals - Software is a team sport. Your metrics should reflect that. We should be looking to improve team performance rather than evaluate individual engineers. Sticking to this rule will help you get team buy in and avoid misuse of metrics.

2) Track Trends, Not Absolutes - Every team is different. We should be measuring the rate of improvement rather than absolutes. This helps us avoid unrealistic comparisons and over optimization of any particular metric. 

3) Improve, Not Blame - Focus on team improvement. In no way should metrics be used to point blame or criticism but rather identify team-wide opportunities to improve. Never point blame to an individual using metrics. 

4) Balanced Metrics, Not Silos - Always strike a balance. You should never focus on one particular metric at the detriment of others. Make sure you’re striking a balance between speed and quality to get the most out of your metrics.

5) Bring Questions, Not Conclusions - Be curious. Always work with your team to identify opportunities and discuss solutions. Presenting solutions without including your team doesn’t allow them to take control of their improvement and can lead to push back. 

Want to start using delivery metrics?

Gone are the days of throwing spaghetti at the wall and hoping for improvement. 

There are proven methods that drive improvement and ignoring them can be fatal not only for your team but, as the research shows, your organization as well.

If you’ve made it this far, then the natural next step is to get started.

Sign up for Haystack and get access to your core delivery metrics (leading and lagging) plus helpful Slack alerts to keep your team on track.

We’ve helped hundreds of teams improve delivery speed by 40% and would love to help you do the same.

‍

Haystack helps engineering leaders identify burnout and trends. Instead of guessing if you're improving, or constantly bothering your team for progress updates, simply use Haystack to get alerts in your inbox every morning. Plus a dashboard to track improvements over time.

‍

Try it for free

‍

‍