How Project Ronin Delivered 90% Faster in 6 Months.

Project Ronin is on a mission to dramatically improve cancer care, through a cancer intelligence platform that provides information for proactive and reactive care.

Swamped by several product lines, their engineering team were struggling to deliver features on time to patients that depended on them. This is when their VP of Engineering, Denali Lumma took charge of transforming Ronin’s software department by utilizing Haystack's insights and alerts.

"Haystack was absolutely a life saver for us to get insight into a little bit more fidelity and a little bit more fine grain awareness.

Having all the Metrics in one place at the company to show to the Board and Executive team in a really quantitative and data driven way was extremely valuable and useful to have"

Denali Lumma - VP of Engineering, Project Ronin

The Key Challenge

Project Ronin was struggling to meet software delivery deadlines and their team was swamped with requests from multiple sources, leaving decisions around prioritization and execution up to intuition.

“One of the key product lines that we were working on was 16 months late and the team had no idea when it would be ready. We didn't really operationally have anything in place from an actual process perspective, it was really chaos.”

This resulted in a lot of hidden work happening behind the scenes without anyone knowing, effectively leaving their management blind. It became piercingly apparent that steps needed to be taken to mature their processes, thinking and infrastructure. This was when Denali set about studying metrics on her own and discovered Haystack which gave her team the visibility and guidance they needed to experiment with different changes and assess in real time what improved their delivery.

Journey

Denali and her team had started off tracking metrics on their own initially utilizing JIRA. Finding this lacked the fidelity they needed, Ronin moved to calculating metrics on their own directly from Git, with Denali making sure to only use the metrics that gave clear insights while avoiding harmful and misleading metrics like lines of code.

“I’ve known that bad metrics can be extremely damaging and metrics can be extremely harmful and you have to be very thoughtful about what metrics you choose and what metrics you track.”

They conducted process experiments by implementing small changes spread across sprints, and measured the impact of said changes to assess whether they improved delivery. Haystack helped Ronin visualize and automate their data collection giving them greater visibility into how their team operated.

‍“We could sort of hack together some calculations on our side, but having all of the metrics in one place, really cohesive and consistent to show very clearly to the company, to the board and to the executive team, that this is how we're doing from a velocity perspective, this is how we're doing from a quality perspective, here are the steps that we're taking and here's the impact that it's having in a very quantitative and data driven way really gave us a ton of credibility and was was extremely valuable and useful to have.”

Cycle time became a key NorthStar metric that their team began to drive towards as an indicator of better delivery.

Ronin then leveraged Haystack’s deployment rate as a guide to understand how they were doing in terms of velocity.

This combined with metrics like code coverage and change/failure percentage allowed them to have a view into how they were doing in a quality perspective as well.

Together this helped paint a picture of how their team was operating in terms of business performance measured through metrics like revenue and churn.

Results

Ronin initially had a 30-day cycle time which had a huge rate of variation, making predicting delivery dates near impossible, in 6 months they brought down their cycle time to just 3 days, and saw their predictability drastically rise to 98%.

“Some pull requests took 30 days, one took 5, one took 4550. It was all over the map, and it's super hard to predict. But as our cycle time dropped from 30 days to 15, to 10, to five to three or two, what we found is that the deviation became almost non-existent.”

This made their work decomposed and much more predictable, as they would know exactly how long it'd take to build something broken down into smaller projects. This also benefited them in terms of quality as In Q2, 180% of all their deployments required an intervention and as they experimented this radically decreased to 36% in Q4, a 143% improvement.

“We went from one deployment every nine days to one deployment every 1.5 days over the year. And that was truly, truly incredible to see.”

Velocity was quite significantly improved by 52%, but what was most astounding was the deployment rate by quarter, which saw a 514% acceleration in how quickly they were deploying code, with other factors like sprint burndown and ontime feature delivery also dramatically improving.

“Haystack were the most advanced in terms of feature capabilities and feature set. Starting to use their tool was absolutely a lifesaver for us to really get insights into more fidelity, and more more fine grained awareness for things like, Git cycle time, Deployment frequency, and change/fail.” ‍

‍