When Agile Goes Wrong

Over recent weeks, the Post Office Horizon IT Scandal has reached a new level of public interest in the UK. In the scandal, faulty accounting software has been blamed for multiple suicides and what has been described as “the most widespread miscarriage of justice in UK history”, with those wrongly imprisoned including a pregnant woman.

However, one of the factors that hasn’t much been discussed in the public domain yet is how this was one of the earliest pioneering projects which used an agile methodology during software development - namely Rapid Application Development (RAD), a precursor to modern development methodologies like Scrum, Extreme Programming (XP) and Kanban.

The lack of concrete requirements through the use of RAD has become a recurring theme in the evidence heard by the Horizon IT Inquiry. Before the inquiry, Terence Austin remarked: “one of the reasons why this got into this situation is that we were forced to do rapid application development and, by doing that, you haven’t got a functional specification”.

David McDonnell, a former Fujitsu engineering manager (and one of the first internal whistleblowers of the issues) said in his evidence to the Horizon IT Inquiry: “Well, a project such as that -- well, any kind of software development project, there should be a framework of how the team work. It should start with the design documents. That's the target of what you are trying to deliver; that's what you are building against.” … “I know that there had been some documents that were reverse engineered, but they were irrelevant and out of date, and they weren't even in the building when I got there. I had to ask for them.”

Toyota is credited as one of the pioneers behind the Agile methodology with their creation of the Toyota Production System (TPS). In another scandal, Bookout V. Toyota, defective software has been blamed for multiple deaths through causing unintended acceleration in vehicles. Essentially placing drivers in a position where their cars were accelerating and the brakes would not operate.

In his professional evidence during the trial, Michael Barr of the Barr Group, pointed to internal communication within Toyota stating: “In truth, technology such as failsafe is not part of the Toyota Engineering division's DNA”.

The speed-focussed approach in Agile has recently been underpinned by metric frameworks like DORA metrics, where the speed of shipping features and resolving bugs is placed as the most important factor. From both my own experience and that of others I’ve spoken to, teams will often turn to Agile to attempt to focus projects towards timely delivery when they fall behind schedule. And software projects do fall behind schedule a lot, with an estimated 70% of software projects failing to be delivered on-time.

The teachings of Agile, like reducing the amount of work-in-progress and reducing task switching, are hugely powerful in improving software delivery. However, Agile is necessary but not sufficient.

Whilst it can be tempting to assume that mounting time pressure is the solution to fixing late delivery, this often does more harm than good. The solution to fixing a shower with low water pressure, is not to make the shower intermittently pump out high-pressure water at scolding high heat. The answer is to have a steady flow at the perfect temperature.

It is therefore critical to actually diagnose and resolve the issues that cause delays, not just try to accelerate broken processes by producing more stress, burnout and risks to society.

Instead of simply acting as a speedometer of broken processes, Delivery Ops Platforms like Haystack instead seek to analyse delivery and unblock the process when issues appear in real-time. This means that engineering rigour and agility can be backed by the ability to unblock delays when they actually happen. Through approaches like this, addressing the underlying issues, it's hoped that these horror movies won't become a trilogy.