Civica's Bob McClean explains how the emerging techniques of Hybrid Agile retain Agile’s user-focus and flexibility – so valuable in the battle against COVID-19 – while meeting government’s requirements on risk allocation, regulation, supplier incentives and business planning
When emerging technologies meet established organisations, something’s got to give. And in the UK civil service, inherent tensions between digital project management techniques and mandatory business planning processes have not been resolved – leaving project leaders struggling to bridge an uncomfortable gap. For in an ideal world, digital systems are best delivered through ‘Agile’ development methodologies, in which developers test and feel their way towards a conclusion. But government-wide spending approval systems ask for that conclusion up-front, demanding clarity over a system’s final form and functionality before development funding is released. Until this conundrum is resolved, digital leaders will be caught in a Catch-22.
As civil servants scramble to address the huge challenges presented by COVID-19 – creating at pace new systems to support remote working, strengthen healthcare delivery, roll out the economic stimulus, and gather and analyse clinical data – it’s more important than ever that working practices and governance systems fit the needs of effective delivery. With time so precious, we must minimise friction in civil service project management while getting the best possible value out of suppliers.
But currently, digital schemes are being forced through spending approval frameworks that discourage the use of appropriate methodologies: a business planning system designed with the best of intentions – to minimise risk in the use of taxpayers’ money – is inadvertently damaging projects’ chances of success.
Now, though, an adapted form of Agile is emerging, wrapping elements of traditional ‘Waterfall’ project management around the Agile process to provide foresight and scope control. For civil servants operating in the real world of government, ‘Hybrid Agile’ offers a way to meet the needs of both digital developers, and those charged with protecting value for money in the use of public resources.
The weaknesses of Waterfall
When applied to hard infrastructure projects, the civil service’s spending approval systems work pretty well. Though ‘optimism bias’ and policy changes can drive up costs and timescales, planning and construction professionals tend to know well in advance the precise specifications of what’s required – and the sector has developed its delivery expertise over centuries. So by scrutinising detailed delivery plans, finance managers and accounting officers can draw well-founded conclusions on their realism and economy, preventing ill-conceived schemes from wasting money.
Yet over recent years, government has found that fixing in advance an IT system’s precise functionality often generates poor outcomes, producing systems that require expensive re-engineering or fail to meet user needs. And once committed to a set plan, project managers can be reluctant to highlight emerging problems; we’ve all seen the unhappy results. The Government Digital Service’s Design Principles represent an attempt to reshape project management around the needs and nature of digital technologies, promoting the Agile techniques developed to support effective delivery in what is still a relatively new field of human endeavour.
Agile: pros and cons
Pure Agile development begins with a ‘Discovery’ phase, in which ‘personas’ are developed for all the intended user groups – the system’s managers and operators, as well as service users – and narrative descriptions of their interactions with the system are created. These ‘user stories’ are then used to create a Product Backlog, setting out the features and functions required to satisfy each user group’s needs.
Next comes the ‘Alpha’ phase, in which developers work out how best to build those features and functions – repeatedly testing each pathway with users, then amending them to meet the project’s goals. These pathways are coded in the ‘Beta’ stage, with developers building a no-frills ‘Minimum Viable Product’ that can itself be tested, tweaked and perfected.
Having repeatedly burnt their fingers during the era of pre-defined, monolithic IT developments, practitioners have learned that in the digital world, the only sure way to ensure that a system meets user needs is to build it hand-in-hand with those who’ll be operating and accessing it. But Agile has major drawbacks when applied in government.
Neither the work required to plan out a system’s operation, nor the nature of that operation, can easily be assessed before the project gets underway (indeed, purist would see that as flying in the face of Agile); so it’s impossible to accurately cost projects before work begins. It’s also demanding for clients, who are repeatedly presented with evolving user stories and asked to map out a way forward at each stage. With multiple user stories being developed simultaneously, coders can end up replicating each other’s work where journeys coincide – leading to needless duplication. And there are particular challenges in highly-regulated environments, where regulators require certainty that a system’s operation will comply with relevant legislation.
Introducing Hybrid
The Government Digital Service is doing what it can to amend business planning processes, but we need a high-quality digital project management methodology that better meets the government’s needs – and Hybrid Agile provides it. Various versions of this methodology exist, and we at Civica have developed one to suit the needs of our clients in the UK civil service.
Hybrid begins with a 4-8 week Discovery exercise, built along pure Agile lines, working with civil service bodies to determine the usual User Stories and Product Back Log. Then, replacing pure Agile’s Alpha phase, we insert a Technical and Commercial Proposal phase: this fulfils some of Alpha’s functions, but shifts the burden of refining the Discovery outputs from the client to our researchers and analysts. Instead of asking clients to navigate each user story, we map them out ourselves, then refine and drill down to a lower level of detail – allowing us to set out a shared understanding of what is needed.
With each story refined down, we conduct a ‘function point analysis’ – using an industry-standard metric to specify and quantify the number and complexity of functions required – and create digital mock-ups to illustrate the system’s processes and screens. So rather than receiving a 300-page proposal narrating each user story, clients can explore a live representation of the system in action. And instead of building out each user story with no clear sense of how the system will work as a whole, project leaders can take a holistic view of its operation before getting stuck into development – identifying potential problems and opportunities for wider innovation.
Pricing and transparency
Importantly, the function point analysis also enables us to provide a fixed price for project delivery – enabling project leaders to get their business cases signed off – without requiring clients to nail down every aspect of the system’s specification. For throughout the project, we can keep on adapting our plans for any parts that remain unbuilt. Using traditional Waterfall, clients pay high charges for any change to the plan; in Hybrid, as long as the total number of function points remains steady, the price does too.
What’s more, in some cases clients can look ahead and identify elements of planned functionality that may not be required – either reducing the project’s cost, or ‘spending’ the points to create new features: in pure Agile, you can’t look forward at all. So Hybrid retains the adaptability of Agile, while offering much greater clarity and confidence over the project’s cost, timescale and structure.
Our Hybrid technique has further advantages during the development process. The function point analysis identifies commonalities between user journeys, enabling us to save time and money by building shared components first. Our digital mock-ups form the basis for an automated demo environment, giving the client a clear view of the evolving system and enabling us to test our work as we go. And as developers work their way through the function points, project managers can see exactly how close the system is to completion.
Incentives and risk
Hybrid also better aligns the supplier’s incentives with the client’s interests. In pure Agile, clients effectively estimate the required functionality during the Discovery and Alpha phases; if they’ve missed something, they’re on the hook for the price of plugging the gap. And the lack of a definitive scope for the project makes it impossible to set a fixed price, so clients typically pay on a time and materials basis: this gives suppliers perverse incentives – encouraging them both to maximise the development time required to deliver the project, and to expand its scope. Too often, clients find themselves taking on unquantifiable bills for unforeseen development work: an uncomfortable prospect as officials continue investing in a system whose final shape and scale remains opaque.
In our version of Hybrid, though, Civica both estimates the work necessary to provide the required functionality, and provides a fixed price for delivery. So we’re incentivised to complete the project as quickly and efficiently as possible; and if we’ve under-estimated the development time required, we pick up the bill for the extra work required.
On the client side, the ability to quantify projects’ scope using fixed-price function points provides another useful feature: by specifying the price of any additional functionality, it mitigates against the ‘project creep’ that can see costs spiral as stakeholders insert additional requirements. The only substantive risk for the client lies in civil servants’ ability to correctly assess our user stories and the mock-up environment: if they’ve failed to spot where required functions are missing from the plan, they may need to add function points further down the line. And the only way in which Hybrid offers less transparency than pure Agile during the development process lies in the coding of user stories: because we replace duplicated stories with common components, some pathways may reach completion later than they would otherwise.
A system that suits
We find that Hybrid has particular advantages for certain kinds of civil service clients and projects. In highly-regulated environments, the ability to provide an overall picture of the system’s operation before full-scale development work begins gives regulators confidence that it’ll meet their requirements. Its tendency to shift risk from the client to the supplier suits the needs of small organisations, which have less to gain from reducing transaction costs and more to lose from project over-runs. And its use of a more traditional project management framework enables clients to monitor the detail of systems’ development and operation, making it suitable for very complex projects.
As the IT industry has morphed into the digital sector – and as developers and project managers have learned how best to build and manage the tools flowing out of the ‘fourth industrial revolution’ – the weaknesses of Waterfall in the digital environment have been repeatedly exposed. As a result, practitioners have developed a brand new approach to project management: Agile. But the methodology, built to meet the needs of digital professionals, takes little account of long-established civil service processes and the complexities around government culture, accountabilities and regulation.
A happy compromise
Hybrid Agile represents an attempt to retain Agile’s flexibility and user-focus, while providing budget holders, central officials, accounting officers and regulators with the confidence they need to sign off digital schemes. Having used Hybrid in a number of successful UK civil service projects, I’m also a big fan of the incentives it creates: pure Agile was built to focus on user needs, but it encourages suppliers to gold-plate systems at the client’s expense.
Coming 12 years after the financial crisis and four after the Brexit referendum, COVID-19 is just the latest in a series of vast delivery challenges to confront the UK government; it certainly won’t be the last. To address these challenges, civil servants need to realise the huge potential of digital – and that requires project management methodologies built around the technologies’ needs. But it’s unrealistic to expect every civil service process and structure to fit itself around pure Agile’s unique approach. Hybrid Agile provides a way for civil servants to deliver digital schemes quickly and successfully, while meeting the needs of users, civil service business owners – and the officials tasked with protecting regulated environments and the taxpayer’s interests.