Data. Governments need it; governments produce it; governments use it. It’s no exaggeration to say that data has become the very lifeblood of government. And as the UK grappled with the onset of Covid-19, it had never been more important.
However, the data needed for informed and timely government decisions and improved patient outcomes was often rendered inaccessible as it sat across a myriad of disparate government organisations and systems. Policymakers, who were operating under immense pressure, often saw valuable analyst time consumed by spreadsheet manipulation as they sought to track down the information they needed.
It’s important to remember that to maximise its value, data needs to be understood and not looked at in isolation. This meant data had to be consolidated, made consistent and accessible, whilst protecting GDPR and citizens’ privacy.
The sheer scale and urgency of the challenge meant that policymakers alone couldn’t shoulder these issues. So, organisations from across the private sector came together to lend their expertise. The result, however, was a patchwork delivery structure.
So, what could be done?
There was a clear and urgent need to create a single platform to house consumable terabytes of data for data scientists and data analysts to use, whilst always adhering to GDPR and other ethical safeguards. The platform needed to drive up quality and massively reduce the manual effort to consolidate data and generate insightful reports. All within a secure environment owned by the department.
Within three weeks we had begun to build a data platform to consolidate the vast array of data needed for the pandemic response and emerging Joint Biosecurity Centre. In only six weeks we had loaded our first dataset onto the platform and within 12 weeks had incorporated sufficient data to enable an 80 per cent reduction in effort to create the weekly statistics report for DHSC.
Where previously it had taken weeks to expose critical datasets to a wide analyst community, the platform produced trustworthy datasets within a matter of days or even hours.
Blending pace with teamwork
The experience of operating effectively in such intense, high-pressured circumstances, holds some important lessons for future projects. The pace of the programme showed that bringing technical teams and user communities much closer together can deliver huge benefits. Although not always possible, the integration of all aspects of service design and delivery transformed the end result.
Keeping platform engineers, data engineers and data scientists very close to our analyst colleagues was critical. Not only did it enhance resilience, access to extended knowledge and planning for future activities, it also ensured we created a community that enabled issues to be prioritised and resolved swiftly – a shared ownership approach that generated real value.
This flexible and inclusive culture was imperative because it ensured that team members worked for the programme, rather than more limited commercial or departmental goals. This was particularly important given the increasing demand for different datasets and the growing need and mandate to bring datasets together within the fluctuating priorities. Such requirements are, quite rightly, likely to become pre-requisites in other government transformation programmes.
A results business
The new platform and capability now enables the processing of more than 2.7 billion records, or 1.9TB of data a day, effectively providing contingency for several heritage systems, and has enabled timely informed decisions to be made across government.
We enabled system-wide stakeholder engagement and secure movement of data across complex systems and organisational boundaries. We have also fostered and facilitated sharing of data between multiple government departments. And critically, we delivered an enduring asset which can support ongoing disease surveillance and future pandemics.
At the heart of these efforts was our central belief that data holds the key to improved performance. Data’s very ubiquity has heralded huge consequences for organisations large and small. Only those which have the ability to capture it, understand it and use it to fuel strategic and operational decisions are able to reap the dividends it has on offer.
About the author
Andy Lethbridge
Head of Consulting, Central Government
BAE Systems Digital Intelligence
andy.lethbridge@baesystems.com