The Royal Statistical Society has launched a project exploring how statisticians have generated data in pressurised situations that it hopes will help others to weigh up high-stakes trade-offs between timeliness and quality of data.
A series of case studies published under the RSS's new Statistics Under Pressure initiative aim to "raise awareness around the need for trade-offs in real-world circumstances, along with the merits of such an approach".
The case studies, which include accounts of the civil service teams tracking Covid infections and generating statistics to support the development of the furlough scheme, will be used to create a range of materials to help decision-makers and the public to interpret and use data that is associated with limitations or uncertainties.
The Covid pandemic "highlighted the importance of data being used to inform decision-making at pace", an introduction to the project on the RSS's website reads.
"For statisticians working under pressure, it can be challenging to judge when imperfect data is good enough to inform decisions, and which trade-offs can appropriately be made," the RSS said.
"This project aims to foster an environment in which statistics, data and modelling that are good enough to inform time-pressured decisions can be used with confidence, especially in cases where the data is not perfect and decisions might otherwise be made in the absence of data."
The RSS published the first three of nine Statistics Under Pressure case studies this week.
Developed in collaboration with academics and officials from across the Government Statistical Service, the case studies focus on three scenarios where “decisions that had to be made at pace in evolving situations under much public scrutiny”.
The first explores the pressures faced by the team setting up the Covid-19 Infection Survey – which would go on to win the RSS's Statistical Excellence in Trustworthiness, Quality and Value Award in 2022 – at the height of the pandemic as they worked to solve the problem of how to track case numbers.
The second looks at a large-scale trial investigating the effectiveness of badger culling when trying to curb Bovine TB. The case study asks how much data is needed to answer a problem and explores how scientific evidence must be weighed alongside other policy, economic, ecological and ethical considerations in policy decision-making.
The third case study examines how a team in HM Revenue and Customs worked to provide rapid statistics to provide real-time information on the Coronavirus Job Retention Scheme put in place in 2020. It looks at how the team allocated its limited time and resources to provide monthly updates on the furlough scheme amid a global health emergency.
Six more case studies will be published later this summer, covering topics ranging from transport statistics to economics.
Once all the studies have been published, the RSS will publish a set of principles learned from the research. These will “provide guidance to statisticians on the key factors that must be considered to successfully make trade-offs to allow data to best inform decision-making”, the RSS said.
Through the project, the society aims to generate a range of materials to support statisticians, decision-makers and members of the public who need to provide and interpret data in fast-paced circumstances, it said.
These materials could include guidance for decision-makers on how to interpret data with limitations; guidance for analysts on how to communicate the limitations of data; circumstances in which the RSS supports making trade-offs to allow data to inform decision-making; accessible wording to help defend decisions in which trade-offs have been made; and policy recommendations on structures and resources to support fast-paced data-informed decision-making.
The project is being overseen by a steering group led by RSS president-elect Prof Sir John Aston, who was chief scientific adviser at the Home Office between 2017 and 2020. The group also includes Clare Griffiths, who was previously head of the UK Covid-19 dashboard, now head of public health analytical product development at the Department for Health and Social Care; and Sarah Walker, chief investigator of the Covid Infection Survey, and professor of medical statistics and epidemiology at the University of Oxford.
Aston said: “Statisticians and analysts face a number of competing challenges in ensuring their data is timely yet of high quality. This is never more the case than when data is needed in evolving and high-stake situations.
“Through our Statistics Under Pressure initiative, we aim to start a discussion as to how statisticians can best be supported when faced with these demands and the lessons that can be learnt for the future.”