Using evidence to inform policy seems like a no-brainer. But often it’s not that straightforward – particularly for politically sensitive issues. Here, Mark Smulian looks at the progress of the What Works Centres, which aim to improve decision-making in key policy areas by exploring, well, what works
When a member of the public accesses any service from social care to policing, can they be confident those providing it really know what they are doing? Can they be sure that the service they are receiving is being delivered based on sound evidence of its effectiveness, following properly conducted trials?
They would be fine with a doctor, since the medical profession has long worked according to evidence. But suppose they needed a job in a struggling local economy? A local authority might be using evidence-based methods to engender economic growth, or just acting on hunches.
And if the person’s lack of qualifications prevents them from securing a job, might that arise from their teachers having ignored evidence of success and sticking to teaching methods fashionable when they trained years earlier?
Who has the best civil service in the world? DfID chief Mark Lowcock on the UK's plan to find out
How do ministers go about seeking advice from outside Whitehall? The simple answer is we don’t know
Science cuts could undermine evidence-based policy, MPs warn
Gus O'Donnell: gathering the evidence
The What Works Centres (WWCs) are attempting to spread the kind of evidence-based working which is common among doctors to other public services. If civil servants based their advice to ministers on evidence – and indeed were expected to do this – could the government get better results and avoid wasting money on ideas that fail?
The WWCs are based on the work of the National Institute of Clinical Excellence, founded in 1999 to produce evidence-based guidance for health practitioners. They began springing up in 2010 and vary widely. NICE is a full-scale organisation, while the Local Economic Growth WWC, for example, is a loose network.
The psychologist David Halpern (pictured), CEO of the Behavioural Insights Team and national adviser for the What Works Network, sees a straightforward financial case for underpinning policy choices with evidence. “It seems a good idea to spend all those billions of pounds and millions of hours on things that are more rather than less effective,” he tells CSW.
There has been too much use of what Halpern calls “spray-on evidence”, and “a strong tendency to believe the thing we already do is the right thing and never imagine the counterfactual” – which means only evidence that reinforces assumptions is used, rather than that which might point elsewhere.
A doctor would not “just wake up in the morning and think, ‘I’ll prescribe green pills today,’” Halpern says. “But when you drop your kids off at school, why should you think that the way they are taught maths is the best way? Was it a fashion when their teacher was learning?”
Halpern highlights work to use evidence to improve school performance by the Education Endowment Foundation (EEF) – a charity endowed by government but arm’s-length from it, and one of the What Works Centres. The EEF’s chief executive Sir Kevan Collins says it was a struggle to convince teachers and heads to understand and act on evidence, but the approach is starting to bear fruit – something which has become more important with the move to academies and free schools, in which head teachers are freer to make their own decisions but may be unused to finding and assessing evidence to support these.
“Previously, evidence was not widely used in education. It was largely a tradition of fads and fashions. We very quickly determined the organisation’s focus would be how to supply evidence to enable better decisions, rather than write guidance and use compliance to make people follow it,” Collins says. He adds that the EEF has made “a big push to make sure we present evidence in a way that is accessible to people”.
“Where an issue is very ideological, evidence is not a major determinant” – Jill Rutter, Institute for Government
“We use the phrase ‘disciplined innovation’, as in education there has been no shortage of innovation, [but] it has just been either based on a whim or never been tested.”
The What Works Centre for Local Economic Growth is a very different beast – a small organisation with a sprawling remit. Its director, Professor Henry Overman, explains that it is a joint initiative of the London School of Economics, the Centre for Cities think tank and engineering and economic consultant Arup. It has five staff but can call on 30-40 people from its parent bodies. “We cover anything people are interested in trying that might improve local employment and productivity,” Overman says.
One example of its work is an evaluation of a scheme by central London councils to provide caseworkers to help people unable to find jobs after leaving the Work Programme. The centre will compare the results to those of people on conventional post-Work Programme support to see if the intensive support is cost-effective.
This “control test” approach uses a control group against which experimental policies can be measured. It is “borrowed” from medicine and is fundamental to using evidence. NICE’s deputy chief executive Professor Gillian Leng says her experience of presenting evidence to civil servants who might be seeking to find and use evidence to make policy has led her to conclude that “because things are never black and white, using evidence is quite laborious. There are always conflicting views and managing stakeholders is tricky.”
“There are ideological constraints when dealing with political issues, which makes them more difficult than anything in clinical services because clinicians are trained to look at evidence. A lot of work has been done to help people really understand evidence so they can explain it to politicians and say ‘at least look at the evidence and understand why it might be important,’” Leng says.
“Previously, evidence was not widely used in education. It was largely a tradition of fads and fashions” – Kevan Collins, Education Endowment Foundation
Overman agrees: “There is generally not a technocratic solution to a problem and we are there to inform local authorities and ministers to help them make decisions. There are some examples of really high quality work in local and central government and some where, let’s say, they could do better.”
Those promoting the use of evidence face the problem that while most people can grasp its application in exact sciences, they are less sure when it comes to human behaviour. And if a politician has nailed their colours to some particular policy mast, un-nailing them in the face of contrary evidence may be too embarrassing.
Jonathan Breckon, head of the Alliance for Useful Evidence, has long met such objections. He says the use of evidence in issues concerning social policy “does translate well from science” and that even though “people are messy to deal with” compared with inanimate objects, there is now enough experience around to enable evidence to be sought for different solutions to policy questions.
Breckon says that if something is politically controversial, “it may be difficult to get evidence involved” and that he gets more traction where something is local or devolved, “such as hospitals or prisons where people are prepared to look dispassionately at evidence”. He adds that civil servants need skills to know and evaluate what is good evidence and what is merely “some dodgy survey from a campaign group”.
Some see limitations in the What Works scheme. Jill Rutter, a programme director at the Institute for Government, says: “What Works is valuable, but it’s very partial with only a small number of centres so far. There can be very different things happening depending on the size of budgets, and the data on centres’ budgets isn’t brought together and published in one place.”
Rutter says some policymakers are more open to evidence than others, and that “where an issue is very ideological, evidence it’s not a major determinant, though it can play a lesser role in big battleground policies.”
“On other, possibly more technocratic issues, evidence can drive policy. But what is regarded as an ideological issue and what is a technocratic issue is also a political matter,” she says.
Are civil servants generally equipped to assess and learn from evidence? And if they are, would politicians sanction an approach that means trying out several options knowing that some will inevitably fail?
“Of course there will be manifesto commitments made by an incoming government, but there will be a thousand different ways of doing those,” Halpern says. The What Works national adviser believes that civil servants should be willing to tell ministers that a variety of paths exist by which they might achieve their goal, and that several approaches should be tried out to see which is successful – rather than officials simply asserting that one idea is sure to work and should be rolled out everywhere. This approach, Halpern says, would produce “a different type of civil service, with this humility built into it, which over time will be much, much more effective.” He adds that Chris Wormald, DfE permanent secretary, is “a great champion of using evidence”.
Wormald (pictured) himself tells CSW: “The centres are leading the way in getting properly validated evidence on what works and what doesn’t directly into the hands of decision makers, helping us all to make better decisions.”
Meanwhile Cabinet Office minister Matthew Hancock has also given the WWCs strong support, calling them “a real success story, showing how through proper research and use of data we can revolutionise the way government and the civil service works”.
Where might this heavyweight enthusiasm take the WWCs? Halpern singles out social work as needing a centre, but thinks they could extend across public services. He also expects using evidence to become second nature to civil servants.
“We will look back and think: ‘What were you doing before?’ Once you get into more experimental empirical methods, it’s difficult to think back to a time when you didn’t do it.”
What Works Centres: Key early findings
- Police patrolling small areas where crime has been concentrated reduces crime and does not simply move it nearby
- People with acute heart failure should be routinely treated by specialist heart failure teams
- Peer tutoring approaches, where learners work in small groups to provide each other with teaching support, have, on average, a high impact on attainment at a low cost
- Major sporting and cultural projects rarely have large local economic impact