Interview: Hetan Shah, The Royal Statistical Society

Does the government value its statisticians highly enough? Hetan Shah, executive director of the Royal Statistical Society (RSS), starts his response with a characteristic statistician’s qualification: “I think it depends: you have to drill down to each department.” But this is one of the few times that Shah sounds anything like the stereotypical statistician, hunched over a spreadsheet discussing sample sizes.

With a background in law and policy work, Shah is clearly interested in statistics, but one senses that his underlying passion is for their use in shaping policy-making and public debates, rather than in the management and analysis of data itself. For him, the value of statistics lies in their potential to shape events in the real world – so he notes with concern that over the years, the status of government statisticians has fallen.

“If you go back to the 1970s, [eminent academic] Claus Moser was the head of the government statistical service; [prime minister] Harold Wilson was very pro the statistical agenda; and statisticians tended to be on the board or at board level,” he says. By contrast, in March 2012 the outgoing national statistician Sir Michael Scholar noted that across government there were just eight statistical managers at director level or above, of whom six were employed by the Office of National Statistics (ONS) or the Statistics Authority.

Shah says he hopes there will be a “shift back to statisticians being recognised as having something to contribute at board level”. Their work will be increasingly important, he argues, because the government wants to make better use of its vast stock of data.

Big data v statistics
“We are big data,” says Shah. “The statistical community must take ownership of this agenda.” Originally the term ‘big data’ referred to data sets so large they couldn’t be understood with traditional means, he adds, but now it’s most commonly used to describe the “ubiquity of data”. If data is everywhere, then society will need more of the statistician’s skills of organising and understanding information.

“Using data sets which we hadn’t thought of using before, and which perhaps weren’t collected for statistical analysis, will really require people who can think through these issues around: ‘What’s the quality of that dataset? How did we gather it? What does it, and does it not, tell us?’,” says Shah. In government, policymakers don’t want data, he continues; they want answers – and bigger data sets are harder to wrest answers from. “So the skills that will be in high demand are those of being able to look at a data set, make sense of it, [assess] if it’s of high quality, and ask: ‘What it can tell you?’”

Big data is much messier than traditionally-gathered statistical data – think of the unstructured, varied information in a Twitter feed, compared to the neat, uniformly-formatted answers to a well-composed survey. “Some of these things are pushing new boundaries for statisticians, and so we have to look again at some of our old methodologies to cope with that,” says Shah: there is a “genuine challenge to the statisticians profession” in developing the tools not only to handle, but also to visualise and explain this data.

The limits of evidence
Shah believes that the quality of statisticians in the Government Statistical Service – and of the data they produce – is high. He is less confident about the levels of statistical understanding across government overall, and says he regards many civil servants’ lack of a statistical understanding as an “important challenge”.

“Partly, I think you address that through raising the [level of statistical] skills,” he says. His own organisation is working with Chris Wormald, the policy profession’s head, to improve policymakers’ statistical skills, “but that’s a long-term agenda.”

In the short term, he suggests, the answer lies in ensuring that statisticians work closely with other teams in their departments. Policy teams are an obvious example – he’d like to see statisticians brought into policy discussions early on, to improve the collection and use of evidence in devising policy solutions. Government is not good at evidence-based policymaking, he believes, “but there are some positive signs” – for example, the creation of the ‘What Works Network’, which brings together academics and civil society organisations to gather evidence around key policy challenges. Though still in its early days, this is “an important step” he says, and it has support “from the very top, so that’s an important signal”.

He also praises work done by the Cabinet Office Behavioural Insight Team to use randomised control trials as a way to test policy options. “We’re not saying RCTs are the only way; indeed, they have some drawbacks,” he says, adding that “in complex areas of social policy” RCTs won’t “explain the underlying causal process for an intervention working.” This can makes it harder to understand whether an intervention can be applied in other contexts, and so other kinds of research, such as qualitative techniques, may be more appropriate. Nevertheless, he says, “the idea that you can use evaluation quite quickly to decide between different policy options and that that evidence will give you a bang for your buck can be very powerful.”

For every policy that’s announced, he continues, “we’d like to see the evidence base published both for and against, so you’ve got a full evidence base and then it can be challenged.” It sounds like an idea unlikely to win friends in politics, but in fact Cabinet Office minister Francis Maude backed this idea at this year’s Conservative conference.

However, Shah is realistic about the constraints on using evidence to shape policy decisions. “We’re not claiming that evidence gives you the answers.” he says: politicians must weigh that evidence to devise workable answers. “Sometimes, they will say the evidence tells us X but we’re going to do Y because X is not publicly acceptable, for example, or [because] we think that the evidence is wrong.”

Statistics speak for themselves
Policy is not the only area where Shah wants to see statisticians involved more closely in colleagues’ work. “Often, it’s the press teams that are the worst at this sort of thing,” he says. “Policy colleagues will take the stats seriously, but once you get into the press team...probably less so.”

Despite his frankness about press teams, Shah doesn’t want to give specific examples of when government has communicated statistics badly. Perhaps he doesn’t have to, since a quick scan of the Statistics Authority website will lead you to the correspondence of its chair, Andrew Dilnot, who regularly and openly chastises ministers – and even the prime minister – if he feels they or their civil servants have misused statistics. Recently, he has criticised work and pensions minister Iain Duncan Smith over a statement about the effects of the benefits cap, and communities minister Eric Pickles over the decision to stop publishing statistics at a regional level.

Criticism also comes from the increasingly powerful select committees. In May of this year, the Public Administration Select Committee (PASC), in a report on the communication of statistics, highlighted mistakes in an ONS press release on trade deficit data: press officers had misinterpreted the data and focused on an improvement in the figures over the last year, rather than the longer-term underlying decline.

Like PASC, Shah thinks that the answer lies partly in statisticians taking a more active role in communicating their work. “Statisticians need to get better at telling the story of what the data means,” he says. This requires the profession to be more confident, but it’s also “a shift in what’s expected from the statisticians. This idea that the data speaks for itself is something we’re moving away from.” Another part of the solution, he says, is to understand who’s using statistics and why, in order to communicate specifically – and, hopefully, more effectively – with them.

Another answer to the problem of spin – one supported by PASC and the UK Statistics Authority – is to reduce the extent of ministerial access to statistics before they are publically released. Currently, ministers and some civil servants can see statistics 24 hours before they are released. The consequent leaks and misrepresentations of the figures have damaged public trust in government stats; and Shah says the issue has been a “longstanding concern” for the RSS, which has argued for this pre-access to be removed entirely. The UKSA wants to see pre-release access limited to just three hours before statistics are released, but it has no power to enforce these rules – something which PASC and the RSS would like to see changed.

Crisis in trust
In a 2010 statement supporting the UKSA’s position on pre-release access, former RSS president David Hand described levels of public trust in official statistics as “appallingly low”, citing ONS data which found that just one in five people thought that official figures were compiled without political interference. Hetan is a little more circumspect about this crisis of trust – in fact, he doesn’t think there is a crisis as such, but rather a gradual decline which mirrors a decline of trust in other public institutions.

“My view is that this is not really about statistics; it’s much deeper than that,” he says. “We live in a society that’s relatively individualistic as opposed to collectivist; there’s less deference towards institutions than there was 50 years ago; and this now cuts across everything, and in particular political institutions. I think statistics is really bound up within that. And it’s not clear to anyone what the route out is going to be.”

Despite this somewhat gloomy conclusion, Shah is positive about the role that Dilnot and his predecessor at the UKSA – Sir Michael Scholar – have played in improving confidence in statistics. He’s also positive about the increasingly active role of select committees in scrutinising statistics and supporting UKSA. In 2011, PASC played a key role in ensuring that it was Dilnot who succeeded the equally robust Scholar.

The government had originally named Dame Janet Finch as its preferred candidate to replace Sir Michael, but she withdrew her application after a PASC pre-appointment hearing in which it became clear that she and the committee had, in her own words, “differing views about how the job should be undertaken, and in particular how the independence of the chair should be exercised”. PASC member Kelvin Hopkins then sat on a new selection panel that chose Dilnot for the role. Shah doesn’t want to discuss personalities and individuals, but thinks “it’s a good thing that Parliament has taken ownership of this body so that it’s clearly not seen as part of this government. And my guess is that [PASC’s involvement in Dilnot’s appointment] would give the chair of the authority a mandate to be independent: that can only be healthy.”

Cuts and creativity
Like all other professions in government, statisticians are adapting to reduced budgets. Individual departments have cut specific research programmes, often amid protest from community groups, charities and local councils. When the communities department chose to stop its annual Citizenship Survey, even the national statistician Jil Matheson protested.

Shah is again circumspect about the impact of these cuts. “Obviously, we are concerned [about them],” he says – but society recognises that “austerity measures must cut across everything”. What is important, he continues, is that government is careful about the long-term effects of stopping surveys. “It’s a bit like cutting capital spending,” he says, “it gives you a short-term hit, a short-term fix, but in the longer term do you know where you are, what you’re doing?”

He also believes that government has “been thoughtful and creative” as it considers new ways to gather statistics, and he cites the ONS’s proposals on replacing the traditional, paper-based census as an example. The ONS launched a consultation last month setting out two options: that the census continues to run more or less as it currently does (once every ten years, covering the whole population), but be mainly based online; or that ONS instead use existing government data, supplemented with smaller annual surveys.

The RSS is reviewing the proposals for its own submission to the consultation, but Shah is sympathetic to the ONS’s position: it doesn’t have much room for budget-cutting, as many of its statistical surveys are mandated from the EU. He outlines three questions which will need to be answered when decisions on the census are made. First, what is the user need? Second, will the new information meet that need? And third – and perhaps less obviously – what are the wider risks around technology, legislation and public acceptance? Using administrative data to build a census would need new data-sharing legislation, for example – but the main risk he flags is securing public support amid “concerns about what the state is doing with the information it holds.”

This is a wider concern for the statistics profession, he continues: “Regardless of what happens to the census, clearly we’re moving in a direction where we’re going to use administrative data more, and that’s all to the good as long as privacy is taken very seriously.” But the public will need to be reassured that their data is being used appropriately and safely, for useful research.

This takes us back to the issue of trust in government and official statistics: people’s opinions of data gathering and sharing are often shaped by their opinions of the organisation doing the gathering, he says. Proponents of open data and transparency often argue that publishing more information about what government is doing will – eventually – lead to greater trust in government. But Shah suggests that people’s views of government are shaped by far wider factors than its use of statistics; and anyway, most people’s understanding of statistics isn’t advanced enough for data releases to challenge popular perceptions.

Shah’s a supporter of open data, and says it can be a powerful tool to improve accountability: he describes improving statistical literacy across the whole population as one of the RSS’s key aims. Yet he’s not optimistic that we will eventually reach a “statistical nirvana” of “a rational society governed purely by the evidence”.

That doesn’t mean one can’t take steps in the right direction, Shah adds. Returning to the question of evidence-based policy, he defends politicians’ right to overrule evidence – but only once they understand it. “It may well be that you can’t follow what the evidence tells you,” he says. “That’s the politician’s judgement: they’ve got a difficult job and we malign them rather too much, I think. But at least if they know what the evidence tells them, that’s a start.”