As churn rates fuel a new era of amnesia, Susan Allott asks if we can ever escape from Groundhog Day
In the spring of 2011, Kathy Melling was settling in for a late night in her office at the Department of Health, where she’d worked in a cross-government team of policy experts for over two years. The team – some seconded, some on fixed-term contracts – were being disbanded under the austerity measures of the new coalition government, and this was their last day. Melling had been working flat out for weeks, wrapping up as much as she could and archiving all the material her team had been working on. Late in the evening, a senior official popped her head into the office and politely suggested Melling ought to go home. Before she did so, Melling took a moment to contact her colleagues across the civil service with her personal email address, asking them to get in touch if they needed help taking forward the work she’d been responsible for.
“It was a frustrating time,” Melling says. She was aware that a smaller team of civil servants was to continue her work – which was still considered a priority – but there had been no formal handover. Within weeks of leaving, she received “a flurry of emails” from officials needing her input. They were starting from a point of almost no policy knowledge, and appeared to be replicating work from scratch. On her own time, Melling made several visits to meet the new team, and soon realised “they didn’t know where to find the digital resources we had created. If I hadn’t reached out to people before I left, leaving them my personal email, all that work would have been lost.”
Fast-forward more than a decade, as a new administration settles into office, and it seems a good moment to ask whether anything has changed since Melling’s experience in 2011. Is the civil service losing valuable knowledge and expertise as people move on? Are civil servants encouraged to seek out historic policy decisions to inform their work, and if so, do they know where to find them? Or could they be duplicating work and repeating mistakes, at huge cost to the public purse?
Most commentators approached by CSW agree there is a persistent problem with institutional memory in the civil service. The question of why, and what might be done to resolve it, is more complex.
“Government is not unique in struggling to manage this,” says Catherine Haddon, programme director at the Institute for Government, pointing out that any large organisation will encounter these issues. But the unique challenge faced by the civil service, Haddon suggests, is an unusually high rate of churn. And this will be exacerbated when a change of political party ushers in a new set of ministers.
“There is a changing of the guard that happens when you’ve had 14 years of one government,” Haddon says. “You can expect changes of permanent secretary in the coming year, and there has been considerable change in private office staff, people who stayed on to see through the new government but have now done their 18 months. There are risks around that.”
This movement at senior level is precisely the reason we need officials at lower grades to remain longer in post, argues a policy official who prefers to remain anonymous: “We have a permanent civil service, and the idea is that ministers churn and civil servants don’t. But in practice, civil servants move on to move up.” This policy, our source argues, is a significant block to institutional memory, which could be resolved with a new approach to pay. “It’s ridiculous that you can’t get promoted in your job. In the private sector they take the view that we don’t want to lose this person who is critical to the business. But in the civil service, you don’t get promoted in post.”
A recent IfG report – 20 ways to improve the civil service – also highlights the issue of churn, recommending that permanent secretaries should be held to account for meeting new staff turnover targets, which would include internal turnover within departments. Given that this recommendation is not directly addressed in the latest Civil Service People Plan, how can we best capture institutional memory before it walks out of the door?
“The memories of what’s important about an issue, and what was relevant to decisions, come from people’s personal recollections” Lord Butler
Robin Butler, who was cabinet secretary and head of the civil service from 1988 to 1998, tells CSW that the loss of most historical sections, and the advisers attached to them, was short-sighted. “It was the economising in the ’90s that led to the abandonment of historical sections in most departments other than the Foreign Office,” Lord Butler says. “The memories of what’s important about an issue, and what was relevant to decisions, come from people’s personal recollections. A historical adviser could refer you to someone who’s an expert on that episode.”
Clearly there have been huge changes to working practices since the 1990s, and the means by which we store and retrieve historic records is one of the changes that Butler would argue has not been uniformly beneficial. There are examples of excellence, such as the FCDO, which has both held on to its physical library and supplemented it with extensive digital resources. “Our intranet has an incredible treasure trove of old Foreign Office documents,” says Sam Navaratnam, team leader, Saudi Arabia. “We also have research analysts who provide historic analysis for us, and they sit in on our policy meetings and discussions and keep records of what we say and do. So that’s all kept for future reference.”
But the FCDO is an exception in this regard. And there is some evidence that the phasing out of physical libraries and historic advisers reflects a culture of short-termism. “When I joined the Treasury 20 years ago there was a library, but it wasn’t valued; nobody used it,” says a senior official, who has since moved departments. “I never had any incentive to give it any thought. It wasn’t laziness, it’s just that we were encouraged to focus on the here and now.”
Have historical perspectives been sidelined, as Butler suggests? Or can new approaches balance out the challenges that come with change? Some departments are taking steps to rebuild a historical perspective through partnerships with a team of academics at King’s College London. The Department for Business and Trade’s partnership will see King’s academic Emilia Braddon write a PhD on the history of the department, while perm sec Gareth Davies will support postgraduate learning through a visiting professorship. Launching the partnership, Davies said he wanted to capture the long history of trade departments and the Board of Trade. “This sense that someone’s previously been grappling with the challenges that we’re grappling with today is so important,” he said. “The context may have changed, technology may have advanced, social norms could have shifted, but I find that opening the files gives you an insight into how people were looking at an issue, what played out as expected, what didn’t; and, crucially, provides a new challenge to your implicit assumptions.”
The issue of how best to manage digital records remains a challenge, however. Most acknowledge that the paper-based approach to record keeping is a thing of the past, and that tapping into the memories of colleagues through verbal exchange is not always possible. Unfortunately, as Haddon points out: “There are plenty of knowledge-management systems out there, but Whitehall has never managed to adapt itself to a way of working that fits different documenting styes.” Some would argue that AI might be the answer, but Haddon is circumspect. “Generative AI is fallible, and it can corrupt as well as resolve issues,” she says.
In some respects, digital technology has brought real positives. Its arrival in the 1990s heralded a culture change, “driven by openness and a drive to share knowledge, which helps you retain institutional memory”, says David Mann, who was brought in by the Blair government as part of the drive to bring all government transactions online by 2005. He went on to become head of innovation for Direct.gov and helped to form the Government Digital Service. “One of the things GDS did well was that we worked in the open,” Mann says. “We encouraged the team to blog about our work as we were doing it. There was a mix of practitioners and leadership writing on the blog in a practical, useful way for other digital practitioners across government to learn from.”
Mann believes this practice of working in the open is only possible when officials feel able to share all stages of learning – including the acknowledgement of mistakes – with the permission and backing of leadership. “I’m talking about storytelling, the useful narrative behind a piece of work, writing this stuff down,” he says. “I’m worried that since 2016, there has been a tailing off of that transparency around government. If the civil service is in a defensive posture, which it has been over the past few years, then it’s not at the top of the to-do list to be open and transparent.”
Is the real issue a cultural one, rooted in a preference for secrecy and discretion over openness? Mann suggests this tendency ebbs and flows with different styles of leadership. “It could be that a lot of the practitioners of the more open style of working are no longer inside government; they will have taken that ethos with them,” he says.
Could it be that Mann and his team were not working on anything highly sensitive, hence their willingness to share? Haddon touches on this, explaining that concerns about Freedom of Information and inquiries such as Covid and Grenfell have made people nervy. “I don’t think there is a significant attempt to avoid scrutiny, but I don’t think people will be as candid as possible in a written record,” she says.
Our anonymous source thinks the problem goes deeper. “The civil service does not value expertise,” he says. “As a minister, you want a team under you who will do what you want, so if you’ve got people there who think they’ve seen it all before, they can become ‘no’ people – and ministers don’t like that.” This perspective echoes Melling’s experience back in 2011. “Our team was unpopular internally,” she says. “We were seen as know-it-alls.”
Is it fair to suggest that ministers are undermining the effort to draw on and record institutional memory, favouring a drive for new ideas and fresh approaches? Is there a view among senior officials that historical perspectives and policy experts are a drag on progress? Not always, but the message coming from ministers is vital, Haddon says: “If officials feel it’s fine to solve policy issues without looking back at the paper trail and then to move on to the next job, if that’s the way to have success, then they won’t value retaining their records.”
Despite these challenges, Haddon is clear that the issue is being tackled – partly because of the likelihood of future inquiries, and FoI. “The paper trail is incredibly important both politically and legally,” she says. But the drive for change is also coming from civil servants, many of whom value institutional memory and recognise its importance, both in the now and for future generations.
Heading up this challenge, the government’s knowledge and information management profession takes responsibility for record keeping and curating information in all its forms. These officials are working to champion the profession and make people aware of what it does. As deputy head of digital and information professions at the Ministry of Defence, Rebecca Dorsett told CSW in 2023, “if we use KIM professionals in the right way, my God they can help transform things across government”.
Given the moment of transformative change we are living through, it seems vital that senior leaders – especially ministers – set the tone in ensuring that institutional memory is preserved. Partly this is about an expectation of good record keeping. But it may also prove transformative to signal that policy advice should include lessons from the past as well as ideas for the future.
The KIM Profession
Knowledge and information management professionals are the custodians and ethicists of official information and records for the civil service.
According to the Civil Service People Survey, there are approximately 2,500 civil servants who declare themselves as aligned to the KIM profession. This will be made up of a mixture of professionals whose work is wholly related to the disciplines within the profession and others who carry out these functions locally within their business unit as part of their wider role (e.g. local information managers or a business unit’s FoI officer).
It is a requirement under the Freedom of Information Act 2000 that each department must have an information management function with a designated manager.
The work of the KIM professional varies from operational activity to policy advice and advice on how to meet the requirements of legislation and regulatory obligations. The civil service code requires civil servants to “keep accurate official records and handle information as openly as possible within the legal framework”.
The Code of Practice on the Management of Records issued under section 46 of the FoIA warns that “information can become a liability if it is not properly managed”.
The current KIM head of profession is Roger Smethurst, who works at the Cabinet Office. Smethurst is a fellow of the Information and Records Management Society and a fellow of the Chartered Management Institute.
Civil servants can contact KIM at: head-of-profession-gkim@cabinetoffice.gov.uk