As the civil service continues to tackle coronavirus – launching new services, transforming existing ones, and using data to understand this huge and novel threat – machine learning (ML) technologies could prove invaluable. But first, we humans must learn how to avoid the risks inherent in these machines.
Deployed intelligently, ML could dramatically improve the efficiency of civil service business processes, the quality of decision-making, and the government’s ability to tackle threats such as COVID-19. And in an era of rapid, disruptive change and fast-rising public expectations, it’s more important than ever that public services meet citizens’ needs. Poorly designed ML systems could lead to discriminatory outcomes, weaken case management and regulatory frameworks, infringe on citizens’ data rights, and undermine accountability and transparency. So civil servants are – rightly – proceeding with caution: the keys to progress lie in understanding the technology’s characteristics, then creating the frameworks and selecting the applications that will realise ML’s potential whilst addressing its weaknesses.
In essence, ML systems comprise algorithms that, supplied with large datasets, ‘learn’ how to perform a set task by identifying patterns, connections and common indicators. And there are many applications within the government’s work to address COVID-19. By combing through data on coronavirus test results, population mobility data from smartphones and social media feeds, for example, they could forecast where new outbreaks of the virus are likely to appear. And by examining government records on benefits claims, company liquidations, economic stimulus applications and VAT returns, they could identify the regions and industries suffering the greatest economic impact – informing the design and targeting of business support programmes.
There are thousands more potential applications within the civil service’s ongoing service delivery work – much of which must also be adapted around the requirements of coronavirus. As the government rolls out business support packages at high speed, for example, there are obvious risks around fraud: using data on previous grant schemes and fraud convictions, ML could pick out the behaviours common to fraudsters and point to current claimants acting in similar ways. Or fed with records on face-to-face and telephone enquiries, it could power chatbots able to settle service users’ most common problems – reducing the need for front desk staff and call handlers and thus supporting social distancing.
The risk of digital discrimination
Yet artificial intelligence can be as foolish as its biological counterpart – drawing the wrong conclusions, then applying them in every decision. This risk is greatest if the source data is incomplete or inaccurate, but ML can also mistake correlation for causation.
Given data on the engineering workforce and asked to pick out the characteristics that make for good engineers, for example, one ML system noted that most were men – and thus recommended avoiding women: it had ‘learned’ to be sexist. And if case management ‘training data’ containing evidence of discrimination is used to develop an algorithm – reflecting, for example, a bias against black men within a criminal justice system – the ML can itself begin making discriminatory decisions. These risks are particularly stark in the case of coronavirus, which disproportionately affects old people, men, low-income households, and ethnic minorities; we don’t yet know whether the causes are medical or social, but ML systems could potentially replicate any existing patterns of human discrimination within new digital processes.
It is essential that public sector decisions help address such inequalities, rather than reinforcing them. And tackling this problem demands more than accurate, complete datasets. Even the tightest historic case management dataset reflects the world as it is, rather than as it should be – so ML introduces a risk of hardwiring past patterns of discrimination and inequality into future decision-making.
The black box conundrum
Second, ML presents challenges around citizens’ privacy and data protection rights. The General Data Protection Regulations give individuals subject to ML decisions the rights to know how their data has been processed, and to demand that a human review their case; so transparency and accessible review systems are essential. More awkwardly, as algorithms evolve, their operation may become opaque even to their own programmers – creating a ‘black box’ system. How then can officials explain to a citizen exactly how their data has been processed?
This opacity creates a third challenge around transparency and democratic accountability: can ministers reasonably be held accountable for ML decisions made by black-box systems? If an individual decision is challenged promptly, programmers may be able to download the data and see how it was made. But the history of government contains many examples of flawed decisions that took years to come to light. Business owners need to be confident that, if an ML system is found to have been consistently making the wrong call, no great harm will have been done – and that the data exists to revisit and correct each of them.
Monitoring mutation
The mutable nature of ML also makes life difficult for regulators, which typically operate by approving a particular product, service or process: what happens when an approved ML algorithm evolves following regulatory scrutiny? Regulators are likely to need new ways of scrutinising ML and providing oversight – perhaps by approving the auditing and management processes governing a system, and by setting out principles with which the algorithm must comply. Although efforts are underway around the world – including the UK – to codify the standards underpinning ML and develop new approaches to regulation, we are some ways from rolling out those standards.
Addressing all of these challenges is made more complex by the scarcity and cost of specialist ML professionals. While the price of data storage and processing continues to fall rapidly, few government departments have the capabilities required to introduce ML in a big way: expertise is required not just within the digital team, but among business owners, project managers and commercial staff.
A way forward
So there are substantive barriers to the introduction of full-blown ML in many parts of the public sector. In our work with civil service organisations we have, however, developed ways of realising many of ML’s benefits without putting the technology in charge of decision-making – side-stepping many of the challenges around deployment.
One sensible first steppingstone on the path to ML is the introduction of advanced analytics and decision-support technologies: non-ML systems that comb through large datasets, generating insights that can be presented graphically on digital dashboards. Using fixed algorithms to guide civil service decision-makers, analytics technologies don’t create the transparency, accountability and regulatory challenges that accompany ML. Nor can they ‘learn’ to discriminate – though they can as easily provide poor guidance if datasets are inaccurate or contain evidence of past discrimination, so strong data and risk management policies are required.
Analytics tech allows civil servants to more easily examine and understand datasets. Rather than laboriously reconciling and poring over Excel documents to pull out conclusions, they can manipulate information in a controllable interface, interacting with visualisations and guidance created using a pool of trusted data. So analytics systems could, for example, combine and present COVID-19 test data from around the world, helping officials to identify risk factors or infection patterns. Applying predictive analytics techniques to historical data, civil servants could also forecast the impact of specific changes – supporting planning for events such as rising COVID-19 infection rates, interest rate falls and extreme weather conditions.
Advice, not decisions
The next step beyond analytics involves introducing ML in decision-support roles – so rather than automating decisions, the ML algorithm provides advice and guidance to a public servant. This approach enables civil service bodies to tap into the power of ML technologies while retaining clear lines of accountability, heading off many of the challenges involved in fully automated ML processes.
Nonetheless, even in this context ML requires careful management: when working with public sector clients, we’re careful to introduce suitable auditing, oversight and review processes, designing systems that address the inherent risks. And whether civil service organisations are using pure analytics or decision-support ML, they’ll need to ensure their data is fit for purpose: both technologies run on the fuel of data – and for good results, they need five-star.
In the next article, we’ll examine the capabilities essential to effectively implementing ML in decision-support roles, and explore the processes, systems and techniques required to realise its potential while avoiding its risks. In time, ML could transform the operation of government, much improving both the services received by citizens and the targeting of resources. And during our current crisis, it could prove invaluable in protecting citizens and businesses from this awful virus. But deploying it safely is as dependent on organic as on artificial intelligence.
Steve Thorn is Executive Director and Richard Shreeve is Technical Director for central government at global public sector software leader Civica.
This is the first of two articles on machine learning. In the second article, Civica explains why managing the development and operation of those systems carefully is crucial to realising their full potential for the benefit of public services.
To contact the Civica team about any enquiries please click here