‘Excel-gate’ highlights need for ‘quality technical capability’ in NHS

This week’s “Excel-gate” has highlighted the need for “good quality technical capability” within organisations, experts have said.

Using Microsoft Excel is not an uncommon “workaround” when managing data but organisations need to find a “more robust” way of doing so, they said.

It comes as reports suggest the use of Excel may have caused nearly 16,000 cases of Covid-19 to go unreported between 25 September and 2 October. Public Health England (PHE) confirmed some 15,841 cases were left out of daily figures.

Speaking to Digital Health News, Dr Marcus Baw, GP and clinical informaticist, said the current “defacto workaround” for collating data from multiple sources is to export it into Excel.

“We’re going through the growing pains of society realising it’s not enough to just have a system that works at NHS Digital or works at Public Health England, you actually need to be able to transport data appropriately between those organisations and that’s where the problem has arisen,” he said.

“What we are faced with is that a lot of our organisations don’t have the internal technical capability to build good solutions to problems like this.

“That’s the thing that we really need to learn from this, and I think Excel-gate could be seen as a good thing, as a turning point where we start to realise the necessity for good quality technical capability within organisations.”

Adrian Byrne, chair of Digital Health’s chief information officer (CIO) network, said Excel is a good analytical tool but is not well suited for data collection.

“Excel is fundamentally an analyst tool that can be used by amateurs and professionals alike,” he told Digital Health News.

“It tends to break down if you use it for large, complicated data collection processes.”

Baw suggested organisations should focus on upskilling the workforce to better handle data rather than continuing to use Excel without learning from the problem.

“What we really need to do is look at what the problem is, this moving data around, and conceive of a better way of doing it that’s more robust,” he said.

“We don’t need centralised data platforms, we don’t need everything to be done centrally, what we need to do is upskill all the people that work within these organisations.

“There are times when Excel is the right solution, but we also need a toolkit to contain tools that are useful when Excel is not the right solution.”

The error which may have led to thousands of cases to go unreported is believed to have been caused by the file format PHE developers had opted to use.

They reportedly opted to use an old file format known as XLS, which can only handle about 65,000 rows of data. When the spreadsheet reached its maximum size it prevent new cases from being added to the file.

XLS dates back to 1987 before being replaced by Microsoft with XLSX in 2007, which is capable of handling more data.

Byrne said Excel is “perfectly good” at managing large amounts of data, especially in the XLSX format, but it doesn’t offer the same control as a bespoke database.

“What you shouldn’t do in Excel, even in the latest version, is use it to store or otherwise manipulate your datasets because it becomes very difficult to do a record-by-record or feild-by-filed audit on that data,” he said.

“There’s a fundamental difference between how Excel works and how a database works which is why, when you’re holding large amounts of data, you really don’t want to do that in Excel.”

He explained databases permit more “record-by-record” control, which allows for better management of the data it holds.

Baw added that Excel is “quite standard, but a very broken workflow” used by multiple organisations across the country.

“There will be a lot of focus on blaming Public Health England for this when we know that every organisation is doing it, it’s just that Public Health England got caught,” he said.

“That’s not what we need to focus on at all, what we need to focus on is increasing that technical capability across all organisations in the country.”

‘Technical debt’

The misreporting of Covid-19 cases has highlighted the need to address technical debt in organisations, providing a potentially necessary prompt to look at ways to better run technical capabilities.

Baw suggests every organisation, from NHS to local government, should have access to experts in computer programming, whether they’re employed full time or on a freelance basis.

“We do need to start addressing that technical debt. That means focusing less on delivery at a quieter time and going back and reviewing problems in your technical stack and fixing them,” he said.

“We need management to recognise what technical debt is and to prioritise it just as much as other operational priorities.”