Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

221–230 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#221

Post by willtim »

jsty wrote: post #197
mjw1007 wrote: post #181

So, what we know is: « PHE had set up an automatic process to pull this data together into Excel templates [...] When [the row limit] was reached, further cases were simply left off. » The terrible thing here is dropping data rather than reporting an error and refusing to run. It isn't clear what piece of software was behind this "automatic process". Clearly the responsible humans are to blame. If the software that d…

> Is there anything in Excel itself that behaves like that? I'd imagine the default error-handling behaviour of 9/10 Excel macros is to throw away data

Yes and "on error resume next" is what I usually see in VBA code

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#222

Post by shmel »

freeone3000 wrote: post #143
sct202 wrote: post #128

Earlier quoted context omitted.

Did they pick XLS because it's easier to parse or something? I can't imagine why someone would pick a format that was largely replaced 10 years ago.

Moving from XLS to XLSX will break your macros and would require all machines to be running Office 2007 or later.

Office 2007 was like... 13 years ago, what's the problem?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#223

Post by addicted »

callamdelaney wrote: post #105
beagle3 wrote: post #85

This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

Except it wasn’t the government. They outsourced the test and trace system to private companies.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#225

Post by munk-a »

beagle3 wrote: post #85

This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…

Yea, I have a personal vendetta against XLSX due to having had to make tooling replacements for business logic being encoded in spreadsheets in the past, but the title of this article is incredibly slanted.

This was absolutely a user error and the title "Excel: Why using Microsoft's tool caused Covid-19 results to be lost" is really disappointingly click-baity.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#226

Post by vharuck »

sct202 wrote: post #128
codeulike wrote: post #125

Earlier quoted context omitted.

If you read the article (perhaps its been updated?), they were using the old XLS format and they hit the limit at 64,000 lines PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team as well as other government computer dashboards. The problem is that the PHE developers picked an old fil…

Did they pick XLS because it's easier to parse or something? I can't imagine why someone would pick a format that was largely replaced 10 years ago.

If somebody finds it easier to do this process using Excel than with a scripting language, I wouldn't expect them to know the differences between file formats. I would be surprised if they understood the concepts behind file formats.

They may have chosen XLS because they used Excel back when that was default, and now they don't want to "risk" anything by switching horses mid-stream.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#227

Post by [deleted] »

bitlevel wrote: post #190
TwoBit wrote: post #161

Earlier quoted context omitted.

It's hard to say from here what would be the best thing for them to use, but simply using a modern spreadsheet format instead of the ancient xls seems fine for interchange.

There's not much to stop the devs from using open source tools to create a PHP web frontend and db backend for this data - and secure it as well. Heck, if they still need to export, they could do that from the data too. Sure - use Excel for POC, but get that DB backend up pronto.

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#228

Post by smsm42 »

Was just reading the chapter about using Excel as a database in this book two days ago: https://www.goodreads.com/book/show/39074550-humble-pi

Apparently it's super common, which fills me with horror. But these guys managed to take it to 11 by abusing it in yet another novel way.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#229

Post by callamdelaney »

addicted wrote: post #223
callamdelaney wrote: post #105

Earlier quoted context omitted.

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

Except it wasn’t the government. They outsourced the test and trace system to private companies.

Public Health England was responsible for collating the data which came in the form of CSV's - PHE is a government agency.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#230

Post by [deleted] »

diarrhea wrote: post #210

Each test result creating several rows of data seems like a problem too. In clean data, every observation is one row. It makes working with the dataset much easier. In this scenario, I would expect one observation to correspond to one test result. The multiple rows are then better off pivoted into columns.

[deleted]
Post reply on HN