Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

51–60 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#51

Post by adam-a »

londons_explore wrote: post #5
Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Sure, we techy types prefer a proper database with a web frontend and an API, but that requires significantly more skill to build than an excel file. No database has the flexibility and flat learning curve that excel has. Until a database manages to meet these goals, it remains a good tool for certai…

This is a very good point and a shame to see it being downvoted. I have been wishing there was an accessible database-ish software for doing my accounts, as the current mega-spreadsheet I use is pretty hairy and none of the specialised apps quite fit the bill. The closest thing seems to be Airtable, which while nice, is quite expensive.

Recently I was visiting hospital quite a bit (ante-natal) and witnessing the horror of the massive NHS form filling software the nurses and midwives have to use. It's clunky and sprawling but one of the main sins I think is it's lack of adaptability. In the old days of pen an paper, a new form could be written, or adapted, and photocopied easily, and on-site. Now if a new field needs to be added or a process is changed it needs to go (I imagine) to some centralised IT development office and fed into a ticketing system where it might get changed in a few weeks or months. There's no room to quickly adapt at a ground-level, so you end up with things like this excel sheet problem.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#52

Post by fuzzy2 »

Mvandenbergh wrote: post #46
chimprich wrote: post #26

Earlier quoted context omitted.

> It's unlikely this will have had much impact since most reported cases were only a few days late I'm not sure if you're being sarcastic or not, but in case you're not, this has meant c.16k people missed from having their contacts traced. That's time-critical. Delays of any number of days are very bad. People will likely die because of this error. This is safety-critical software. It should be stress tested and have…

I was being serious. Distributionally, most of the cases were 1-2 days before the error was spotted. To be clear though, when I said "not much impact", I didn't mean that people might not die because of it, only that it wouldn't affect the large scale functioning (such as it is) of the tracing operation.

> Distributionally, most of the cases were 1-2 days before the error was spotted.

That’s always true though if the virus is spreading exponentially (dunno whether it currently does in the UK).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#53

Post by Mvandenbergh »

pjc50 wrote: post #7

Professionalism has been entirely absent from the top-level handling of the pandemic. It's not entirely clear from the article who's responsible for this, but I note that the so-called "NHS" test and trace is in fact an outsourced organisation: https://en.wikipedia.org/wiki/NHS_Test_and_Trace run by Tory placewoman Dido Harding. > "She is a former chief executive of the TalkTalk Group where she faced calls for her to…

While that is true, I have a little difficulty imagining that Harding or anyone else senior was involved in the process of shoveling around spreadsheets. As the overall leadership she obviously has the accountability but just saying, "oh what do you expect when you put a crony in?" ignores the reality that a number of people on an operational level had been doing this for months and nobody seems to have stopped it.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#54

Post by hef19898 »

jiggawatts wrote: post #33
benaadams wrote: post #27

Earlier quoted context omitted.

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Columns!? You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.) Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow: - Transposed data (new entries in columns) - Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by co…

Oh, I saw excel files that contained screen shots of other excel files over cells containing data. Comments where in text fields, if you were lucky, or again screen shots, from word.

And then the same people wondered why things based on this file didn't work.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#55

Post by poooogles »

brigandish wrote: post #10
DanBC wrote: post #6

Earlier quoted context omitted.

I agree, but I'm genuinely glad they're not[1] using fax machines. [1] they almost certainly are using fax somewhere.

It's probably Japan but that would be the least of the egregiousness. [1] Japan's COVID-19 Reports - 140KBs of Unadulterated Incompetence https://stdio.sangwhan.com/wtf-japan-covid-19-report/ [2] The previous HN discussion https://news.ycombinator.com/item?id=22728674

I get a cert warning on https://stdio.sangwhan.com/wtf-japan-covid-19-report/ which really is a good start.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#56

Post by yodelshady »

benaadams wrote: post #27
avian wrote: post #22

Earlier quoted context omitted.

> Using an Excel sheet as a database How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Worth noting the primary source for that claim is the Daily Mail.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#57

Post by OJFord »

pjc50 wrote: post #8
OJFord wrote: post #3

Overblown, IMO - only the useless absolute figure is affected, so it just creates another reason it's useless. Unless positive result files are more likely to exceed the file size limit than negative ones, of course, which I haven't heard suggested is the case. Edit: Perhaps I should clarify that I'm not suggesting it isn't a cockup. I just don't see it as a massive scandal that renders the data useless - it just dec…

Care to explain why you think the absolute figure is useless?

What are you going to use it for?

Mostly it seems to be used for comparing differently populous countries.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#58

Post by lini »

Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

Excel has a limit of ~1M rows per sheet. Total cases in the UK until now are ~500K. So it seems they use 2 rows per case in the sheet.

Edit: oops, it seems they used columns for the cases and the ~16K column limit was hit when daily cases exceeded that.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#59

Post by ajsnigrutin »

Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

Once you have a couple of thousands of rows in excell files, one file per day, possibly one per testing center.... possibly with inconsistent structure.... non standard date formats (even different in each row),... add to this the inconsistent namings ("Tested at" column with data: "Testing center 1, London", "TC1, London", "TC1 London", "TC1 LONDON",....), etc...

Yes, anything-sql or whatever would be better, but deciding between putting all the data into sql by hand, or "uhm... i'll think of something and do it tomorrow", me, being lazy, I'd pick the second option,...

...until i'd get shamed on here. Maybe even a few days after.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#60

Post by sails »

benaadams wrote: post #27
avian wrote: post #22

Earlier quoted context omitted.

> Using an Excel sheet as a database How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

A question, was it possibly a csv?

ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership.

(Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)

Post reply on HN