Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

41–50 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#41

Post by cblconfederate »

jiggawatts wrote: post #33
benaadams wrote: post #27

Earlier quoted context omitted.

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Columns!? You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.) Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow: - Transposed data (new entries in columns) - Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by co…

lets assume that local centers people do save them in rows, which is the most logical thing to do. But probably the software where they copy/paste them requires the data in columns, so maybe they transposed the data without realizing that it was truncated to max size.

Or else it wouldn't explain why it took so many days to realize that they couldn't add more columns.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#42

Post by jimmcslim »

I'm wondering if there could be an opportunity for an open-source contract tracing system, similar to the allReady [1] application ? Maybe something backed by a graph database such as Neo4J [2]

Looking at the allReady repo, maybe it isn't a great example since it hasn't been touched in years...

[1] https://github.com/HTBox/allReady

[2] https://neo4j.com/

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#43

Post by roel_v »

glandium wrote: post #19
throw_m239339 wrote: post #14

Earlier quoted context omitted.

I mean let's call a cat a cat: this is cronyism thus corruption. Unfortunately, voters never care about these immoral practices. When a social system/administration is dysfunctional, corruption is often the first cause of these disfunctions. The result of corruption is always sheer incompetence.

OT: are you French? AFAIK, calling a cat a cat is a French idiom. The English equivalent would be calling a spade a spade.

Also Dutch

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#44

Post by arethuza »

smcl wrote: post #36
spzb wrote: post #15

Earlier quoted context omitted.

She was also CEO of ISP TalkTalk when they had a massive personal data breach and told an investigation that she didn't know if the data was encrypted or not. https://www.msn.com/en-gb/money/other/people-want-to-know-wh...

In the anglosphere when you're in the managerial class, you fail upwards it seems.

We have a meritocracy apparently, so if you have got to the top you must be good and therefore your talents shouldn't be wasted.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#45

Post by jasiek »

pjc50 wrote: post #24
DanBC wrote: post #13

Earlier quoted context omitted.

> People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Test and Trace is a multi billion pound programme of work, involving huge companies (eg, Serco) interfacing with national governmental bodies (Public Health England). We've had months to get something in place. One of the reasons organisations like Serco is used is their apparent expertise with IT.

The main reason Serco is used is their political connections; they really ought to have been banned from consideration after the fraud scandal: https://www.bbc.co.uk/news/uk-50806919 (my one-step programme for improving UK public contracting would be that after a significant failed contract the contracting company and its directors and any other companies they are directors of would be banned from tendering for a per…

Unfortunately the procurement rules currently in place prevent procurement personnel evaluating a tender from taking into account any prior performance. I'm unsure whether this is a result of EU legislation and common market rules, or whether it is just the UK.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#46

Post by Mvandenbergh »

chimprich wrote: post #26
Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

> It's unlikely this will have had much impact since most reported cases were only a few days late I'm not sure if you're being sarcastic or not, but in case you're not, this has meant c.16k people missed from having their contacts traced. That's time-critical. Delays of any number of days are very bad. People will likely die because of this error. This is safety-critical software. It should be stress tested and have…

I was being serious. Distributionally, most of the cases were 1-2 days before the error was spotted. To be clear though, when I said "not much impact", I didn't mean that people might not die because of it, only that it wouldn't affect the large scale functioning (such as it is) of the tracing operation.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#47

Post by summner »

OJFord wrote: post #3

Overblown, IMO - only the useless absolute figure is affected, so it just creates another reason it's useless. Unless positive result files are more likely to exceed the file size limit than negative ones, of course, which I haven't heard suggested is the case. Edit: Perhaps I should clarify that I'm not suggesting it isn't a cockup. I just don't see it as a massive scandal that renders the data useless - it just dec…

Actually not only that. It affected contact tracing, and people exposed to infected patients were not informed to self isolate. Possibly leading to extra spreading of the virus.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#48

Post by mellosouls »

avian wrote: post #22
Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

> Using an Excel sheet as a database How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".

I noticed this as well, and started worrying the various news items that were citing Excel since had been reading HN!

I still haven't seen an authoritative source, or one that predates that comment.

But it definitely sounds possible.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#49

Post by Mvandenbergh »

londons_explore wrote: post #5
Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Sure, we techy types prefer a proper database with a web frontend and an API, but that requires significantly more skill to build than an excel file. No database has the flexibility and flat learning curve that excel has. Until a database manages to meet these goals, it remains a good tool for certai…

Sure. That's why I said it wouldn't bother me if this was in the first few weeks of the pandemic or for one-off ad-hoc analysis. I actually use Excel for that all the time, including for tasks where technically a db is the correct solution. Working is better than perfect and everyone knows (sort of) how to use Excel. To still be using it in production this many months in is not acceptable. Someone should have taken that Excel based workflow and set it up to run through a database, they had the time to do it.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#50

Post by mellosouls »

benaadams wrote: post #27
avian wrote: post #22

Earlier quoted context omitted.

> Using an Excel sheet as a database How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Actually from The Daily Mail, and not clear where they are getting it from.
Post reply on HN