Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

31–40 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#31

Post by sgt101 »

pjc50 wrote: post #7

Professionalism has been entirely absent from the top-level handling of the pandemic. It's not entirely clear from the article who's responsible for this, but I note that the so-called "NHS" test and trace is in fact an outsourced organisation: https://en.wikipedia.org/wiki/NHS_Test_and_Trace run by Tory placewoman Dido Harding. > "She is a former chief executive of the TalkTalk Group where she faced calls for her to…

>She holds a board position at the Jockey Club, which is responsible for several major horse-racing events including....

I'll point out that Matt Hancock is MP for Newmarket - a centre for horse racing and associated businesses.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#32

Post by mhandley »

The article states that it is not known where in the country was affected, but I think we can deduce that. Here are the daily cases by specimen date. Dotted line is Saturday's data, solid line is Sunday's. The difference between the two is partly the normal daily update, but around 2/3 of it is correcting the underreporting:

7-day moving average: http://danger.handley.org.uk/misc/rates-uk-recent.png

Raw data: http://danger.handley.org.uk/misc/rates-uk.png

Looks like all regions were affected, but by far the largest corrections are in NW, NE and Yorkshire regions. In particular, NE had looked like cases were declining, but we can now see this was incorrect, and they're still increasing rapidly

Edit: note the most recent 3 days are always incomplete, so any decline shown there is not a real effect.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#33

Post by jiggawatts »

benaadams wrote: post #27
avian wrote: post #22

Earlier quoted context omitted.

> Using an Excel sheet as a database How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Columns!?

You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.)

Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow:

- Transposed data (new entries in columns)

- Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by computer).

- US date formats mixed in with non-US date formats.

- Each day in a separate file, often with as little as 100 bytes per file. Thousands upon thousands of files.

- Random comments or annotations in numeric fields (preventing graphing the data).

We don't all need to be data scientists, or machine learning wizards, or quantum computing gods.

But come on. This is the most basic, data entry clerk level Excel 101 stuff. Trivial, basic stuff.

Put rows in rows.

Don't mix random notes into columns you might want to sum or graph.

Don't mix units.

Have headers.

Use a sane date format.

That is all.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#34

Post by throwmeaway_pls »

chimprich wrote: post #26
Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

> It's unlikely this will have had much impact since most reported cases were only a few days late I'm not sure if you're being sarcastic or not, but in case you're not, this has meant c.16k people missed from having their contacts traced. That's time-critical. Delays of any number of days are very bad. People will likely die because of this error. This is safety-critical software. It should be stress tested and have…

The WHO estimated 750 million cumulative infections on Friday. This error is not unbelievable because the system is just a bit of political theater. There’s nothing more we can do against an endemic human coronavirus. It’s time to move on.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#35

Post by iso1631 »

social_quotient wrote: post #11

Wish we knew more details. Maybe helpful for people that don’t know... Xls excel has a 64k row limit. Newer xlsx has 1mm row limit. Or maybe they were putting people in by columns and hit the 16.3k column limit?

Or maybe it wasn't excel at all.

One number the UK government doesn't release is how many people are tested. They have spent the summer championing "testing capacity", which we found out at the start of September was a lie. They release the number of tests done, and the number of people returning positive. They don't release the number of people done -- if you're tested twice, that counts as two on the number of tests, but one on the number of people.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#36

Post by smcl »

spzb wrote: post #15
mytailorisrich wrote: post #12

Earlier quoted context omitted.

For those not in the UK, Dido Harding was already on front pages a couple of weeks ago when she was grilled by a Parliamentary committee about the shambolic Covid testing organisation and her reply was " no-one could predict such a strong demand for Covid tests "... As the NY Times said yesterday, sometimes it feels like " Britain is operating without adult supervision ".

She was also CEO of ISP TalkTalk when they had a massive personal data breach and told an investigation that she didn't know if the data was encrypted or not. https://www.msn.com/en-gb/money/other/people-want-to-know-wh...

In the anglosphere when you're in the managerial class, you fail upwards it seems.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#37

Post by [deleted] »

throw_m239339 wrote: post #14
pjc50 wrote: post #7

Professionalism has been entirely absent from the top-level handling of the pandemic. It's not entirely clear from the article who's responsible for this, but I note that the so-called "NHS" test and trace is in fact an outsourced organisation: https://en.wikipedia.org/wiki/NHS_Test_and_Trace run by Tory placewoman Dido Harding. > "She is a former chief executive of the TalkTalk Group where she faced calls for her to…

I mean let's call a cat a cat: this is cronyism thus corruption. Unfortunately, voters never care about these immoral practices. When a social system/administration is dysfunctional, corruption is often the first cause of these disfunctions. The result of corruption is always sheer incompetence.

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#38

Post by [deleted] »

jiggawatts wrote: post #33
benaadams wrote: post #27

Earlier quoted context omitted.

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Columns!? You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.) Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow: - Transposed data (new entries in columns) - Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by co…

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#39

Post by tannhaeuser »

daanlo wrote: post #16

We did a bit of work for laboratories this year and csv is not an uncommon exchange format between labs. In general almost all exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ at some point in the future. So seen in this context a csv or an excel file doesn‘t really surprise me and should probably also be seen in this context.

> exchange formats are text based, with labs saying they will upgrade to „modern xml formats“

Worth noting that XML is also a text format. SGML even can treat CSVs as markup. There's nothing wrong with CSVs/TSVs anyway - it's a concise tabular format using only minimal special coding for a record and a field separator, as envisioned by ASCII and EDIFACT. The problem seems more like that there was no error checking in place to capture file write errors, or more generally the use of non-reproducible, manual operating practices which seems common in data processing.

Post reply on HN