Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

141–150 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#141

Post by shadowgovt »

gridlockd wrote: post #119
beagle3 wrote: post #85

This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…

The way I heard it, it was an Excel problem: They had a CSV file that had more than 1 million lines, and Excel dropped the lines beyond 1 million. I can imagine an automated process that appended to the CSV and then Excel saving back to that same file with the lines stripped. I hope we can all agree that this is not the way Excel or any other software should behave, whether the user is incompetent or not.

If the conversion was failing silently, it's a Microsoft software problem, but I bet you money the conversion was failing in a way that fell into a `catch(Exception e) { }` somewhere in their conversion script. Or even a `catch(Exception e) { log e; }`, and the log is stored to local storage on a machine that no human being ever logs into.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#142

Post by codeulike »

bencollier49 wrote: post #133

Good lord. Old XLS files. I'd love to have seen how this was concocted.

Perhaps some old integration code from 10+ years ago that was written to spit out XLS rather than XLSX

e.g. some old versions of Crystal Reports can only do XLS. There must be loads old systems around that can only do XLS

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#143

Post by freeone3000 »

sct202 wrote: post #128
codeulike wrote: post #125

Earlier quoted context omitted.

If you read the article (perhaps its been updated?), they were using the old XLS format and they hit the limit at 64,000 lines PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team as well as other government computer dashboards. The problem is that the PHE developers picked an old fil…

Did they pick XLS because it's easier to parse or something? I can't imagine why someone would pick a format that was largely replaced 10 years ago.

Moving from XLS to XLSX will break your macros and would require all machines to be running Office 2007 or later.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#144

Post by gridlockd »

sct202 wrote: post #128
codeulike wrote: post #125

Earlier quoted context omitted.

If you read the article (perhaps its been updated?), they were using the old XLS format and they hit the limit at 64,000 lines PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team as well as other government computer dashboards. The problem is that the PHE developers picked an old fil…

Did they pick XLS because it's easier to parse or something? I can't imagine why someone would pick a format that was largely replaced 10 years ago.

Compatibility. It's reasonable to predict that some of the participating entities are running software that is at least a decade old.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#145

Post by iso1631 »

pjc50 wrote: post #8
OJFord wrote: post #3

Overblown, IMO - only the useless absolute figure is affected, so it just creates another reason it's useless. Unless positive result files are more likely to exceed the file size limit than negative ones, of course, which I haven't heard suggested is the case. Edit: Perhaps I should clarify that I'm not suggesting it isn't a cockup. I just don't see it as a massive scandal that renders the data useless - it just dec…

Care to explain why you think the absolute figure is useless?

It's not much good for comparrison with March when tests were only done in hospital, and I see a lot of people doing that

Most people who had covid by August (which was at least 6%, or 4 million, in the UK according to antibody tests) caught it between start of February and end of April. That's at least 3.5 million over 90 days, or 40k a day on average. Peak was likely double, maybe even treble that, given lockdown on 23rd of March dramatically cut infections.

We're likely testing somewhere in the region of 50% of actual cases. There's the non-symptomatic cases where about equal to the number of symptomatic cases, so double the absolute number -- some with symptoms will refuse to be tested because they don't want to miss work, some without will be caught by contact tracing, those two groups probably cancel each other out.

As such I'd expect 50k/day to be the "March equivelent" - with doubling every 10 days that means another 2 or 3 weeks.

It's not just about absolute cases though. Cases have been doubling roughly every 10 days, as have deaths. Deaths lag cases by 2-3 weeks, so I'd expect deaths in 20 days to be continuing to double even if we all stayed in an isolated booth from now.

Ultimately the concern is we're heading into flu season when hospitals are stretched, and cases, hospital admissions, and deaths are all increasing.

Hopefully flu season will be milder due to social distancing and due to more vulnerable people having been killed off, but either way we need to get a grip soon. Or just abandon any pretense of trying to stop its spread.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#146

Post by mark-r »

sct202 wrote: post #128
codeulike wrote: post #125

Earlier quoted context omitted.

If you read the article (perhaps its been updated?), they were using the old XLS format and they hit the limit at 64,000 lines PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team as well as other government computer dashboards. The problem is that the PHE developers picked an old fil…

Did they pick XLS because it's easier to parse or something? I can't imagine why someone would pick a format that was largely replaced 10 years ago.

Inertia, most likely. They were using XLS 10 years ago and never saw a reason to change.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#147

Post by ivanstame »

My mind is blown how can you do any real change in the world without good data? We need exact classification of the data of dead people. If they had COVID, was it the main source of death, real cause of death, was it diagnosed(tested) or not...ect. Until then, this stuff about COVID is total bullshit and you are being deceived and you are being to scared. SNAP OUT OF IT!

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#148

Post by pfortuny »

ivanstame wrote: post #147

My mind is blown how can you do any real change in the world without good data? We need exact classification of the data of dead people. If they had COVID, was it the main source of death, real cause of death, was it diagnosed(tested) or not...ect. Until then, this stuff about COVID is total bullshit and you are being deceived and you are being to scared. SNAP OUT OF IT!

Compare to the flu... So many deaths of people with the flu are NOT flu deaths...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#149

Post by n4r9 »

bencollier49 wrote: post #133

Good lord. Old XLS files. I'd love to have seen how this was concocted.

My guess: somewhere down the line someone who doesn't often use Excel typed ".xls" at the end of a string variable or selected it from a dropdown list and made a mental note to come back to it if it caused any issues in testing.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#150

Post by langitbiru »

So the limit for Excell is 65,000 rows for the old format, and one million-plus rows for the new format (from the article).

Then I wonder, is there any tool that mimic Excell but with Sqlite as the backend? The limit of rows in Sqlite is 2 raised to the power of 64 (18446744073709551616 or about 1.8e+19).

https://sqlite.org/limits.html

Post reply on HN