Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

341–350 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#341

Post by strbean »

sjansen wrote: post #307
strbean wrote: post #293

Earlier quoted context omitted.

Couldn't you achieve exactly what was needed here with Airtable? Or, easier yet, Google Sheets and a Google Form?

Probably not while complying with privacy laws. I would be much more upset to find out a governemnt employ on a deadline used Google Sheets or Airtable to share my medical data instead of an Excel doc on a secure government file server.

Ah, HIPAA.

Having looked at the marketplace for HIPAA compliant solutions for educational therapy patient record keeping (patient CRM, whatever the industry term is), I'd feel so much safer having my data in the hands of Airtable than the absolute dumpster fire that was every offering I saw.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#342

Post by [deleted] »

Reason077 wrote: post #329
happytoexplain wrote: post #296

Earlier quoted context omitted.

Frankly I think you're being optimistic/pessimistic (depending on your point of view). A person can be a "decent coder" by very reasonable definitions of the term and still not realize that Excel can't handle this particular volume of data.

Excel can handle the volume of data just fine. The (very outdated) .xls file format with its 65,536 row limit was the problem here.

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#343

Post by xupybd »

gxs wrote: post #334
TeMPOraL wrote: post #286

Earlier quoted context omitted.

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…

Not to mention the "system" solution will have to go through its own debugging and will undoubtedly have bugs in it for some time after initial release. On top of that, there is STILL room for human error as a user is still the one actually inputting data and designing the experiment/data flow. I'm going to get mauled on this forum given the audience, but parent comment reeks of the technical elitism on this forum an…

Yep Excel is a solid choice for some systems. Business people know it and it's a hard sell to tell them you want more time / money to use something they see giving them less.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#344

Post by dllthomas »

airstrike wrote: post #314
croes wrote: post #280

Earlier quoted context omitted.

Nothing wrong? Excel autoconverts values if it thinks it knows the data type of a column and Excel treats 1900 as a leap year. Genetic researchers changed the names of genes to prevent auzoconveesion errors.

You can find idiosyncrasies about any tool. Excel is a fine tool for 99% of use cases. Sure, it's quirky here and there, but the sheer amount of work that gets done globally in Excel should be testament enough to its value. Highlighting edge cases to bash on the tool is either myopic or intellectually dishonest

Let's consider three claims:

    1) X delivers tremendous value.
    2) X has some significant problems.
    3) X has no problems.
Obviously 1 and 3 are compatible. I claim 1 and 2 are also compatible (and, actually, not uncommon...).

You said that "[t]here's nothing wrong with Excel", and someone responded with an example of a problem that they considered significant. If we read "there's nothing wrong with Excel" as a strong claim of 3, then that's obviously a refutation of your claim. You could argue that the specific problems are not actually significant enough to rise to the level of notice. You could argue that they are not, in fact, problems at all. You could argue that you didn't, in fact, mean to make claim 3 in any strong sense (which I think is what was actually going on here - that's valid, English works that way).

You've instead interpreted it as a flawed refutation of claim 1 (and took the opportunity to demean your conversation partner). I don't think that's productive.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#345

Post by srtjstjsj »

TeMPOraL wrote: post #286
lordnacho wrote: post #247

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…

Excel never works for math, at any scale. It's a constant trashfire of "looks fine" until it shows a result someone doesn't want, and then an error is found that changes results and the cycle repeats.

Excel is nice for visualizing and browsing data you already have, and informally searching and sorting for hypothesis generation.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#346

Post by sheetjs »

nickff wrote: post #333
nabla9 wrote: post #322

The most consequential Excel error was The 2010 Reinhart-Rogoff error in the paper "Growth in a Time of Debt" https://www.nber.org/papers/w15639 The paper claimed that average real economic growth declines 0.1% when natonal debt rises to more than 90% of gross domestic product (GDP). When you correct the error it shows 2.2% average increase in economic growth. Paul Ryan used it in the US for Republican budget proposa…

I think you're overstating both the paper's impact, and the significance of the error. Most such papers are used to support existing policy preferences, not drive them. In addition, the error didn't reverse or erase the correlation, it diminished it, and removed the inflection point from the curve.

It was literally cited by Britain's finance minister George Osborne as justification for UK government austerity:

> As Ken Rogoff himself puts it, "there's no question that the most significant vulnerability as we emerge from recession is the soaring government debt. It's very likely that will trigger the next crisis as governments have been stretched so wide."

https://web.archive.org/web/20100414205630/http://www.conser...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#347

Post by testplzignore »

bencollier49 wrote: post #133

Good lord. Old XLS files. I'd love to have seen how this was concocted.

It may have been a user education and/or communication issue.

I had someone come to me recently asking for some data "in an XLS". I asked them if they specifically need XLS or if they just need something that can be read by Excel. It was the latter, but they didn't know that there was a difference. To some people, XLS == Excel. Sort of like how TLS is still being referred to as SSL - somewhere in the world there is a developer being asked to use SSL for a new project because SSL == secure.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#348

Post by strangeattractr »

nickff wrote: post #333
nabla9 wrote: post #322

The most consequential Excel error was The 2010 Reinhart-Rogoff error in the paper "Growth in a Time of Debt" https://www.nber.org/papers/w15639 The paper claimed that average real economic growth declines 0.1% when natonal debt rises to more than 90% of gross domestic product (GDP). When you correct the error it shows 2.2% average increase in economic growth. Paul Ryan used it in the US for Republican budget proposa…

I think you're overstating both the paper's impact, and the significance of the error. Most such papers are used to support existing policy preferences, not drive them. In addition, the error didn't reverse or erase the correlation, it diminished it, and removed the inflection point from the curve.

The most salient point of the paper was the inflection point of 90% for sharply lower growth. It was and still is used by proponents of fiscal austerity to argue against fiscal stimulus.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#349

Post by politelemon »

jl6 wrote: post #315

It’s fun to point and laugh, but is this really just the consequence of an unprecedented health emergency demanding a rush job? As far as bugs go, it doesn’t sound that bad. They didn’t lose data - they just processed it late? And they spotted it within days/weeks, and have a workaround/correction already? And it’s only the reporting that was wrong, not the more important part where they inform people of results? I’d…

They had a budget of GBP 36 million. It's pretty bad.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#350

Post by Smoosh »

ImaCake wrote: post #305
jonplackett wrote: post #288

Earlier quoted context omitted.

Come on, this is a government department of a first world country. They have databases for all sorts of things. They're not using excel for any other NHS databases, or the terrorism watchlist. There are plenty of people capable of making a good decision here. I think you let them off too easily by just assuming they're dumb. This a bad decision by people who definitely should have known better.

Yep. If the UK is anything like Australia, they will have a small army of contractors who specialise in building SQL databases for various important and pointless requirements for the government. These contractors are used to working on giant, sensitive data, databases with insufficient allocated time. It should have been pretty much trivial for the UK govt to solve this.

And yet, I have first-hand witnessed a scenario like the one described in an AU government department. A small system developed by business (one person who had subsequently moved on) using Excel and Word and macros to keep track of information and generate letters to "customers" had become an essential tool, but unsupported and undocumented. When we (IT) recommended redeveloping it in Java with an SQL database (and documentation and testing and backups), they balked at the cost. Partly because the original person did the work "in their spare time", so the business never saw the actual costs. And partly because there are zero infrastructure costs with Excel and Word. There was very much a feeling that IT massively inflate the costs and will build a Rolls Royce when a Ford will do. It was a house of straw vs house of bricks situation, but they were happy with their straw house, they thought their only problem was that the little pig who built it was no longer there.
Post reply on HN