Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

511–520 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#511

Post by konjin »

gxs wrote: post #493
konjin wrote: post #422

Earlier quoted context omitted.

The same way that engineers saying you shouldn't use dung to build houses are guilty of architectural elitism. Excel is the wrong tool because it mixes presentation and data, to the point that geneticists had to change the names of genes so their spreadsheets would stop turning them silently into dates and then destructively changing the original data[0]. >Excel works great - I've personally built some very complicat…

And I've cleaned up after many, many shitty engineers, one example being engineers who want to use a sledge hammer to drive a finishing nail and again, that's just one example. And if you want to get personal to that level they all the same know it all personality who think they write MUCH better code than they actually do. Pretty sure they cost companies even more money. Luckily for me, I have the practical sense to…

Imagine asking people to use the custom solution sledge hammer of python, git, unit tests and sqlite. Not one off the shelf piece of software there.

There comes a point where using a spork to drill holes in walls is not the right solution. That you're shitting on people pointing out the obvious tells me you're still stuck in the 90s windows developer mindset.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#512

Post by curiousllama »

jl6 wrote: post #315

It’s fun to point and laugh, but is this really just the consequence of an unprecedented health emergency demanding a rush job? As far as bugs go, it doesn’t sound that bad. They didn’t lose data - they just processed it late? And they spotted it within days/weeks, and have a workaround/correction already? And it’s only the reporting that was wrong, not the more important part where they inform people of results? I’d…

You’re right - but the biggest issue is clearly one of experience. Someone was put into a critical role they weren’t prepared for - a position in which their failures cost lives. They don’t have a redundant process, or error checking, or even someone familiar with spreadsheets.

That’s the scandal: it’s the type of basic error that it screams “they’re not handling this well.” You wouldn’t tolerate your new SWE colleague asking “What’s an array again?”

If publishing the whole set of data was delayed by excel crashing - fine. But silent failure because they ran out of columns? Cmon...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#513

Post by jeltz »

ashtonkem wrote: post #259
tannhaeuser wrote: post #39

Earlier quoted context omitted.

> exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ Worth noting that XML is also a text format. SGML even can treat CSVs as markup. There's nothing wrong with CSVs/TSVs anyway - it's a concise tabular format using only minimal special coding for a record and a field separator, as envisioned by ASCII and EDIFACT. The problem seems more like that there was no error checking in…

I feel like text-based is colloquial shorthand for "schemaless" or "dynamically typed".

CSV is not schema less and it does not have to be any more dynamically typed than XML either.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#514

Post by tetrahedr0n »

aschatten wrote: post #237
daanlo wrote: post #16

We did a bit of work for laboratories this year and csv is not an uncommon exchange format between labs. In general almost all exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ at some point in the future. So seen in this context a csv or an excel file doesn‘t really surprise me and should probably also be seen in this context.

There is nothing wrong with CSV or text base formats. You can even use AWS Athena to query CSV feels stored in S3. It's a good format for data import/export, that many systems can natively understand or have tools to parse, given it's known how to interpret data. It was a data pipeline issue. Software has little to do with it. If they received data in json and tried to interpret it as CSV, the same could have happene…

I agree.

Tools exist, for analysts and engineers (MS Access comes to mind for the analyst, python for the engineer), that would rectify the problem. And I think it's a fair assumption to say that those tools would be readily available.

Kinda sounds like a management issue, as well. No one ever said "hey you know XLS doesn't support all of this data"?

What a mess.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#515

Post by logicOnly »

tomc1985 wrote: post #503
bobbylarrybobby wrote: post #494

Earlier quoted context omitted.

The issue with is excel is that its failure mode is suboptimal. Programming languages will generally crash at runtime when an unrecoverable error occurs (index out of bounds, memory allocation failed, null pointer, etc). But excel just throws away your data?

Indeed, you have to manually inspect results to verify correctness. If you're lucky maybe your error will result in one of those weird error values being shown instead, but way too often that is not the case Not to mention that complex formulae are still usually expressed as a bunch of gobbledygook in the cell value textbox, which is about as easy to parse as minified Javascript. And that's to technical users like ou…

I had Excel error out when I tried to open 10 million rows.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#516

Post by gxs »

konjin wrote: post #511
gxs wrote: post #493

Earlier quoted context omitted.

And I've cleaned up after many, many shitty engineers, one example being engineers who want to use a sledge hammer to drive a finishing nail and again, that's just one example. And if you want to get personal to that level they all the same know it all personality who think they write MUCH better code than they actually do. Pretty sure they cost companies even more money. Luckily for me, I have the practical sense to…

Imagine asking people to use the custom solution sledge hammer of python, git, unit tests and sqlite. Not one off the shelf piece of software there. There comes a point where using a spork to drill holes in walls is not the right solution. That you're shitting on people pointing out the obvious tells me you're still stuck in the 90s windows developer mindset.

The problem is you only hear about botched excel jobs - you never hear about all the successful projects, so when you hear a horror story you just jump to the conclusion excel sucks.

Ok, ask a geneticist to build what they need with their “off the shelf” parts you mentioned vs 90% of the time being self sufficient with excel and allowing them to, you know, do genetics work. (The other 10 percent being where it makes sense to build something more custom).

Reminds me of the classic HN post when Dropbox had just launched where the user exclaimed “this is just a mounted blah blah using subversion blah anyone can do it” (paraphrased)

That old Dropbox post summarizes this mindset quite well.

And if being practical with your head not in the clouds making sane business AND technical decisions is what the 90s were like, man I missed out.

I guess at the end of the day we’re arguing over where you draw the line regarding when to stop using excel and graduate to something different, a blurry line that at the end of the day is a judgement call.

Oh and for the record I wasn’t even arguing against or for this particular use case in this post - speaking more in general terms.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#517

Post by sueders101 »

logicOnly wrote: post #515
tomc1985 wrote: post #503

Earlier quoted context omitted.

Indeed, you have to manually inspect results to verify correctness. If you're lucky maybe your error will result in one of those weird error values being shown instead, but way too often that is not the case Not to mention that complex formulae are still usually expressed as a bunch of gobbledygook in the cell value textbox, which is about as easy to parse as minified Javascript. And that's to technical users like ou…

I had Excel error out when I tried to open 10 million rows.

Excel tops out a little more over a million rows without specifically handling of larger files. I’m not sure excusing it, just a heads up to all the junior analysts out there who haven’t dealt with the problem(not implying you’re a junior analyst, just wanted to point something out to those who aren’t familiar with the issue).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#518

Post by tsjq »

totaldude87 wrote: post #491

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :( 5 – London Olympics Oversells Swimming Event by 10,000 Tickets 4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contra…

and one more:

"Scientists rename human genes to stop Microsoft Excel from misreading them as dates. Sometimes it’s easier to rewrite genetics than update Excel" https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#519

Post by thedudeabides5 »

totaldude87 wrote: post #491

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :( 5 – London Olympics Oversells Swimming Event by 10,000 Tickets 4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contra…

Well, not exactly like non-Excel systems are bulletproof either.

Entire Japanese stock market went down last week, and it's not like there was a flood of people on HN bemoaning that. At least with Reinhart and Rogoff et al you have a responsible party.

As opposed to 'nameless machine failed, and nameless backup machine also failed, and now it's in JIRA so don't worry about it'.

https://www.nytimes.com/2020/09/30/business/tokyo-stock-mark...

The glitch stemmed from a problem in the hardware that powers the exchange, said the Japan Exchange Group, the exchange’s operator, during a news conference. The system failed to switch to a backup in response to the problem, a representative said.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#520

Post by konjin »

gxs wrote: post #516
konjin wrote: post #511

Earlier quoted context omitted.

Imagine asking people to use the custom solution sledge hammer of python, git, unit tests and sqlite. Not one off the shelf piece of software there. There comes a point where using a spork to drill holes in walls is not the right solution. That you're shitting on people pointing out the obvious tells me you're still stuck in the 90s windows developer mindset.

The problem is you only hear about botched excel jobs - you never hear about all the successful projects, so when you hear a horror story you just jump to the conclusion excel sucks. Ok, ask a geneticist to build what they need with their “off the shelf” parts you mentioned vs 90% of the time being self sufficient with excel and allowing them to, you know, do genetics work. (The other 10 percent being where it makes…

I worked with a couple. They stop being geneticists rather quickly because the salary for someone who can use python well is about 4 times that of someone working on genes. Keeping them away from coding is now an executive decision since university departments can't pay competitive salaries.

The dropbox post was right. Knowing someone who wins a lottery is no reason to conclude spending all your money on lottery tickets is a good investment. Which is as much as I said in that thread when it happened. That they were solving a problem for idiots comes with the problem that idiots are too stupid to realize when a problem is solved, so you need to focus on looking like you've solved it instead. Funnily enough dropbox spent herculean amounts of effort on polishing the UI.

>And if being practical with my head not in the clouds making sane business AND technical decisions is what the 90s were like, man I missed out.

Spoken like someone who has done neither.

Post reply on HN