Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

601–610 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#601

Post by visarga »

throwaway_pdp09 wrote: post #595
edanm wrote: post #589

As a data engineer, I think blaming Excel for this is the wrong framing. The real problem here is with the entire process, and specifically, the lack of testing. Excel is great for many use cases, especially if you need people to enter data somewhere. Its UI is unparalleled in terms of quickly giving something to users that they can understand, mess around with, and verify. It's a very common use case to then need to…

I don't think data engineering is hard or needs much resources. It does need people who care and who will think about things. Anyone who comes at a problem with the mindset 'this is going to he hard' probably lacks experience and will throw big-data frameworks at it, really screwing things up. The most significant, and valuable, resource needed is thought first, and knowledge+experience second. All IMO anyway.

Depends on context, you're talking in general, but in particular as the GP post said, there can be pressure from management to rush things, different data formats, bad or missing data and so many other pitfalls. It's hard when it's hard.

You can write a simple test to check a function is working correctly, but how do you make sure your 100,000 item database doesn't have corrupted or missing data caused by the latest pipeline update, especially if the corrupted parts are rare?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#602

Post by corty »

sriram_malhar wrote: post #534
kelnos wrote: post #492

Earlier quoted context omitted.

Sure, and I think at least the headline ("ill-thought-out use of Excel") avoids blaming MS or Excel itself. Ultimately tools are built for particular things, and if you choose to use a tool for something it's not built for, and it breaks catastrophically, that's on you.

No. Excel is definitely to blame. A simple error "cannot import/export more than 65k rows" would have done the trick. Not silently throw away rows.

The premise of excel is very much "don't bug me, just do it". The highly praised UI simplifies away corner cases and errors intentionally. Summing over an empty cell? Most probably zero, why throw an error and confuse the user? The simplified UI makes correctness hard to impossible, so excel is just unsuitable for tasks where correctness is desired.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#603

Post by [deleted] »

jugg1es wrote: post #538

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#604

Post by corty »

not_kurt_godel wrote: post #584
iso8859-1 wrote: post #351

Earlier quoted context omitted.

Certificates are not applicable, the link scheme is HTTP.

It presents a self-signed certificate if using a browser/extension that automatically redirects to HTTPS (which everyone should be).

If the extension redirects to a broken cert not intended for public use, the extension is broken and should not be used. This is not making things safer, this is training users to click through warnings again. Very much "don't".

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#605

Post by visarga »

adwww wrote: post #387
jl6 wrote: post #315

It’s fun to point and laugh, but is this really just the consequence of an unprecedented health emergency demanding a rush job? As far as bugs go, it doesn’t sound that bad. They didn’t lose data - they just processed it late? And they spotted it within days/weeks, and have a workaround/correction already? And it’s only the reporting that was wrong, not the more important part where they inform people of results? I’d…

There's a test and trace system that is supposed to phone everyone with a positive test result and get details of all their close contacts (people they sat in an office with, restaurants they visited, etc), who are in turn contacted. A delay in the test results means the contact tracing could not happen, and people will have been going around spreading the virus they caught off the original, known cases. Also, the wh…

> A delay in the test results means the contact tracing could not happen, and people will have been going around spreading the virus they caught off the original, known cases.

A recent piece in The Atlantic argues that we got it all wrong, we should test backwards (who infected the current patient) instead of forwards.

This particularity is caused by the dispersion factor K which is a metric hidden (conflated) by the R0, meaning that the same R0 would be handled differently depending on the K factor.

A related suggestion was that we should focus more on super-spreaders as COVID doesn't spread uniformly. That's why the probability of finding a cluster is higher looking back than forward - most people don't get to actually spread the virus.

https://www.theatlantic.com/health/archive/2020/09/k-overloo...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#606

Post by Turukawa »

I've been teaching data curation for open data managers in governments around the world for the last decade. All the issues identified in this BBC article are covered in my syllabus.

If anyone's interested, here's the openly-licenced syllabus, in English and French: https://github.com/whythawk/data-wrangling-and-validation

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#607

Post by mFixman »

swiley wrote: post #528
totaldude87 wrote: post #491

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :( 5 – London Olympics Oversells Swimming Event by 10,000 Tickets 4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contra…

Is it really that hard to spend 5 minutes setting up a decent database? Use sqlite if you really want something simple. I guess you still have to understand locking (especially on distributed filesystems.) I've certainly seen people mess that up with spreadsheets.

Every time I see a problem like this I can't stop thinking that computer education is terrible everywhere.

We are able to teach almost everyone how to use complex software like Word and Excel. Why can't we teach people how to use a terminal, SQLite, or how to create a very simple Python script?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#608

Post by lukeramsden »

corty wrote: post #604
not_kurt_godel wrote: post #584

Earlier quoted context omitted.

It presents a self-signed certificate if using a browser/extension that automatically redirects to HTTPS (which everyone should be).

If the extension redirects to a broken cert not intended for public use, the extension is broken and should not be used. This is not making things safer, this is training users to click through warnings again. Very much "don't".

The extension is not at fault, all it does is rewrite HTTP links to HTTPS (which should be the default IMO, I agree with the parent commenter). The fault is with the site providing a self-signed certificate when accessed over HTTPS.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#609

Post by _pmf_ »

vertere wrote: post #599
sheetjs wrote: post #567

Earlier quoted context omitted.

The correct way to generate a CSV cell with a leading 0 is ="01" You can verify this with 01,"01",="01"

Perhaps by "correct way" you meant "dodgy hack to make Excel happy and risk breaking more sensible implementations"? Excel may predate the RFC but AFAIK MS didn't invent or coin the term CSV, so you can't just say whatever Excel does is correct. The RFC is loose because of nonsense like this, it doesn't mean it was ever a good idea.

Do you want to be a useless sophist or develop software?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#610

Post by mFixman »

jl6 wrote: post #315

It’s fun to point and laugh, but is this really just the consequence of an unprecedented health emergency demanding a rush job? As far as bugs go, it doesn’t sound that bad. They didn’t lose data - they just processed it late? And they spotted it within days/weeks, and have a workaround/correction already? And it’s only the reporting that was wrong, not the more important part where they inform people of results? I’d…

That has been the COVID experience in Britain in the last couple of months: the government is spending a ton of money in training new people and creating new processes, which is causing a ton of slowdowns and bugs.

In the grand scheme of things, having problems because of inexperienced people is much better than not having enough people like during April and May.

Post reply on HN