Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

671–680 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#671

Post by isolli »

totaldude87 wrote: post #491

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :( 5 – London Olympics Oversells Swimming Event by 10,000 Tickets 4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contra…

And how about this one: an Excel error was the cause for much unnecessary austerity after the 2008 financial crisis, costing potentially tens (hundreds?) of billions in lost output.

https://www.nytimes.com/2013/04/19/opinion/krugman-the-excel...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#672

Post by gpmcadam »

shadowgovt wrote: post #163
BuyMyBitcoins wrote: post #83

I love how Excel has become a de facto database - even though it absolutely shouldn’t be.

The first rule of software is "If it works, it'll get used." This is why people's email inboxes double as their TODO lists.

It doesn't even necessarily have to "work" (as in, work well) it just has to feel like it's the path least resistance.

You can work in a department using Excel for this type of data collection and reporting, and you can continuously suggest to your superior that something more mature could be used. They would probably agree. As would the entire team!

But if the team of analysts are only properly trained in this system and it's all they know coupled with a huge backlog of cases and time pressures, then they're going to keep using the thing that causes least headaches in the short term. And that might be objectively worse to everyone involved but they just keep ploughing through.

Toxic culture, bad management, poor working practises and external pressures can force even the most sane of people to choose the worst technology on the basis that they perceive it as "saving time" in the short term, even when they know full well they're borrowing Peter to pay Paul, they still do it.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#673

Post by jbergens »

EpicEng wrote: post #548
swiley wrote: post #528

Earlier quoted context omitted.

Is it really that hard to spend 5 minutes setting up a decent database? Use sqlite if you really want something simple. I guess you still have to understand locking (especially on distributed filesystems.) I've certainly seen people mess that up with spreadsheets.

Sure, I'll just teach tell my wife to learn SQL and some devops while she's at it. After that I'll coach her on how to get buy in from management on this new system while she neglects her primary duties. Throwing up a database and integrating it into a workflow/system isn't something anyone can just get up and do. I have to imagein you know that.

Was it really a manual process? I read it as some system importing info automatically.

And if it was manual I am surprised that Excel did not complain about adding more than 65000 rows (or saving more than 65000 rows as XLS). If a user gets a warning about possible data loss they should investigate more.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#674

Post by account42 »

corty wrote: post #602
sriram_malhar wrote: post #534

Earlier quoted context omitted.

No. Excel is definitely to blame. A simple error "cannot import/export more than 65k rows" would have done the trick. Not silently throw away rows.

The premise of excel is very much "don't bug me, just do it". The highly praised UI simplifies away corner cases and errors intentionally. Summing over an empty cell? Most probably zero, why throw an error and confuse the user? The simplified UI makes correctness hard to impossible, so excel is just unsuitable for tasks where correctness is desired.

Interpreting an empty cell as 0 is making an (often correct) assumption about what the user wanted.

Throwing away data without warning is almost certainly never what the user wanted.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#675

Post by reportgunner »

Maha-pudma wrote: post #585
jugg1es wrote: post #538

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

Every single problem I've ever had with excel stems from excel trying to be too clever. Nearly all my problems with importing data into it could be solved if excel left everything as is until I asked it to format values a certain way.

Well Excel trying to be clever stems from people not trying to be clever. We truly reap what we sow.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#677

Post by reportgunner »

jugg1es wrote: post #538

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

The right way to open CSVs in Excel is: Data -> From Text -> Browse to your csv -> Delimited -> Comma -> Next -> Select Text formatting for columns with leading zeroes -> Finish

You can either send xlsx with format or csv without format. If this would be disabled then we'd have another group of people complaining that their dates from CSV are not parsed.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#678

Post by reportgunner »

theshrike79 wrote: post #639
corty wrote: post #600

Earlier quoted context omitted.

Disagree on the "users can verify" part. Excel is very hard to verify, because of corner cases like date conversion. Formulas and macros, maybe spread out all over multiple sheets in hidden away white-on-white cells, make things very opaque and impossible to verify for most users. After a long gone colleague has passed the usage of some excel file, it is treated as an arcane and unchangeable wisdom from the past, nev…

I vaguely remember a discussion about this and the problem boils down to Excel sheets being impossible to unit-test. There might be, for example, a formula going through 20k lines in column F. But the one on line 1138 has a typo and the formula references an incorrect cell. No human will ever go through all the lines to check. Excel itself doesn't check stuff like that. And there are no tools for it either.

Excel actually checks that and gives you an 'Inconsistent formula' warning with a little green corner similar to when you comment a cell.

All you need to do is scroll through the worksheet you have just made.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#679

Post by konjin »

TeMPOraL wrote: post #627
konjin wrote: post #527

Earlier quoted context omitted.

Pretty common for finance in the 00s and 10s to be honest. It's slowly getting better, but until you fire everyone who isn't on board with this and replace them with people who can at least script - at every level of the business - you're going to have the same problem.

That would be good. But to be honest, Excel is scripting - in a 2D FRP REPL (something structurally more advanced than most programmers are using day to day). If we're changing hiring requirements to test for bash/Python/Powershell competency, why not actually test for Excel competency? Maybe that's the problem? Excel is so easy to start with that people with no experience think they've mastered it, and the industry…

Programming language expressions represent arbitrary trees, which are strictly more powerful than finite dimensional tables.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#680

Post by corty »

lukeramsden wrote: post #608
corty wrote: post #604

Earlier quoted context omitted.

If the extension redirects to a broken cert not intended for public use, the extension is broken and should not be used. This is not making things safer, this is training users to click through warnings again. Very much "don't".

The extension is not at fault, all it does is rewrite HTTP links to HTTPS (which should be the default IMO, I agree with the parent commenter). The fault is with the site providing a self-signed certificate when accessed over HTTPS.

All it does is assume that a https site serves the same content and audience as the corresponding http site. That is a broken assumption. The consequences of such a broken assumption are very much the fault of the extension.
Post reply on HN