Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

551–560 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#551
post #475

Earlier quoted context omitted.

Okay, so you write a script to take the data from your internal server and post it somewhere that Excel can access. You don't use Yahoo.

How about don't use excel in the first place?

Still a very long way from that. What tools should non-programmers use instead?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#552
post #496
post #442

Earlier quoted context omitted.

>>>I've personally cleaned up after people like you, Yep... The critical question that always gets blank stares is "How is this data and calculations validated" There is never an answer to that question from the ExcelMaster, it is always a variation of "it looks correct to me" Wonderful....

Except you can do it if you know what you're doing. No one is saying to manage millions of records etc. But you can get more mileage out of excel than people think - and people who half understand it are the worse because they know enough to know that it doesn't work. Custom solutions are great - but aren't always the answer and with proper process Excel will work just fine in many use cases.

How can you achieve anything equivalent to unit testing? How can you do any kind of change control (unless you treat every edit to the sheet as a change)? Excel does some things well but for anything that needs to work robustly as part of a process (as opposed to one-time exploratory data analysis) it lacks things that were basic in 1996. It's an "unsafe at any speed" situation: it may work as long as you do everything right, but it's missing basic safeguards as soon as you make a single mistake; I'd put it in the same category as using a memory-unsafe language.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#553

Earlier quoted context omitted.

> Not to mention the "system" solution will have to go through its own debugging and will undoubtedly have bugs in it for some time after initial release. Doing code reviews in Excel is hard if the developers are pathologically disciplined. It's impossible most of the time. And so is debugging. It's very unlikely any such system would face these limitations and would silently ignore data the same way Excel did.

> It's very unlikely any such system would face these limitations and would silently ignore data the same way Excel did. I’ve seen plenty of production systems that ignore or hide errors. Sometimes they’re still logging them, but it just goes to some log store or file that the team doesn’t check until their customers or support team inform them that it’s broken. Good practice? No. But there are plenty of ways to mess…

I'm sure that a developer who's determined enough can mess up any system. And Excel is by no means the only system with that kind of dangerous default (I remember MySQL 4 had very similar behaviour, as did early versions of PHP). But the industry knows this is far from best practice.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#554

So the limit for Excell is 65,000 rows for the old format, and one million-plus rows for the new format (from the article). Then I wonder, is there any tool that mimic Excell but with Sqlite as the backend? The limit of rows in Sqlite is 2 raised to the power of 64 (18446744073709551616 or about 1.8e+19). https://sqlite.org/limits.html

Microsoft actually has a reasonable migration path where you can use Excel backed by Access and then transition to Access backed by SQL Server. But you'd have to recognise that you had a problem first.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#555
post #437
post #376

Earlier quoted context omitted.

> formal double-blind QA passes I don’t know what you think I have in my unit tests, but it’s almost certainly not whatever you mentioned. Using excel as an alternative to CSV is fine. It’s not like anyone was doing anhthing complicated here. Just storing data.

>Using excel as an alternative to CSV is fine. Clippy the Paperclip Hi, it looks like this alphanumeric constant is the date format used by the Democratic Peoples Republic of Arstotzka for the year 127 BC. I have changed the encoding for the file to Arstotzka standard and overwritten the original.

Clippy doesn’t have to worry about COVID cases because Arstotzka probably executes them all

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#556
post #410

Earlier quoted context omitted.

Why are there infrastructure costs for a database server? Once you have one it can host all your small databases.

Developer salaries are the main cost.

The problem with my Private sector Australian company is everything IT related is aggressively outsourced through various networks of contractors.

In the old days when the relevant business application was written, hosted and supported in house there was a clear chain of responsibility I could pick up the phone and I'd have a direct line to the person who "owned" the application.

Nowadays if there is a problem it's pick up the phone talk to helpdesk get assigned a ticket number and get the buck passed between different teams, The database team will blame the server team, server team will blame the networking team, networking team will reply to ticket with 'looks ok no problem on my end' and the ticket will get closed without resolution.

From what I can tell there are a bunch of incentives in the support contract around how quickly support can close out tickets, so rather than trying to fix the problem support try to do everything they can to farm the ticket off to someone else so it won't impact their metrics. The whole thing feels maddening and has to be rather inefficient.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#557
post #528

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :( 5 – London Olympics Oversells Swimming Event by 10,000 Tickets 4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contra…

Is it really that hard to spend 5 minutes setting up a decent database? Use sqlite if you really want something simple. I guess you still have to understand locking (especially on distributed filesystems.) I've certainly seen people mess that up with spreadsheets.

Do note that anything other than Excel, UI/UX wise, failed.

When people wants a “database” they fire up Excel, start punching in numbers, solar calculators next to keyboard, and use eyeballs to search for strings.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#558

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

Quote your strings. The problem is more with how you generate CSV (which is easy to break in the first place) than Excel’s fault. It’s like writing your identifiers as numbers in JSON instead of string then complain the leading zeros are removed.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#559
Most people don't know about solve order and possible multiple solves for a given cell on any update.

From : http://www.decisionmodels.com/calcsecretsc.htm

When a cell in a spreadsheet refers to another cell it must be finally calculated after the cell it refers to. This is called a Dependency.

Excel recognizes dependencies by looking at each formula and seeing what cells are referred to. See Dependency Trees for more details of how Excel determines dependencies.

Understanding this is important for User Defined Functions because you need to make sure that all the cells the function uses are referred to in the function arguments. Otherwise Excel may not be able to correctly determine when the function needs to calculated, and what its dependencies are, and you may get an unexpected answer. Specifying Application.Volatile or using Ctrl/Alt/F9 will often enable Excel to bypass this problem, but you still need to write your function to handle multiple executions per calculation cycle and uncalculated data.

Post reply on HN