Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

281–290 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#281
post #253

Earlier quoted context omitted.

Umm, who is qualified to write programs in any programming language, if both programmers and developers frequently have errors in their code? https://cve.mitre.org/cve/

That exactly right, noone is. Thats why noone uses C++ or javascript in safety critical devices Even in 'normal' industry its is a huge problem and we need safer languages, i.e. Rust (merits to be seen)

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#282

Earlier quoted context omitted.

Who? Almost anyone who wanted to get something done without going through the red tape of working with IT. Even in a private corporation that red tape is enough to make many a things end up in excel (or for the more tech savvy, access). Government is surely even worse (better?) at having red tape so the pressure was likely stronger.

Yup. What also helps is that the data literally sits on your drive, or your network drive. No SaaS vendor is holding it hostage.

It doesn't feel like that. Even if I save the file (365 subscription and desktop apps) on my disk it steel feels 'cloudy'.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#283

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

Add to this the understanding that getting this on contract will take a few months, when the boss was asking for these figures yesterday...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#284
post #251

Excel is great but I think at a certain level folks should start using Python and Pandas. The challenge is understanding when and getting the right resources to the right people. https://training.talkpython.fm/courses/move-from-excel-to-py...

The conceptual leap from Excel to Python+Pandas is pretty huge. Especially if someone hasn't done programming before. That is part of the reason I created https://www.easydatatransform.com .

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#285

Always use checked pre- and post-conditions in a data pipeline. This simple post-condition would have caught this issue: The sheet after merge operation must have a number of rows equal to the sum of number of rows for all merged sheets. Assuming this is a merge of a standardized input, then another post-condition might be: The number of columns in output shall equal the number of columns in the input. Might want to…

Even just within my own sheet I always build in some sort of idiot-check, the assumption being that I’m the idiot.

Do a SUM and match it against a COUNTIF on another column, or something. It doesn’t really matter what it is but if the data is important at all, I always scatter little checks throughout.

The case described in this article sounds like someone who just doesn’t know how to use Excel. I mean why in the name of VLOOKUP would you override the default and choose to save as .xls? That was a conscious decision. Anyone worth their salt knows that’s stupid.

Excel is not to blame here.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#286

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system".

The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentication module built in (a file sent from a government mail address). There's a risk of user errors, but the system will be working immediately[0]. Deployment can be done by anyone, as it's just clicking on File->New and starting to type data in.

Meanwhile, a "pigsqueal" solution will take half a year to design, produce and deploy (that's in an emergency, a year otherwise). And that's with competent IT people, and not someone who wants to profiteer off the crisis. Then you'll have to train the users, and hope the reality won't necessitate any changes, because they will take a while.

I think plenty of decent coders appreciate the fact that Excel is suitable for a surprisingly wide range of tasks, and that if you spot it widely used somewhere, it most likely means there isn't any comparable alternative available.

--

[0] - Note that the user error that finally happened did not happen in the place where you'd expect it to.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#287

I hate solving problems and not knowing how I did it, so this particular experience has stuck in my craw. Trying to develop a budget to pay off debts, my partner made this elaborate Excel spreadsheet and the output was that basically she had no spending money and I had very little, until one or both of us got a raise. It was far more austere than either of us were willing to go. So I started over using a different eq…

the secret to using Excel well in large complex tables/spreadsheets is the same secret as coding with large complex programs.

You use structure, design away the complexity, implement constraints, error checking and tests.

Note it's true that an inexperienced person is liable to make spaghetti code either way, but there's nothing fundamental about a spreadsheet format that makes it inherently unusable for a lot of small/medium problems. Indeed it even has benefits in terms of interactivity/ turn around/accessibility.

Of course, there's also a lot of problems with Excel and reasons not to use it like a database or anything which fundamentally relies on maintaining data integrity, and it's liable to be the first tool reached for by the non-experienced, who will generally make a mess of things large and complex as a rule.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#288

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

Come on, this is a government department of a first world country. They have databases for all sorts of things. They're not using excel for any other NHS databases, or the terrorism watchlist. There are plenty of people capable of making a good decision here.

I think you let them off too easily by just assuming they're dumb. This a bad decision by people who definitely should have known better.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#289
post #197

Earlier quoted context omitted.

> Is there anything in Excel itself that behaves like that? I'd imagine the default error-handling behaviour of 9/10 Excel macros is to throw away data

Oh, my God. "On Error Resume Next" is one of my earliest memories of learning to program for Windows. It never occurred to me that it might result in deaths.

https://github.com/ReactiveX/RxJava/wiki/Error-Handling-Oper...

Lots of love for it in Rx

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#290

I hate solving problems and not knowing how I did it, so this particular experience has stuck in my craw. Trying to develop a budget to pay off debts, my partner made this elaborate Excel spreadsheet and the output was that basically she had no spending money and I had very little, until one or both of us got a raise. It was far more austere than either of us were willing to go. So I started over using a different eq…

I wrote this a decade ago and have been using this for all of my personal and business spending: https://zetabee.com/cashflow/

It's not automated but it can help future income/expenses/balance in an Excel-like UI.

Post reply on HN