Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

421–430 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#421

Post by WrtCdEvrydy »

chasd00 wrote: post #383
WrtCdEvrydy wrote: post #122

Earlier quoted context omitted.

Alternatively... "We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it"

>"We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it" you have no idea how right you are :)

Been there, done that, got the sqlite t-shirt...

Wasn't a bad choice, but once you load 20TB into it, you realize you done goofed.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#422

Post by konjin »

gxs wrote: post #334
TeMPOraL wrote: post #286

Earlier quoted context omitted.

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…

Not to mention the "system" solution will have to go through its own debugging and will undoubtedly have bugs in it for some time after initial release. On top of that, there is STILL room for human error as a user is still the one actually inputting data and designing the experiment/data flow. I'm going to get mauled on this forum given the audience, but parent comment reeks of the technical elitism on this forum an…

The same way that engineers saying you shouldn't use dung to build houses are guilty of architectural elitism.

Excel is the wrong tool because it mixes presentation and data, to the point that geneticists had to change the names of genes so their spreadsheets would stop turning them silently into dates and then destructively changing the original data[0].

>Excel works great - I've personally built some very complicated models that work just fine.

I've personally cleaned up after people like you, which cost the business millions of dollars because they didn't realize how unqualified they were for doing their jobs.

>Goes through the same process of QA and line-by-line checking of requirements.

Somewhat difficult as excel is not line based. "Oh yes, these constants are on a different sheet, if you just click here, here, here and here you can see where we get the original data, we multiply it by 1.0 to make sure it's not a date over there and then we feed it back ..." if excel was sanely serializable you could easily version control it and see meaningful diffs between check ins. Double points if you could run tests on sheets and run them in batch mode easily (I said easily).

[0] https://interestingengineering.com/genes-renamed-to-stop-mic...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#423

Post by TeMPOraL »

konjin wrote: post #418
TeMPOraL wrote: post #402

Earlier quoted context omitted.

Back in the day, it wasn't "very good", at least not in terms of getting it to work. That RK4 in Excel happened because I got extremely annoyed at Octave.

Peering into my crystal ball: You were between 16 and 22, you were 'great with computers', you had never used an OS other than Windows. There is a difference between a calculator and a type writer and there is a difference between an office os and one for programming. Using Windows and blaming Octave for Windows sucking is a rite of passage for everyone in a BSc program. I hope you got better and switched to a Unix.

Holy shit, how much does it cost to rent out your crystal ball for a few hours? Because you got it 80% right.

I was 19 or 20 back then, and Windows was my main OS - though I did have some experience with Linux as well (running several distributions for desktop use, as well as working with Cygwin and SFU on Windows), and I was a relatively proficient C++ programmer, having spent ~6 years of pretty much all my after-school time coding game engines.

So it's not that I couldn't make it work - I eventually did. But it was so rough around the edges that I gave up in frustration twice.

And yeah, these days, Linux is my daily driver (well, technically Emacs - Linux distros are just various flavors of Emacs bootloaders for me).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#424

Post by ACow_Adonis »

bradford wrote: post #338
ACow_Adonis wrote: post #287

Earlier quoted context omitted.

the secret to using Excel well in large complex tables/spreadsheets is the same secret as coding with large complex programs. You use structure, design away the complexity, implement constraints, error checking and tests. Note it's true that an inexperienced person is liable to make spaghetti code either way, but there's nothing fundamental about a spreadsheet format that makes it inherently unusable for a lot of sma…

> You use structure, design away the complexity, implement constraints, error checking and tests. I've never thought of Excel as an ideal tool for any of these things. I'm struggling to think of how it would have change-verifications/tests in the same way that software projects do. (I'm certainly open to the possibility that I'm ignorant/unaware on this subject)

depending on your point of view, you don't do them in the "same" way (I.e with separate unit tests or compiler enforced safety), and explicit difs are hard/impossible. And ideal is dependent on the task (it's ideal for me to get a working interactive graphic + visualisation to a user behind a corporate firewall via email in the same afternoon I get the request, I wouldn't actually choose to do 'programming' in Excel, but I consider a budget small and simple and programming is probably overkill)

what you do is much closer to old-school low level programming. define the relationships between your tables and variables well, set up explicit corresponding arrays of 1s and 0s that are themselves error checks on the underlying structure/ contents of your tables, calculate things in two different spots/ways and verify equivalence holds, and use simple red/green conditional formatting to draw attention to when things fall outside of expected state, etc.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#425

Post by outworlder »

Many comments already talked at length on use-cases for Excel, so I won't go for a 'me too'.

Instead, I just want to share this video from Joel Spolsky, aptly titled "You Suck at Excel".

Turns out, most people do suck at Excel. Myself included.

https://www.youtube.com/watch?v=0nbkaYsR94c

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#426

Post by michaelt »

srtjstjsj wrote: post #374
Smoosh wrote: post #350

Earlier quoted context omitted.

And yet, I have first-hand witnessed a scenario like the one described in an AU government department. A small system developed by business (one person who had subsequently moved on) using Excel and Word and macros to keep track of information and generate letters to "customers" had become an essential tool, but unsupported and undocumented. When we (IT) recommended redeveloping it in Java with an SQL database (and d…

Why are there infrastructure costs for a database server? Once you have one it can host all your small databases.

Paying your share of having a unix sysadmin and a DBA available 24/7.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#427

Post by rossdavidh »

While I am not in the habit of defending Microsoft, I think it should be pointed out that it wasn't using "Excel" that was the biggest problem, but using "13-years obsolete format for Excel". Not that there was any reason to be using Excel for this, but it's a bit unfair to blame Microsoft for something they never claimed XLS could do, and provided a much better capacity in XLSX 13 years ago.

Again, it's hard to cry too many tears for Microsoft, but it does seem a bit off-target to blame "Excel" for this...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#428

Post by ccktlmazeltov »

TeMPOraL wrote: post #286
lordnacho wrote: post #247

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…

> that's rock solid (so no need to unit test anything)

that is crazy talk, the amount of user error that can happen in an Excel spreadsheet means that you would need to do crazy macro work in order to maintain the integrity of the data

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#429

Post by m-p-3 »

mordae wrote: post #398
ksenzee wrote: post #363

Earlier quoted context omitted.

> Excel never works for math, at any scale. That is patently untrue. I've borrowed and lent money based on Excel spreadsheets, more than once. That's small-scale math with real-life consequences, and Excel is an excellent tool for the job.

Couch GnuCash Cough

You're right, but if you asked me to do something in GnuCash which I never used or do the equivalent in Excel with formulas I'm already familiar with, I'd go with Excel in a pinch.

This is why Excel is so well known, because once you get the hang of it, it becomes a great tool for something small.

It scales poorly though.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#430

Post by ccktlmazeltov »

gxs wrote: post #334
TeMPOraL wrote: post #286

Earlier quoted context omitted.

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…

Not to mention the "system" solution will have to go through its own debugging and will undoubtedly have bugs in it for some time after initial release. On top of that, there is STILL room for human error as a user is still the one actually inputting data and designing the experiment/data flow. I'm going to get mauled on this forum given the audience, but parent comment reeks of the technical elitism on this forum an…

I'm sorry but you seem to not understand what data integrity is if you think Excel is a perfectly acceptable solution here.
Post reply on HN