Missing Covid-19 test data was caused by the ill-thought-out use of Excel • Hacker News Board

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#411

Post by askvictor » Mon, Oct 05, 2020, 10:08 PM UTC

ImaCake wrote: post #305

jonplackett wrote: post #288

Earlier quoted context omitted.

Come on, this is a government department of a first world country. They have databases for all sorts of things. They're not using excel for any other NHS databases, or the terrorism watchlist. There are plenty of people capable of making a good decision here. I think you let them off too easily by just assuming they're dumb. This a bad decision by people who definitely should have known better.

Yep. If the UK is anything like Australia, they will have a small army of contractors who specialise in building SQL databases for various important and pointless requirements for the government. These contractors are used to working on giant, sensitive data, databases with insufficient allocated time. It should have been pretty much trivial for the UK govt to solve this.

Victoria (Australia), until a few weeks ago, did contact tracing via hand-written notes and faxes. Sure, it's not going to overflow at 16,000-odd cases, and doesn't require a DB admin and test cases, and all of that. But it performs at a fraction of the speed, and is one of the major reasons the second wave here got out of hand. Just because that army of contractors exists doesn't mean the relevant departments know of it's existence, or decide to use it.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#412

Post by lordnacho » Mon, Oct 05, 2020, 10:08 PM UTC

TeMPOraL wrote: post #391

lordnacho wrote: post #385

Earlier quoted context omitted.

> Not to mention, the entire financial industry That IS the dumpster fire. Trust me, the reason I know is that I used to be that guy who thought Excel was a great tool. I built derivatives spreadsheets, backoffice spreadsheets, trading systems with realtime data, all sorts of crap in Excel. Really, it's Stockholm Syndrome. People who previously had zero computing power at their disposal think they've found the hammer…

You bring up some very good points. But to those, I have a question: what is the alternative? Both now (for UK healthcare), and in general? The "people who previously had zero computing power at their disposal" may be wrong in thinking "they've found the hammer that solves all problems" - but there's literally no other hammer available for them. They're not programmers, they won't write their own software (nor would…

Grovel to Google or MSFT or whatever big name tech firm to borrow their coders.

They already have the infrastructure.

They have loads of coders sitting at home, not completely utilized. (Dunno how true it is, but a lot of people here comment that.)

They've already written programs to do something similar.

They're well aware of side issues like data protection, security, cross-platform, etc.

They could use the goodwill.

Wasn't Google already involved somewhere? Surely they can figure out how to count some tests as well.

I suppose if you're clueless you'll think Accenture or Capita are the same as Google, so yeah maybe we are screwed.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#413

Post by gsich » Mon, Oct 05, 2020, 10:11 PM UTC

srtjstjsj wrote: post #372

rbanffy wrote: post #361

Earlier quoted context omitted.

> Not to mention the "system" solution will have to go through its own debugging and will undoubtedly have bugs in it for some time after initial release. Doing code reviews in Excel is hard if the developers are pathologically disciplined. It's impossible most of the time. And so is debugging. It's very unlikely any such system would face these limitations and would silently ignore data the same way Excel did.

Excel will silently corrupt data, not just drop. It loves corrupting anything that resembles a date, and there no way to say "stop it" globally.

use templates. or use the data importer correctly. No problem.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#414

Post by askvictor » Mon, Oct 05, 2020, 10:12 PM UTC

Reason077 wrote: post #329

happytoexplain wrote: post #296

Earlier quoted context omitted.

Frankly I think you're being optimistic/pessimistic (depending on your point of view). A person can be a "decent coder" by very reasonable definitions of the term and still not realize that Excel can't handle this particular volume of data.

Excel can handle the volume of data just fine. The (very outdated) .xls file format with its 65,536 row limit was the problem here.

I understood (different source; not sure what the true picture is) that the problem was with storing each record in a column. There is a limit of 16,000-odd columns. This would, as you say, not be a problem with Excel as such, but with the user(s) not being aware of the limitations (which is still the case in your scenario). I wonder how much the general problem has to do with mistrust of 'experts' so, hey, I can do this myself in a spreadsheet. (Or maybe it was just a decision on-the-fly in a fast moving situation)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#415

Post by masterproc » Mon, Oct 05, 2020, 10:12 PM UTC

Just dropping this here: In Apple Numbers you can work with tables that contain up to 1,000,000 rows (and 1,000 columns). :)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#416

Post by LegitShady » Mon, Oct 05, 2020, 10:13 PM UTC

FabHK wrote: post #401

nickff wrote: post #333

Earlier quoted context omitted.

I think you're overstating both the paper's impact, and the significance of the error. Most such papers are used to support existing policy preferences, not drive them. In addition, the error didn't reverse or erase the correlation, it diminished it, and removed the inflection point from the curve.

> I think you're overstating [...] the paper's impact Krugman [0] called it "surely the most influential economic analysis of recent years" and adds that it "quickly achieved almost sacred status among self-proclaimed guardians of fiscal responsibility; their tipping-point claim was treated not as a disputed hypothesis but as unquestioned fact." However, indeed, he cautions that "the Reinhart-Rogoff fiasco needs to b…

Krugman opeds in the nyt are not academic papers.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#417

Post by masterproc » Mon, Oct 05, 2020, 10:13 PM UTC

Just saying in Apple Numbers you can work with tables that contain up to 1,000,000 rows and 1,000 columns. :)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#418

Post by konjin » Mon, Oct 05, 2020, 10:14 PM UTC

TeMPOraL wrote: post #402

sudosysgen wrote: post #392

Earlier quoted context omitted.

If you don't already know, Octave is a very good free and open source Matlab compatible interpreter and solution. Was of great use in my time in college.

Back in the day, it wasn't "very good", at least not in terms of getting it to work. That RK4 in Excel happened because I got extremely annoyed at Octave.

Peering into my crystal ball: You were between 16 and 22, you were 'great with computers', you had never used an OS other than Windows.

There is a difference between a calculator and a type writer and there is a difference between an office os and one for programming.

Using Windows and blaming Octave for Windows sucking is a rite of passage for everyone in a BSc program. I hope you got better and switched to a Unix.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#419

Post by theptip » Mon, Oct 05, 2020, 10:18 PM UTC

theptip wrote: post #238

tannhaeuser wrote: post #39

Earlier quoted context omitted.

> exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ Worth noting that XML is also a text format. SGML even can treat CSVs as markup. There's nothing wrong with CSVs/TSVs anyway - it's a concise tabular format using only minimal special coding for a record and a field separator, as envisioned by ASCII and EDIFACT. The problem seems more like that there was no error checking in…

> There's nothing wrong with CSVs/TSVs I can see the theoretical purity of this statement, but based on my experience working with CSV files generated by actual non-technical users I have to disagree here. There are a number of footguns here that are really subtle and the average non-technical user has no hope of spotting them. Problems that I've seen in the wild, off the top of my head: * Windows vs. Linux line term…

Sorry, typo on the encoding point -- I meant to say that I've seen issues with UTF-8 vs. Latin-1, not "Linux encoding".

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#420

Post by gsich » Mon, Oct 05, 2020, 10:19 PM UTC

croes wrote: post #280

airstrike wrote: post #268

Earlier quoted context omitted.

You could say the same thing about writing tests. There's nothing wrong with Excel. Build an organized spreadsheet with clear separation of data and presentation, and write checks throughout and you will end up with a perfectly error-free workbook.

Nothing wrong? Excel autoconverts values if it thinks it knows the data type of a column and Excel treats 1900 as a leap year. Genetic researchers changed the names of genes to prevent auzoconveesion errors.

Yes because they couldn't use their tools correctly.