Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

261–270 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#261
post #228

Was just reading the chapter about using Excel as a database in this book two days ago: https://www.goodreads.com/book/show/39074550-humble-pi Apparently it's super common, which fills me with horror. But these guys managed to take it to 11 by abusing it in yet another novel way.

Definitely recommend 'Humble Pi'. Funny as well as informative.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#262

When I use Excel to open a CSV with too many rows, I get an alert saying 'File not loaded completely'. I have to dismiss it by clicking 'OK' before I can interact with the spreadsheet. So Excel isn't silently discarding data.

The BBC reports the problem was saving spreadsheets with more than 65536 rows as XLS. Can you verify Excel also throws an error for that case?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#263
post #134

I've been bitten by this exact issue (albeit on non critical data), save a file as XLS it will silently drop rows beyond 65k - save it as XLSX and it's all good..

Silent? Are you positive? Not doubting you but if so; WTF?!

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#264
post #134

I've been bitten by this exact issue (albeit on non critical data), save a file as XLS it will silently drop rows beyond 65k - save it as XLSX and it's all good..

Until you reach 1,048,576 and then XLSX silently drops rows.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#265

Earlier quoted context omitted.

I'm betting the exporter isn't running out of file, but out of loop counter.

I mean, same difference. "I cannot generate a file representing state of data in RAM" is a 'fail noisy' error, not a 'fail silent' error.

(EDIT: numerous comments point out that Excel displays an explicit warning when truncating data above 256x65k. This comment was written under the assumption that it doesn't.)

Not if it was a bug in the code. 65k sounds suspiciously close to the limit of 16 bit unsigned int.

I'm guessing the exporter looked roughly like:

  int exportRow( ... ) {
    if( /* can't export or no more rows */ ) {
      return 0;
    }
    /* export row */
    return ++someInternalCounter;
  }

  void export( ... ) {
    unsigned short nextRow = 0;
    do {
      nextRow = exportRow(...);
    } while(nextRow > 0);
  }
In the above example, the export would silently stop after 65k entries.

The way people write C in the wild, this wouldn't surprise me in the slightest. And with Microsoft being all about backwards compatibility, Excel probably defers to some ancient and long forgotten code when exporting to XLS.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#266
post #242

Earlier quoted context omitted.

Your post is so stupid that someone could make a copy-pasta of it, e.g. "I tried to write a program in C++ but it didn't want to compile, so I spent 90 minutes poking at the code, looking for the error and never did find it. I don't know what the right solution is to this sort of problem, at least for developers. ------------------- "I don't know how to use Excel (most popular spreadsheet program used for budgeting),…

Umm, who is qualified to use Excel if both scientists and economists frequently have errors in their published works? https://www.economist.com/graphic-detail/2016/09/07/excel-er...

There is of course a difference between complicated scientific models and someone budgeting some expenses.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#267

When I use Excel to open a CSV with too many rows, I get an alert saying 'File not loaded completely'. I have to dismiss it by clicking 'OK' before I can interact with the spreadsheet. So Excel isn't silently discarding data.

... Yeah, but if you have an automated script compiling the reports into one file, and you forget to add correct error handling .... you end up with this mess :-(

I suspect that wherever this foul up happened there wasn't someone sitting and clicking through sheets ignoring errors (I hope).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#268

Earlier quoted context omitted.

You made two different spreadsheets, they came up with different results, and you're blaming Microsoft for the error? Really? Excel, whether you like it or loathe it, is in such wide use around the world that $300 math errors would have been noticed a very long time ago. I could believe that there are still many lurking bugs with obscure corner cases, nasty floating point rounding minutiae and so on, but I would bet…

But they aren't noticed. The classic blunder is dropping the last item off of a summation, and nobody notices until they add another item and things look janky. Those happen all the time, we just only hear about it when the consequences are outsize.

You could say the same thing about writing tests.

There's nothing wrong with Excel. Build an organized spreadsheet with clear separation of data and presentation, and write checks throughout and you will end up with a perfectly error-free workbook.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#269

Earlier quoted context omitted.

You made two different spreadsheets, they came up with different results, and you're blaming Microsoft for the error? Really? Excel, whether you like it or loathe it, is in such wide use around the world that $300 math errors would have been noticed a very long time ago. I could believe that there are still many lurking bugs with obscure corner cases, nasty floating point rounding minutiae and so on, but I would bet…

I've re-read the parent comment a few times, and I really don't see how this is your takeaway. The author is complaining that they find Excel difficult to inspect and different spreadsheets difficult to compare. There's no claim that Excel did a calculation wrong.

> The author is complaining that they find [insert-least-favorite-programming-language] difficult to inspect and different [programs] difficult to compare. There's no claim that [least-favorite-programming-language] did a calculation wrong.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#270
post #85

This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…

It won't be a surprise to anyone who lives in the UK that the problem was blamed on Excel, rather the UK government taking any responsbility for it.
Post reply on HN