Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

401–410 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#401
post #333
post #322

The most consequential Excel error was The 2010 Reinhart-Rogoff error in the paper "Growth in a Time of Debt" https://www.nber.org/papers/w15639 The paper claimed that average real economic growth declines 0.1% when natonal debt rises to more than 90% of gross domestic product (GDP). When you correct the error it shows 2.2% average increase in economic growth. Paul Ryan used it in the US for Republican budget proposa…

I think you're overstating both the paper's impact, and the significance of the error. Most such papers are used to support existing policy preferences, not drive them. In addition, the error didn't reverse or erase the correlation, it diminished it, and removed the inflection point from the curve.

> I think you're overstating [...] the paper's impact

Krugman [0] called it "surely the most influential economic analysis of recent years" and adds that it "quickly achieved almost sacred status among self-proclaimed guardians of fiscal responsibility; their tipping-point claim was treated not as a disputed hypothesis but as unquestioned fact."

However, indeed, he cautions that "the Reinhart-Rogoff fiasco needs to be seen in the broader context of austerity mania: the obviously intense desire of policy makers, politicians and pundits across the Western world to turn their backs on the unemployed and instead use the economic crisis as an excuse to slash social programs. What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses."

I'd say the paper made it certainly easier for the Very Serious People[1] to push through their austerity agenda.

[0] https://www.nytimes.com/2013/04/19/opinion/krugman-the-excel...

[1] Krugman's derogatory term for duplicitous policy wonks (like Paul Ryan) that pretend to be thoughtful and serious and concerned about the nefarious long term effects of debt when discussing stimulus and spending and social programs, but then typically turn around and implement huge tax cuts without second thoughts...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#402

Earlier quoted context omitted.

My 4th order Runge-Kutta ODE solver I did in an afternoon in Excel the other day, because I didn't want to pirate Matlab to do a homework assignment, begs to differ :). Not to mention, the entire financial industry and many (most?) small businesses. Lots of people use Excel for various types of math, and it works fine, if you know what you're doing . Excel not being idiot-proof doesn't mean it's not usable. And that'…

If you don't already know, Octave is a very good free and open source Matlab compatible interpreter and solution. Was of great use in my time in college.

Back in the day, it wasn't "very good", at least not in terms of getting it to work. That RK4 in Excel happened because I got extremely annoyed at Octave.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#403

I hate solving problems and not knowing how I did it, so this particular experience has stuck in my craw. Trying to develop a budget to pay off debts, my partner made this elaborate Excel spreadsheet and the output was that basically she had no spending money and I had very little, until one or both of us got a raise. It was far more austere than either of us were willing to go. So I started over using a different eq…

You made two different spreadsheets, they came up with different results, and you're blaming Microsoft for the error? Really? Excel, whether you like it or loathe it, is in such wide use around the world that $300 math errors would have been noticed a very long time ago. I could believe that there are still many lurking bugs with obscure corner cases, nasty floating point rounding minutiae and so on, but I would bet…

The OP said that using a spreadsheet program can be messy and prone to mistakes, and implied that there could exist alternative approaches. The OP never said a bug in Excel was responsible for the mistake or blamed Microsoft for corrupting the calculation.

In the same sense, one could say manual memory management in C can be messy and prone to mistakes, it doesn't mean that one believes C should never be used in any circumstance, and it doesn't mean one's trying to blame the C complier, libc or Dennis Ritchie for corrupting one's program.

Hypothetically, if someone writes the following reply.

> The C programming language, whether you like it or loathe it, is in such wide use around the world that a NULL pointer dereference would have been noticed a very long time ago. I could believe that there are still many lurking libc bugs with obscure corner cases, but I would bet the memory corruptions were solely due to your own failures.

It would completely miss the point.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#404

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

No need for a “front end” or “backend” or “pigsqueal”. MS provides a perfectly workable all in one tool, Access, for this kind of work. You can even use Excel as a front-end to access data if you want.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#405

Earlier quoted context omitted.

Frankly I think you're being optimistic/pessimistic (depending on your point of view). A person can be a "decent coder" by very reasonable definitions of the term and still not realize that Excel can't handle this particular volume of data.

Excel can handle the volume of data just fine. The (very outdated) .xls file format with its 65,536 row limit was the problem here.

This is the test and trace system for the worst pandemic this country has ever faced, six months in. I’m sorry Excel spreadsheets won’t cut it.

This is a management failure. PHE has capable coders. And if not, they could hire some.

The NHSX team has managed to write the dang COVID-19 app twice. Once not using the Apple/Google contact API (because management) and a second time properly. It’s all open source and it looks pretty decent.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#406

Earlier quoted context omitted.

Excel never works for math, at any scale. It's a constant trashfire of "looks fine" until it shows a result someone doesn't want, and then an error is found that changes results and the cycle repeats. Excel is nice for visualizing and browsing data you already have, and informally searching and sorting for hypothesis generation.

> Excel never works for math, at any scale. That is patently untrue. I've borrowed and lent money based on Excel spreadsheets, more than once. That's small-scale math with real-life consequences, and Excel is an excellent tool for the job.

I'd concur if s/excellent/adequate/.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#407

Earlier quoted context omitted.

Excel never works for math, at any scale. It's a constant trashfire of "looks fine" until it shows a result someone doesn't want, and then an error is found that changes results and the cycle repeats. Excel is nice for visualizing and browsing data you already have, and informally searching and sorting for hypothesis generation.

My 4th order Runge-Kutta ODE solver I did in an afternoon in Excel the other day, because I didn't want to pirate Matlab to do a homework assignment, begs to differ :). Not to mention, the entire financial industry and many (most?) small businesses. Lots of people use Excel for various types of math, and it works fine, if you know what you're doing . Excel not being idiot-proof doesn't mean it's not usable. And that'…

Having worked in the financial industry, my first job was writing a system to cover for a $10m fuck up in excel where yahoo exchange rates stopped updating and the same spreadsheet was used for a few weeks until someone noticed "Shit, we're losing a lot of money here.".

Excel is the right solution in the same way playdoh is a valid building material.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#408

Earlier quoted context omitted.

You made two different spreadsheets, they came up with different results, and you're blaming Microsoft for the error? Really? Excel, whether you like it or loathe it, is in such wide use around the world that $300 math errors would have been noticed a very long time ago. I could believe that there are still many lurking bugs with obscure corner cases, nasty floating point rounding minutiae and so on, but I would bet…

The OP said that using a spreadsheet program can be messy and prone to mistakes, and implied that there could exist alternative approaches. The OP never said a bug in Excel was responsible for the mistake or blamed Microsoft for corrupting the calculation. In the same sense, one could say manual memory management in C can be messy and prone to mistakes, it doesn't mean that one believes C should never be used in any…

(Side note: there were sarcastic/affectionate calls of "curse you, Dennis Ritchie!" at a previous workplace.)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#409
post #401
post #333

Earlier quoted context omitted.

I think you're overstating both the paper's impact, and the significance of the error. Most such papers are used to support existing policy preferences, not drive them. In addition, the error didn't reverse or erase the correlation, it diminished it, and removed the inflection point from the curve.

> I think you're overstating [...] the paper's impact Krugman [0] called it "surely the most influential economic analysis of recent years" and adds that it "quickly achieved almost sacred status among self-proclaimed guardians of fiscal responsibility; their tipping-point claim was treated not as a disputed hypothesis but as unquestioned fact." However, indeed, he cautions that "the Reinhart-Rogoff fiasco needs to b…

Krugman's editorials are polemics, not economics. I don't think anyone should trust the content of an opinion piece which purports to describe the logic, motives, and rationales of ideological opponents, as was the case in your linked article.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#410
post #350

Earlier quoted context omitted.

And yet, I have first-hand witnessed a scenario like the one described in an AU government department. A small system developed by business (one person who had subsequently moved on) using Excel and Word and macros to keep track of information and generate letters to "customers" had become an essential tool, but unsupported and undocumented. When we (IT) recommended redeveloping it in Java with an SQL database (and d…

Why are there infrastructure costs for a database server? Once you have one it can host all your small databases.

Developer salaries are the main cost.
Post reply on HN