Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You shouldn’t use a spreadsheet for important work

lemire.me

Post reply on HN
Print view View original

151–160 of 200 posts

Re: You shouldn’t use a spreadsheet for important work

#151

Post by briandh »

joe_the_user wrote: post #143

Just a default note that the whole "he got stuff wrong" claim itself might not stand up to scrutiny. http://krugman.blogs.nytimes.com/2014/05/24/is-piketty-all-w... Edit: Also regarding the Rogoff data. Sure, those findings really were crap but surprise, surprise, it was crap that policy makers were oh so eager to hear. One might blame the falsity on the dumb use of spreadsheet but it seems more likely it was a case…

> it seems more likely it was a case of "let's fudge data in a crap fashion till we come up with what people want to hear"

That is a pretty bold claim.

Re: You shouldn’t use a spreadsheet for important work

#152

Post by doorhammer »

dragonwriter wrote: post #147
bane wrote: post #101

Earlier quoted context omitted.

What's interesting to me is that software spreadsheets are more or less just evolved version of what apparently VisiCalc got right on the first try. Sure software spreadsheets are basically just parroting the physical ledgers that came before them, but turning them into software almost immediately opened up a kind of dynamic that didn't exist before. The latest Excel is basically just an organically evolved VisiCalc.…

> Efforts to move them into databases or whatever be damned. A spreadsheet is != to a database table except maybe in the most abstract possible sense in some cases. The problem is that very often, spreadsheets are used as databases because they are the tool the user is familiar with, not because the application is inherently more suited to a spreadsheet. > They really are different things and I agree that issues with…

>There is no reason a database using a relational data model (but not the multiuser/concurrency features that are also part of the relational model) needs to have a "long and highly skilled environment setup requirement", or, in fact, be any harder to setup than installing an app just like a spreadsheet app would require.

This is a very salient point to me. A lot of people take the route of talking about why spreadsheets are so bad. In most programming language discussions, I see the more sane folks arguing that most languages aren't bad, but some have terrible idioms, or reams of ancient legacy code from before best practice was a best practice, or maybe they make it too easy to do the wrong thing, and inconvenient to do the right thing (I'm thinking of a lot of stuff Rich Hickey has said).

Instead, I like to think about how the spreadsheet environment or tooling could be altered to make it more natural to do the "right" thing, or to prevent certain classes of errors (maybe in similar ways to how type systems can). I'm not proposing a specific solution, but more a mindset for what I think is interesting (and probably not a terribly original mindset at that).

Re: You shouldn’t use a spreadsheet for important work

#153

Post by waps »

analog31 wrote: post #145
bane wrote: post #101

Earlier quoted context omitted.

What's interesting to me is that software spreadsheets are more or less just evolved version of what apparently VisiCalc got right on the first try. Sure software spreadsheets are basically just parroting the physical ledgers that came before them, but turning them into software almost immediately opened up a kind of dynamic that didn't exist before. The latest Excel is basically just an organically evolved VisiCalc.…

I'm not a computer scientist, but I think that spreadsheets were possibly ahead of their time in a couple ways: 1. They let novices build fairly effective user interfaces, just by entering headers for rows and columns. 2. If I understand what "data flow" programming is (based on using LabVIEW), then the formula cells in spreadsheets are such a thing. Each formula is recomputed whenever its inputs change. That creates…

They have loads of theoretical advantages :

1) spreadsheets are a purely functional programming language. When you point this out to management users of it (after explaining what it is), they wouldn't want to live without it.

So in reality the most widely used programming language in the world, is a purely functional one.

2) they are NOT turing complete (assuming you stay away from VBScript)

Of course, they're worse than BASIC when it comes to naming things, which is what everyone here is complaining about. But these are major advantages, to be fair. The only thing that comes even vaguely close to how spreadsheets work are the IPython notebooks.

Re: You shouldn’t use a spreadsheet for important work

#154

Post by sukuriant »

dragonwriter wrote: post #150
sheetjs wrote: post #149

Earlier quoted context omitted.

The excel database functions (DAVERAGE, DCOUNT, DGET, DMAX, DMIN, DPRODUCT, DSTDEV, DSUM, DVAR) give you about 90% of what you want in a database with a query "language" that's simpler than SQL.

> The excel database functions (DAVERAGE, DCOUNT, DGET, DMAX, DMIN, DPRODUCT, DSTDEV, DSUM, DVAR) give you about 90% of what you want in a database with a query "language" that's simpler than SQL. As someone who has worked rather extensively with both, no, they give a lot less than 90%, and for even fairly simple uses they are often more complex to use than SQL.

For you and the way you think. Don't forget the non-programmer mind.

Re: You shouldn’t use a spreadsheet for important work

#155

Post by coldtea »

jmount wrote: post #139
acqq wrote: post #110

Earlier quoted context omitted.

Steps require "human transcription"? I think you've got a wrong term, and if you mistype the data you have on the paper when making a spreadsheet I don't see why you wouldn't when doing the same typing using your text editor.

"Manual intervention" would have been better. The steps involved can be copying and pasting, selecting a range (very nasty if you miss and don't notice), and even typing data from another source. Compare that to something like R where you typically load the data (even from a URL) with command like "read.table()" and most operations are implicitly over the whole data frame (no copying of data from paper required).

Somebody has to type the data you loaded.

Re: You shouldn’t use a spreadsheet for important work

#156

Post by joe_the_user »

briandh wrote: post #151
joe_the_user wrote: post #143

Just a default note that the whole "he got stuff wrong" claim itself might not stand up to scrutiny. http://krugman.blogs.nytimes.com/2014/05/24/is-piketty-all-w... Edit: Also regarding the Rogoff data. Sure, those findings really were crap but surprise, surprise, it was crap that policy makers were oh so eager to hear. One might blame the falsity on the dumb use of spreadsheet but it seems more likely it was a case…

> it seems more likely it was a case of "let's fudge data in a crap fashion till we come up with what people want to hear" That is a pretty bold claim.

Rogoff has a pretty long history of publishing what well-heeled people apparently want to hear, it's amazing that he was able to hide behind the "this was just a spreadsheet error" point.

The problem is I've mostly observed his chameleon act in radio interviews pre and post crisis. The pre-crisis interview, he claimed their was no problem and the post-crisis interview, he talked about how angry he was against the folks who claimed there was no problem, so unfortunately I can't provide a link (the files probably wouldn't be online anymore even if I took the trouble to find the link - smart shills do things by voice, it seems).

Re: You shouldn’t use a spreadsheet for important work

#157

Post by joe_the_user »

jimmcslim wrote: post #5

There have been a number of efforts to build a better spreadsheet. One that I was quite familiar with (I bought a license!) was Resolver One [1] that was trying to build a spreadsheet with support for Python at the cell-level. Unfortunately it didn't take off, and the team moved onto Python Anywhere [2]. Excel clearly has strong network effects! I loved this article [3] from 2008, that argues that Excel and VBA were…

Ah,

I can believe "flawed product" were constructed by spreadsheet but that seems a kind of shallow analysis.

Clearly, the housing bubble involved a rush to buy whatever - spreadsheet might seem the "immediate cause" but "wishful thinking" would clearly be the larger cause with various inherent tendencies of the financial system behind that.

Re: You shouldn’t use a spreadsheet for important work

#158

Post by orbifold »

dworin wrote: post #90
twistedpair wrote: post #48

Earlier quoted context omitted.

I used to work for Deutsche Bank. All financially modeling ran through Excel, though admittedly a massive C++ library that is a big Excel macro plugin. These sheets get massive and there are a few clusters of blades that do nothing but run Excel macros day and night. But why is this madness? It works for them . The iBanking world is the madness. You can put in place effective change controls and auditing with any sys…

There is actually a business reason why this is the case. The general assumption in most investment banks is that either the market or the regulations will change so quickly that by the time you've written software the "right" way, either the market or the government has already moved on. Excel lets you move fast, and make changes quickly, even if it results in a bloated mess later on. Like you say, Excel isn't the i…

I would still expect that with their profit margins they would all have come up with custom solutions by now.

Re: You shouldn’t use a spreadsheet for important work

#159

Post by NamTaf »

sanarothe wrote: post #56
opium_tea wrote: post #13

In my experience most civil engineering organisations use spreadsheets for the majority of design calculations. Some large organisations even still insist on engineers writing out calcs by hand. For example, a calculation that determines whether or not a retaining wall is of sufficient size more often than not will be completed in excel. Ditto with the calculation that checks the weight bearing capability of a column…

This is what I'm hoping to do with a software background going to more traditional engineering. It's just a little offputting when people talk about the tools they (have) to use -- When I first started, I would never have expected Excel to be so prominent, and Matlab so absent.

Because not every engineer is a programmer. Especially not every non-CS/EE engineer. Especially not every non-CS/EE engineer over ~30.

Yes, they've been exposed to programming at some level, but they are not programmers. Even as a late-20s mech engineer, I am far above most of my peers when it comes to slinging code and I'm a rank amateur. Many wouldn't even want to touch it.

You're mistakingly assuming that the path of least resistance is "teach non-CS/EE engineer to program because they are technically minded and can easily pick it up" rather than "adapt mathematical process to suit excel".

Re: You shouldn’t use a spreadsheet for important work

#160

Post by ChuckMcM »

pinaceae wrote: post #38

Wrong end of the correlation/causation pattern. Human beings make mistakes. A lot of humans use Excel. Hence a lot of errors in Excel. Would all those humans program their logic in Python - we'd have a lot of wrong calculations in Python code and an article stating to not use Python.

I got a different message, the message was that mistakes are harder to find in Excel than in other programming languages, and correct operation is harder to validate. Had Piketty used Python it might be that the libraries would be more amenable to code review as reading the structure is easier than it is reading the formula contents of all the cells in a spreadsheet.

There are some excellent tools that are used by civil engineers where the formulas all have to be published when working on any thing. They miss a decimal point and bridges fall down. My brother in law is working on automating some of the drudge work but essentially they produce a falsifiable sequence of claims so that anyone can assure themselves they got the correct answer, and if the formualae are wrong it is immediately apparent.

Post reply on HN