Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You shouldn’t use a spreadsheet for important work

lemire.me

Post reply on HN
Print view View original

81–90 of 200 posts

Re: You shouldn’t use a spreadsheet for important work

#81

Post by morganherlocker »

bhartzer wrote: post #52

Excel can only handle a certain amount of records. I've personally found it hard to deal with more than 100,000 rows in Excel.

The limit went up considerably with the introduction of 64-bit Excel. Also, there is a plugin called "Power Query" that allows the actual data to be stored in MSSQL. I have seen this work on tables with 100s of millions of records without a hitch.

Re: You shouldn’t use a spreadsheet for important work

#82

Post by thatthatis »

Spreadsheets ARE software, anyone arguing differently has insufficient experience in at least one of the fields to have an informed opinion. I've worked professionally in both, and the argument in this article is bunk.

Spreadsheets are a different kind of software with advantages and disadvantages compared to compiled/scripted languages.

How do you treat a spreadsheet as real software:

* One operation per cell

* Inputs, processing, and display separated into distinct sheets

* Every formula checked twice (or three times)

* Write row and column check sums

* Treat row and column locking ($F4 or F$4) as the footguns they are.

* Name constants

Testing spreadsheets requires a different set of tools and behaviors, but to say that "quick and dirty" spreadsheets are less durable than "well engineered software" is really to say that "quick and dirty analysis" is less durable than "well engineered analysis."

Spreadsheets are remarkably easy to learn to use and to use correctly compared to scripted/compiled software. Not the least benefit being that they force abstraction in a way that is extremely intuitive to most people -- spacial context with the labels and state simultaneously displayed leads to much much lower context loading cost (for the human brain) than keeping all the abstract variables in your head when writing code.

Mr. Lemire seems to treat the tool as a toy then claim it is only good as a toy. Flub and not Flub languages alike can be run quite easily without unit tests, in which case they are no more durable to errors than excel (and arguably less as they don't display intermediate state).

Re: You shouldn’t use a spreadsheet for important work

#83

Post by habosa »

I think tools like MATLAB could bridge the gap between spreadsheets and general-purpose programming/data analysis. It has the huge standard library Excel users want, a REPL for prototyping, visual data inspection, and the ability to very quickly create charts and figures. The scripts are also easy to share and it doesn't involve use of the command line unless you want it to.

Anyone want to teach "MATLAB for Bankers"?

Re: You shouldn’t use a spreadsheet for important work

#84

Post by Iftheshoefits »

jasode wrote: post #49

(Since the blog's website is not responding, I had to read the article from google's cache[1]) The author has well-intentioned advice about avoiding MS Excel but it's misguided. The criticism fails to accommodate the reason why MS Excel was used. MS Excel is the lingua franca of non-programmers . Thomas Piketty is a trained economist, not a programmer. It's not realistic to expect Mr. Piketty to set aside months (yea…

It strikes me as rather odd that a discipline in which a person purports to be on solid analytical ground would eschew learning how to write their research and modeling code in Fortran, C, C++, Scala, Python, etc. I can't think of a single colleague of mine, when I was in academia, who did not know how to write code in at least one of the languages you mentioned. That's because such knowledge is necessary to do rigorous, proper numerical computation in an analytical discipline like math, a variety of sciences and, yes, economics.

The "GIGO" issue you note is irrelevant, but even if it weren't GIGO is compounded and amplified by the copy-and-paste paradigm of spreadsheet "programming", so if anything bringing up GIGO is even more damning of the use of spreadsheets for analytical work.

Re: You shouldn’t use a spreadsheet for important work

#85

Post by laurencei »

I'm pretty sure patio11 says on one of his blogs that anywhere anyone is using an excel file for important work is a SaaS waiting to be born.

edit: found it:

"My favorite symptom of an unmet need for software is any Excel spreadsheet which is ever updated by one employee, sent to a second employee, updated, and then sent back. Every time that happens a SaaS angel gets its wings."

https://training.kalzumeus.com/newsletters/archive/validatin...

Re: You shouldn’t use a spreadsheet for important work

#87

Post by dragonwriter »

thatthatis wrote: post #82

Spreadsheets ARE software, anyone arguing differently has insufficient experience in at least one of the fields to have an informed opinion. I've worked professionally in both, and the argument in this article is bunk. Spreadsheets are a different kind of software with advantages and disadvantages compared to compiled/scripted languages. How do you treat a spreadsheet as real software: * One operation per cell * Inpu…

> Spreadsheets are remarkably easy to learn to use and to use correctly compared to scripted/compiled software.

I don't think this is true. IME, people who aren't primarily programmers/developers and have been using Excel for years tend to have worse practices developing Excel tools than the same people have with much less time learning, e.g., Python. Excel is much easier to learn to use to get some kind of result, perhaps, but not easier to learn to use correctly. (Part of it, I think, is because the cost of incorrect use is somewhat more deferred, which is also part of what makes it more attractive to use initially in many cases.)

Re: You shouldn’t use a spreadsheet for important work

#89

Post by saosebastiao »

pinaceae wrote: post #38

Wrong end of the correlation/causation pattern. Human beings make mistakes. A lot of humans use Excel. Hence a lot of errors in Excel. Would all those humans program their logic in Python - we'd have a lot of wrong calculations in Python code and an article stating to not use Python.

I am human, therefore there is no difference between coding in C and coding in Rust/Cyclone?

I don't think it works that way. The surface area where you are permitted to make mistakes is completely different in language X compared to excel. In Excel, `="a"+4` is a cell error but a valid spreadsheet. In python it is an invalid program, throwing a runtime error. In haskell, it doesn't even compile.

Re: You shouldn’t use a spreadsheet for important work

#90

Post by dworin »

twistedpair wrote: post #48
panarky wrote: post #32

JP Morgan's $6.2 billion "London Whale" trading loss: An unnamed overworked staffer in London copied and pasted the wrong figures into an Excel spreadsheet, throwing the firm's risk models out of whack. http://finance.fortune.cnn.com/2013/01/18/jpmorgan-london-wh... Spreadsheet errors are reaching epidemic proportions globally. What we need is a transnational organization to contain the threat with research, best pra…

I used to work for Deutsche Bank. All financially modeling ran through Excel, though admittedly a massive C++ library that is a big Excel macro plugin. These sheets get massive and there are a few clusters of blades that do nothing but run Excel macros day and night. But why is this madness? It works for them . The iBanking world is the madness. You can put in place effective change controls and auditing with any sys…

There is actually a business reason why this is the case. The general assumption in most investment banks is that either the market or the regulations will change so quickly that by the time you've written software the "right" way, either the market or the government has already moved on. Excel lets you move fast, and make changes quickly, even if it results in a bloated mess later on. Like you say, Excel isn't the issue, it's the nature of the business.
Post reply on HN