Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Tyranny of Spreadsheets

timharford.com

Post reply on HN
Print view View original

81–90 of 282 posts

Re: The Tyranny of Spreadsheets

#81

Post by markus_zhang »

rvba wrote: post #59
markus_zhang wrote: post #23

In my last company we keep a spreadsheet that has tabs occupying almost the maximun number of columns allowed and in different headers. Updating that Excel file needs the full team to work for two full days each month. That was after I created a VBA script to scrape data from internal dashboards and put them directly in the spreadsheet. From my experience these monstrosities are usually created by two reasons: Push f…

Maximum number of columns is 16 384 (in modern Excel). PostreSQL has a limit of 16 000. So Excel > PostrGRE? :) I wonder what kind of data did you use and why it had to be stored in so many columns. This approach would probably kill a "real" database too.

Oops I definitely got it wrong. Probably a bit more than a thousand columns now that I think about it. Still a beast. The thing is a database with every column a product id.

Re: The Tyranny of Spreadsheets

#82

Post by goatlover »

CJefferson wrote: post #62
asdff wrote: post #57

Earlier quoted context omitted.

Python or R have tons of ways to edit, extend, and quickly filter and search within stuff like a CSV. You could even do all of that with a bash script. It will probably run a whole lot faster on your computer than opening the excel workbook.

I think we might be talking at cross purposes. How, in Python, do I extend a CSV with new numbers, or edit existing values? Use 'input'? I could implement an excel-like interface, but then I already have Excel. I could just open it in a text editor I suppose, but then I have to deal with escaping, and moving around cells, and that sounds painful.

The Pandas library makes manipulating tabular data easy. And it's better than Excel when it comes to cleaning messy data.

Re: The Tyranny of Spreadsheets

#83

Post by CivBase »

romwell wrote: post #73
CivBase wrote: post #17

I hate Excel, not because it's a bad program but because in my experience people misuse it more often than not. Excel excels at creating spreadsheets. Spreadsheets are pretty reports used to report tabular data. Excel is capable of so much more, but for eveything beyond spreadsheets it's medicore at best and its complexity often makes it a liability. It's a bad calculator. It's a bad database. It's a bad medium for t…

Nah. Spreadsheets are poor man's MapReduce. Most Excel tables I've seen aren't just data and presentation. It's computing the data as well as presenting it. And how is the data computed? * Applying formula to a row/column range to get another range * Applying formula to a range to get a single-cell output out of it Obviously, I am not talking about clustering and performance here, but a computational paradigm.

> Spreadsheets are poor man's MapReduce.

Yes. Emphasis on "poor man's". Excel is mediocre at many things, but it's highly accessible so people use it anyways. That can be good or bad depending on the situation.

Need to quickly crunch numbers for a small dataset? Excel is perfect. You can probably put something together quickly to get what you want.

Need to maintain a contact tracing database? Not so great. Excel scales poorly and the low implementation cost isn't worth the limitations and risks.

Re: The Tyranny of Spreadsheets

#84

Post by jay_kyburz »

echelon wrote: post #45
rexreed wrote: post #37

* The Utility of Spreadsheets No single application has been as widely adopted by as wide a number of people for such a wide range of uses as the spreadsheet. Love them or hate them, but the spreadsheet metaphor is at once highly useful, highly adaptable, and highly usable requiring minimal support to extract needed value. The spreadsheet, with all its questionable glory, is here to stay. From the first Visicalc to E…

You're correct, but this entirely misses the point of the article. It's not calling for the end of spreadsheets -- it's stating that there's an unspoken problem that demands our attention. Spreadsheets aren't 100% reliable for use cases where you need to collaborate and share immutable health records. Especially during a time of global emergency when tensions are heightened. Spreadsheets don't impose validation, sche…

It would be hard, but I think a new kind of spreadsheet that was easier to validate, easily impose a schema and constraints and helps us identify human errors would be helpful.

I bet excel has some excellent features around this, but they are not front and center in the UI.

Re: The Tyranny of Spreadsheets

#85

Post by r00fus »

dan-robertson wrote: post #65
ttul wrote: post #61

Earlier quoted context omitted.

Personally, I strongly prefer working with GSheets to Excel. It handles large sheets better and you never worry about the app crashing and taking your precious work along with it.

Instead one only needs to worry that one can’t get one’s work done. I don’t think I’ve ever had a time when I wanted to use a spreadsheet for anything moderately complicated where gsheets was up to snuff. That’s not to say that excel doesn’t have problems or does things easily, but it doesn’t lead to quite the same level of frustration and hair-pulling, and indeed it’s usually possible to achieve things in excel when…

As far as competing anecdata, I found gsheets absolutely much better for joining together multiple sheets of data. The lookups are so much less touchy simply because "it's all in the cloud" (as long as you have permissions).

With Excel you have to ensure the file is there, and readable, not tampered with, etc.

This allowed us to create better multi-workbook "apps" that have a bit more modularity - you could even embed sheets into a webpage as a view-only client.

In the end, Excel is good for us only for interoperability with external groups.

Re: The Tyranny of Spreadsheets

#86

Post by swader999 »

LeifCarrotson wrote: post #49
swader999 wrote: post #25

Many businesses risk it all by building actual applications with Excel. Security, version control, automated testing are pretty much set aside. Case in point: https://insideparadeplatz.ch/wp-content/uploads/2013/01/ubs-...

It's not like they had a meeting where a consultant said "We reviewed all your requirements, and you could build this with normal software engineering practices, or we could build this in spreadsheets with associated deleterious effects on privacy, correctness, durability, extensibility, readability, and interoperability, likely resulting in an estimated $N to $NN million shortfall in 5 years. What do you want? Yeah,…

Very true, especially for small business. But large multi national enterprises are also doing this.

Re: The Tyranny of Spreadsheets

#87

Post by rexreed »

echelon wrote: post #45
rexreed wrote: post #37

* The Utility of Spreadsheets No single application has been as widely adopted by as wide a number of people for such a wide range of uses as the spreadsheet. Love them or hate them, but the spreadsheet metaphor is at once highly useful, highly adaptable, and highly usable requiring minimal support to extract needed value. The spreadsheet, with all its questionable glory, is here to stay. From the first Visicalc to E…

You're correct, but this entirely misses the point of the article. It's not calling for the end of spreadsheets -- it's stating that there's an unspoken problem that demands our attention. Spreadsheets aren't 100% reliable for use cases where you need to collaborate and share immutable health records. Especially during a time of global emergency when tensions are heightened. Spreadsheets don't impose validation, sche…

You're also correct, but I think miss the point as well. The spreadsheet isn't going anywhere. Curse it all we might, blame it for deaths or attribute any aspect of malignment to it, and you'd be right. But the spreadsheet isn't going anywhere.

Spreadsheet mistakes have been directly attributed not only to thousands of deaths but billions of dollars of mistakenly wasted money. Heck, in the UK alone, they have attributed billions of pounds and thousands of deaths specifically to spreadsheet errors and flaws, and that's not even counting the COVID-related issues from the article. Check this scary article: https://theconversation.com/excel-errors-the-uk-government-h...

There have been hordes of purpose-built apps that have aimed to replace the common spreadsheet for any number of tasks from inventory management to financial planning to even tracking nuclear weapons (true story: our nation's nuclear weapons stockpile is tracked in a spreadsheet). And yet, the spreadsheet is still here. The spreadsheet is the cockroach of all apps. At once utterly adaptable and seemingly indefeatable.

The spreadsheet is a horror show with lack of control, schema, and collaborative management. We surely can do better. But we haven't. Google and Microsoft have the best developers in the world who can do anything. And they have produced more spreadsheets. But spreadsheets are the bane of our existence! And yet they are still here.

The spreadsheet has survived from the mainframe era to the cloud-based era in pretty much the same form with enhancements. The spreadsheet is not going away, even with all the complaints in the article. The points are well made and well founded. But unfortunately they miss the point. 20 years from now we'll still be cursing the tyranny of the spreadsheets because of their utility.

I close with a Haiku:

| Rows and cells, alas |

| What makes your simple structure |

| So very useful? |

Re: The Tyranny of Spreadsheets

#88

Post by TremendousJudge »

CJefferson wrote: post #56
goatlover wrote: post #38

Earlier quoted context omitted.

Notebooks are the next-gen spreadsheets, except they use programming languages instead of formulas.

Jupyter is a joke compared to spreadsheets, it doesn't auto-recalculate when you edit cells.

yeah, and also it's 1-dimensional; you can only go up or down, never to the right

Re: The Tyranny of Spreadsheets

#89

Post by [deleted] »

asdff wrote: post #70
CJefferson wrote: post #62

Earlier quoted context omitted.

I think we might be talking at cross purposes. How, in Python, do I extend a CSV with new numbers, or edit existing values? Use 'input'? I could implement an excel-like interface, but then I already have Excel. I could just open it in a text editor I suppose, but then I have to deal with escaping, and moving around cells, and that sounds painful.

You would read in the csv, edit what you need or add/remove data, then write a new file or overwrite the old one. You don't need to implement any interface, there are tools that do all of this. You could probably write the code up for something simple like this in the time it takes for excel to open the file and you to make the change manually by finding the field, clicking away with the mouse, and typing it into the…

[deleted]

Re: The Tyranny of Spreadsheets

#90

Post by codesnik »

Closi wrote: post #11
gcthomas wrote: post #4

Spreadsheets are a hammer so if that's all you know then all problems look like the nail. Really, many power users of Excel ought to be looking at more capable, testable and readable solutions from professional data analysts and scientists. Python would be a good start, but there are many options better than Excel for critical systems.

> Really, many power users of Excel ought to be looking at more capable, testable and readable solutions from professional data analysts and scientists. Python would be a good start. I can write python, but I can get the answer out of excel in a fraction of the time for most use cases. As an added bonus, excel documents are far more ubiquitous and I can share them with clients, and they can see my workings. Python is…

> they can see my workings

How? Do they check that every cell in a column actually has the same formula? Do they check data format everywhere?

Python script is something that is possible to be actually reviewed, and results - reproduced, and "formula's" there are actually readable.

But for some stupid reason excel files are still shared over email.

Post reply on HN