Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Tyranny of Spreadsheets

timharford.com

Post reply on HN
Print view View original

91–100 of 282 posts

Re: The Tyranny of Spreadsheets

#91

Post by Jiro »

asdff wrote: post #39
narush wrote: post #20

Fantastic blog post. I highly recommend reading it in full, and also checking out the work of the European Spreadsheet Risks Group, and Felienne Hermans specifically (referenced in the original post). I've been working on a spreadsheet startup [1] for the past 8 months or so and those folks have a large amount of absolutely upsetting-but-helpful research on spreadsheet usage/errors. After the past few years working w…

I don't know why we think the user is some child and try to abstract away any and all complexity from their job, which ends up leading to some proprietary solution with huge inefficiencies somewhere compared to a flat file and a script in python or R. Software like excel is often seen as a way to do stuff you could do in R but without having to write code. Imo that thinking is wrong because it makes this assumption t…

>It's not hard to code, children do it.

It is hard to code, and average children don't do it. The fact that exceptional children do it more than they do other adult things is because coding doesn't require a lot of expensive equipment and you can afford to do it as a kid.

Re: The Tyranny of Spreadsheets

#92

Post by ncphil »

goatlover wrote: post #82
CJefferson wrote: post #62

Earlier quoted context omitted.

I think we might be talking at cross purposes. How, in Python, do I extend a CSV with new numbers, or edit existing values? Use 'input'? I could implement an excel-like interface, but then I already have Excel. I could just open it in a text editor I suppose, but then I have to deal with escaping, and moving around cells, and that sounds painful.

The Pandas library makes manipulating tabular data easy. And it's better than Excel when it comes to cleaning messy data.

Plus 100 for pandas. It is a truly amazing library. If I need to do some exploratory data analysis with it I'll often do so in a Jupyter notebook, but always make sure serious coding happens in a real editor like vscode.

But things really are hopeless.

Even Microsoft can't control the monster they unleashed on the world. Execs and managers using Excel instead of Planner for project tracking ("Spreadsheets are for manipulating numbers, not word processing!"). Used in that way, Excel isn't so fragile, but why is it being used at all? For f*cks sake, hasn't anyone ever heard of Word tables?

Honestly, anyone who uses Excel as a database should be exiled to... someplace where they can't do any more damage. Maybe connectivity-poor rural Arkansas. That's technological malpractice. It's indefensible, and an embarrassment to the whole tech sector that anyone is trying to excuse it.

Re: The Tyranny of Spreadsheets

#93

Post by goatlover »

CJefferson wrote: post #56
goatlover wrote: post #38

Earlier quoted context omitted.

Notebooks are the next-gen spreadsheets, except they use programming languages instead of formulas.

Jupyter is a joke compared to spreadsheets, it doesn't auto-recalculate when you edit cells.

True, but you have access to whatever Python libraries are installed, and your output is web-based. So you can potentially output anything a browser can display, including interactive 3D graphics and widgets. You also have access to the command line in Jupyter, so you can run whatever shell commands you like.

Re: The Tyranny of Spreadsheets

#94

Post by goatlover »

TremendousJudge wrote: post #88
CJefferson wrote: post #56

Earlier quoted context omitted.

Jupyter is a joke compared to spreadsheets, it doesn't auto-recalculate when you edit cells.

yeah, and also it's 1-dimensional; you can only go up or down, never to the right

Because the data is inside a data structure like a table or dictionary that you can output, iterate and map over however you like with the various Python and library methods or functions.

Re: The Tyranny of Spreadsheets

#95

Post by newbie2020 »

asdff wrote: post #50
spoonjim wrote: post #30

Excel has its flaws, as do spreadsheets in general, but the upside is that spreadsheets allow people to become programmers who never would grok programming in purely abstract terms. Spreadsheets allow the user to see each intermediate state in their application, as well as each iteration over the input data set, laid out in a 2-dimensional grid. There just isn't any substitute I've seen that is as accessible to the c…

I really don't think excel is accessible. I don't know how to do anything in it. If I had to make a plot I'd have to search for an article that would probably be about the same word length as the one I'd find searching "how to make a plot in python." Imo excel isn't easier for the lay person, it's just entrenched because you have people now in the workforce using excel since windows 95. It allows people to become pro…

Conversely, I am a professional programmer and I haven’t found any easier way to plot your data. What could be easier than highlight your data, insert, scatter chart?

Re: The Tyranny of Spreadsheets

#96

Post by Someone »

rvba wrote: post #59
markus_zhang wrote: post #23

In my last company we keep a spreadsheet that has tabs occupying almost the maximun number of columns allowed and in different headers. Updating that Excel file needs the full team to work for two full days each month. That was after I created a VBA script to scrape data from internal dashboards and put them directly in the spreadsheet. From my experience these monstrosities are usually created by two reasons: Push f…

Maximum number of columns is 16 384 (in modern Excel). PostreSQL has a limit of 16 000. So Excel > PostrGRE? :) I wonder what kind of data did you use and why it had to be stored in so many columns. This approach would probably kill a "real" database too.

PostreSQL has way more data types than Excel. You can have arrays, json, xml, ranges, polygons, etc. in columns (https://www.postgresql.org/docs/current/datatype.html)

Because of that, chances are the natural format for a data item taking 16,384 columns in excel likely takes fewer than 16,000 columns in PostreSQL.

Re: The Tyranny of Spreadsheets

#97

Post by eropple »

codesnik wrote: post #90
Closi wrote: post #11

Earlier quoted context omitted.

> Really, many power users of Excel ought to be looking at more capable, testable and readable solutions from professional data analysts and scientists. Python would be a good start. I can write python, but I can get the answer out of excel in a fraction of the time for most use cases. As an added bonus, excel documents are far more ubiquitous and I can share them with clients, and they can see my workings. Python is…

> they can see my workings How? Do they check that every cell in a column actually has the same formula? Do they check data format everywhere? Python script is something that is possible to be actually reviewed, and results - reproduced, and "formula's" there are actually readable. But for some stupid reason excel files are still shared over email.

It's only "stupid" if one suffers from a catastrophic lack of empathy for people who are not programmers and for the incentives to which they are exposed--which do not include using a software developer's preferred tools, nor do they include the carving out of time with which to learn them.

I would hope that we would be generally wiser than that here.

Re: The Tyranny of Spreadsheets

#98

Post by ncphil »

spoonjim wrote: post #69
asdff wrote: post #50

Earlier quoted context omitted.

I really don't think excel is accessible. I don't know how to do anything in it. If I had to make a plot I'd have to search for an article that would probably be about the same word length as the one I'd find searching "how to make a plot in python." Imo excel isn't easier for the lay person, it's just entrenched because you have people now in the workforce using excel since windows 95. It allows people to become pro…

Because the layperson I'm talking about can't think in terms of abstractions, kind of like a math student who hasn't learned algebra yet. They can only verify the correctness of their algorithm by inspecting the intermediate values at every step, and scrolling down through the input set to inspect the corner cases. The problems with this approach are extremely obvious (which is why spreadsheets error out so often, or…

This is a really important point that underscores the persistent tech education gap that exists across our economies. Mike Duncan observed some time ago that our politicians can barely handle e-mail on their own, making them particularly vulnerable to infrastructure disruption. Low and behold, we've now got an army of firms _legally trading_ in tools whose only purpose is to disrupt (or capture) infrastructure.

Re: The Tyranny of Spreadsheets

#99

Post by Gatsky »

Just as an aside, eradicating small pox really doesn't get enough press. The day the last case of small pox happened should be a global holiday celebration. It is probably the greatest achievement of humanity in the last few hundred years, and is a gift to all future humans with unbounded upside. Sure we could have done it earlier, better, with more equality etc, but we still did it and we did it everywhere. I try to remember this on the days when the world seems to be going to hell.

Re: The Tyranny of Spreadsheets

#100

Post by prosaic-hacker »

rvba wrote: post #52

The article is a nice read, but very inaccurate. Most companies do not keep their books in spreadsheets. Perhaps there are some poor souls who use Excel for bookkeeping, but this is generally impossible for any company that is audited (so basically everything bigger than some micro company). Auditors will demand to store all bookkeeping data in some ERP system - that generally allows double entry bookkeeping and many…

"Most companies do not keep their books in spreadsheets." Iceberg issue (only n% visible). Most companies do not have the self awareness to know that their ERP systems and server class database apps are shadowed by an enormous volume of spreadsheets created by their own employees. Audits and Auditors will never see them because they are not part of the official work flow.

I have seen a case where entry happened in the spreadsheet and "locally processed/validated" in some way that satisfied the clerical person. Then the now "correct" mass of data was cut and pasted from Excel into the ERP system which used a development control that worked like a crippled Excel sheet.

They did this to avoid the cryptic and most unhelpful errors from the ERP system, if the data was not ERP rules "perfect".

Post reply on HN