Live data from Hacker News

Use Spreadsheets Everywhere

simplethread.com

31–40 of 123 posts

Re: Use Spreadsheets Everywhere

#31
post #8

Spreadsheets versus programmers is a war that can be much more peaceful. In my work, I have found that, especially in data science, the spreadsheet user and the programmers are often trying to accomplish similar tasks, but the “language barrier” between them leads to much more fragmented workflows. I also think this article does a good job of identifying spreadsheets as a low-code programming language — spreadsheets…

Mito looks very interesting. I couldn't find pricing info. Is this open source?

Re: Use Spreadsheets Everywhere

#32
We need a spreadsheet format that's more conducive to the professional programmer workflow.

Parsing them is actually very complex. You just have to hope your library can handle everything in excel. If you're a cli native I'm not really sure what you do.

Spreadsheets don't play very will with source control. If the underlying format was text and every cell and formula was on a new line it would work out ok. As it is now, merging them is very cumbersome....impossible for the laymen that might be the main user of the sheet.

By going with a speadsheet and not something like SQL, you lose a lot of rigor.

There's probably a few more needs but these are top of mind for me.

Re: Use Spreadsheets Everywhere

#33

Dang, what software engineers hate spreadsheets? I've never worked anywhere that didn't use them for project planning and other such stuff.

I do – but I'm an academic, not a software engineer. Spreadsheets are abused and don't make a formal distinction between analysis and data. I've seen many horrible, horrible things happen because people used Excel when they really, really shouldn't have. Intelligent biologists reinventing numerical integration (badly) in Excel, for example (with huge floating point errors, often comparable to the size of the change they are looking for biologically).

Re: Use Spreadsheets Everywhere

#34
Article covers a great point, just does not provide a framework to make decisions regarding when to use spreadsheets.

I am a CFO that has used spreadsheets my entire career (30+years). They can definitely have a vital role in any company and can also cause incredible problems.

A few thoughts that were not mentioned in the article:

   1) in many companies the users of spreadsheets are not allowed to use anything more powerful than spreadsheets (or maybe MS Access which causes more problems).  To get a SQL/Python (or other programming language) solution requires submitting a request to IT and then waiting two years to work up the priority queue.  This is not a complaint against IT and the priority process.  The reality is most spreadsheets do not start out as important enough to warrant high priority.  
   2) Spreadsheets are great containers.  Not only can they hold a model, but also the raw data, and even emails regarding the model/inputs/etc.  This makes them very handy for keeping everything in one file regarding an analysis.
   3) IME, most significant spreadsheets get rewritten as they evolve, especially when they change hands.  This is both good and bad.  While the rewrite allows correction of built up badness, it is rarely done with the discipline one sees with refactoring in more traditional code.  Moreover, when a hand off happens, the rewrite is generally done because the recipient does not fully understand how the existing model works.
   4) Large spreadsheets cannot be effectively audited outside of locking/password protecting non-input cells. Very rarely is this actually done.  As a result, you get the scenario the author describes of hard-coded values in cells that once contained formulas.  These are almost impossible to find in worksheets with large numbers of tabs and thousands of formulas.  
  5) Spreadsheets generally do not use true version control.  For most of my career, one simply used "File Save As" to create a new version.  In the early days it was due to a fear Windows would crash and you would lose your work.  Later, it became a way to step through changes and determine the impact of the change.  While this method provides some ability to go back to a prior state, people really do not track properly what each version contains and what are the true differences (would love to find a way to use Git effectively on spreadsheets).
   6) Most spreadsheets do not have test cases.  They certainly do not increase the test case universe as modifications are made.  As a result, cannot tell if things break when changes are made.  
Given all of these things, I recommend using spreadsheets for ad hoc analyses and to pilot analytics that will become part of a routine process. The latter point is the point the author is making (and the point of his title). I would add that the analysts need a plan and methodology to determine when they need to convert from the anything-goes freedom of spreadsheets to a more disciplined, maintainable, controlled program.

Re: Use Spreadsheets Everywhere

#35
post #19

I have a love hate relationship with spreadsheets. While in University I was a bit of an Excel Monkey, I knew everything there was to know and I prided myself in not having to use my mouse to make complex models... Now that I worked at a major bank, learned to code and moved to the data science side I despise spreadsheets. Most people don't realize this but out entire banking system is built on a couple excel models…

Maybe we need spreadsheets where each cell can embed python or something.

Re: Use Spreadsheets Everywhere

#37
post #28
post #5

Article raises some valid points but how do you catch the point where the spreadsheet gets out of control? Maybe MS could build in a warning system and then suggest some potential solutions. Minor quibble: it's an Apple III in the ad, not an Apple ][ as the post says. I remember running Visicalc on my Apple ][ but it wasn't until Excel appeared that I realized how seductive spreadsheets can be.

Clippy: I see you're trying to create a database. Would you like to use Access instead? I'm not sure this is really a problem Microsoft can solve as an Excel feature. I think the better approach would be for MS to identify common ways in which people misuse spreadsheets and create new tools to accommodate those use cases. With how commonly Excel is misused as a database, I think there's a market for an alternative to…

Excel can connect to a Jet DB, IIRC. If it can connect, it could probably also recognize common patterns and create some common Jet/Access structure to replicate the Excel structure on the fly. Basically, transparently use Access-like structure as the original source.

Just thinking off the top of my head. But... if that would be useful/doable, they may have already done it.

Re: Use Spreadsheets Everywhere

#38

I agree with a lot of the points raised here. I think many of the problems with spreadsheets are due to the software rather than the users. As mentioned in the article, its hard to slowly iterate from a small manageable spreadsheet to an larger software solution. For example, Excel would be a lot more usable and maintainable for me if there was a way to make a special "data sheet" in which data types are forced to be…

This article is dead on from an analysis/criticism/insight: devs are called in when it's a sinking ship and don't see all the cargo the ship has hauled.

Why hasn't microsoft or someone taken the basic spreadsheet model to a shared-database scalable one? The UI is basically set at this point.

A naive schema (filename, tab, x, y, value) is what an excel sheet is. It's not like we are dealing with "impedence mismatch" and even shared editing can be reasonably handled with database transactions (or RAFT if you want to get really big/distributed)

I think the lack of this is a sign of Microsoft Office completely owning the space and not wanting to innovate at all. And the huge amount of effort it would take to replicate excel-level operations in a database application server is nontrivial.

But man, you could have an API for doing excel operations against a database schema, and export to excel...

And as you said, you could do lots of schema based options in databases that aren't natural to excel.

Re: Use Spreadsheets Everywhere

#39

I agree with a lot of the points raised here. I think many of the problems with spreadsheets are due to the software rather than the users. As mentioned in the article, its hard to slowly iterate from a small manageable spreadsheet to an larger software solution. For example, Excel would be a lot more usable and maintainable for me if there was a way to make a special "data sheet" in which data types are forced to be…

Tables are probably the most overlooked feature of Excel.

Why use tables?

* Each column is uniquely named - no more wondering if you are referencing the right cell, no more thinking about "to $ or not to $"

* The table's rows and columns are reliably discovered by pivot tables - no more wondering if the entire dataset is referenced by the pivot

* New columns that are formulas are automatically applied to every row

* Tables have names, so it is easy to understand which table a pivot is referencing

The true, reliable and sane power of excel lies in Tables + Pivots + Charts. If you drive most of the problem solving into those paradigms you will keep hair!

Re: Use Spreadsheets Everywhere

#40

I agree with a lot of the points raised here. I think many of the problems with spreadsheets are due to the software rather than the users. As mentioned in the article, its hard to slowly iterate from a small manageable spreadsheet to an larger software solution. For example, Excel would be a lot more usable and maintainable for me if there was a way to make a special "data sheet" in which data types are forced to be…

I think Airtable is starting to get us there, but it's a long way to go.

Something that combined Airtable, excel, and maybe a more userfriendly (and more restricted) version of darklang for defining formulae could be really slick.

Post reply on HN