There was a new app posted here a few weeks ago that combined a Jupyter notebook with a spreadsheet-like automatic recalculation. Does anyone here know what it's called?
You mean https://observablehq.com/ ?
Yes, that's it. Thanks!
I was wrong about spreadsheets (2017)
71–80 of 378 posts
Re: I was wrong about spreadsheets (2017)
#72I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…
On the same hardware, with the same table, Excel in Windows 7 does a cross-tab faster than MySQL in Ubuntu does. Indeed, MySQL chokes if there are more than a few hundred columns, but Excel just keeps going. And it uses all CPU cores. Edit: In case anyone is wondering why I did that, I wanted a simple visualization of ping-location results for thousands of IPv4 from several hundred ping-probe locations. So that meant…
Meanwhile open any document in Photoshop with a few layers and effects and the computer grinds to a halt, no matter the specs, no matter the year, through the ages. New hardware comes out? Booya, new Photoshop XYZ -> let's put your fancy hardware on its knees, begging for air.
Re: I was wrong about spreadsheets (2017)
#73Earlier quoted context omitted.
Has Access solved the corruption issue that plagued it for tens of years? While you can connect it to a "real" database engine trivially, a common use-case is via a single MDB file that could become damaged. You could damage the MDB a number of ways including: - Leave Access running before shutdown (or power loss) - Intermittent network outage to the file server - Multiple users trying to access the same MDB (or anti…
Access and FileMaker made sense before the advent of open source, specifically Rails. Django, Postgresql, and MySQL. Both products make little sense now.
> Rails. Django, Postgresql, and MySQL
These are much harder for non-programmers to use than Access. With Access, almost completely non-technical people can set up their own database and make the queries they need to answer their own questions. With Postgres accessed from a general-purpose programming language, non-technical people need to hire someone to help them with every basic task.
As far as I can tell Access has nothing to do with making websites, so it’s unclear what Django or Rails has to do with anything.
Re: I was wrong about spreadsheets (2017)
#74I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…
But how do you handle sharing? The main issue with Excel, and its main downfall, is the sharing of data and versioning, with person X has an older version of person Y by a few hours and needs an email of person X to keep doing her job.
Re: I was wrong about spreadsheets (2017)
#75One major drawback of using spreadsheets is that it is harder to spot calculation mistakes. A famous/infamous recent example is the Reinhart and Rogoff incident in 2013, which involved their heavily cited paper "Growth in a time of debt". When a research student named Thomas Herndon tried to replicate the paper's findings, he found, after a careful inspection of the Excel spreadsheet the two Harvard professors used, that the spreadsheet was ridden with errors. Some of them were even in principle rather trivial, such as mistaking the sum of one column as the sum of another. The incident was widely reported in many business newspapers. Paul Krugman had also commented on it on New York Times.
Had the steps of computations been written as a program, it might be easier for the authors to discover their mistakes. With a spreadsheet, if you put formulae inside data cells, you need to click all these cells one by one to see if the formulae have been input correctly. This is tedious.
Re: I was wrong about spreadsheets (2017)
#76I worked with excel for some years early in my career. I did not know how to program back then beyond if else and for loops.
We used it for budget estimation at multi million / up to 100 m. I have seen other uses in Corp world in many departments.
IMHO I think it’s good for specific use cases. But it is routinely abused beyond that.
The key shortcomings for excel uses by non-programmers for key business applications:
- How do you test that your calculation is correct after update. The same argument holds for business applications without tests?
- how do you maintain the knowledge of the inner workings without having to follow the arrows around cells. Probably there are good practices but how likely are you to ensure them in your product / team.
- the data / computation is spread in user space. In case your app is useful many people would like to use it. How do you manage the updates and bug fixes beyond it’s the user’s Responsibility
- how do you avoid the black box effect: mission critical software that no one knows how to touch inside without a full rewrite?
- how do you convince stakeholders that they need to migrate to an adapted solution while they have a working one now and often are oblivious to the hidden costs of users copy pasting data / reformatting for hours sometimes to fit an existing tool that is no longer adapted.
The data frame structure in R or python solves a large portion of these issues. Yes you need training. It’s the same for excel if you want to avoid in inferno machine case you need the same concepts.
Why not train for programming with data frames and have the option to gradually extend to web app without a full it project that needs 3 levels of validation.
Re: I was wrong about spreadsheets (2017)
#77My issue with spreadsheets is that they could be improved a lot with minimal changes, not nobody seems to do it in popular office packages. Making them more database-like and making table data first-class (at least you can make named tables in excel on windows) could be used to push people a bit more towards organised data, without changing how anything works. Half of the mess that makes excel hell comes from the fac…
I'd recommend watching this YouTube video; it's only one person's take on the right way to use Excel, but Excel has many ways to handle these issues: https://m.youtube.com/watch?v=0nbkaYsR94c
That's a classic, definitely to be watched!
Re: I was wrong about spreadsheets (2017)
#78This is what I am currently working on: Making programming fluid and easy like spreadsheets (or alternately, elevating the niche occupied by spreadsheets into something that can be called "real programming"). (If this is your jam, hit me up.)
Me too! What are you working on?
Re: I was wrong about spreadsheets (2017)
#79My issue with spreadsheets is that they could be improved a lot with minimal changes, not nobody seems to do it in popular office packages. Making them more database-like and making table data first-class (at least you can make named tables in excel on windows) could be used to push people a bit more towards organised data, without changing how anything works. Half of the mess that makes excel hell comes from the fac…
I'd be over the moon to see one change to Excel. Native support for a language other than VBA. Perhaps the CLR. Perhaps Java/Typescript. Just something (optional) for programmers who want to use Excel and not want to deal with the garbage that VBA is. Some workarounds exist, but afaik they require collaborators to also have the tool install, which is dead in the water. I know MS has considered it, I'm still pretty su…
MS hasn't just considered it. They have done it. This is called Visual Studio Tools for Office.
https://en.wikipedia.org/wiki/Visual_Studio_Tools_for_Office...
Re: I was wrong about spreadsheets (2017)
#80I worked as a spreadsheet DevOps in a financial company. We used to have a python script to spin up windows vms with the exact version of windows that was known to be good, with the exact version of excel and then automatically kickoff the computation.
The reson for this was someone once got the order of execution wrong and screwed up a couple billion dollars worth of trades.
Excel is great until you have to maintain it. Same reason why you don't let people build bridges with Lego.