Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I was wrong about spreadsheets (2017)

reifyworks.com

Post reply on HN
Print view View original

121–130 of 378 posts

Re: I was wrong about spreadsheets (2017)

#121

Post by pidg »

JohnJamesRambo wrote: post #97
keyle wrote: post #72

Earlier quoted context omitted.

Meanwhile open any document in Photoshop with a few layers and effects and the computer grinds to a halt, no matter the specs, no matter the year, through the ages. New hardware comes out? Booya, new Photoshop XYZ -> let's put your fancy hardware on its knees, begging for air.

while paying a monthly fee...

The monthly fee for Photoshop is around 1/100th of the old retail price. That works out at about 8 years of use before you have to 'buy' the software again. Seems fair to me.

And for those who don't like paying for Photoshop - which, given it's an astonishingly powerful piece of software, probably means "people who don't actually need Photoshop" - there's always cheap or free alternatives that provide about half the functionality.

Re: I was wrong about spreadsheets (2017)

#122

Post by simonh »

I used to work at a bank that built a huge integrated application development and deployment stack on Python. One aspect of it was a custom library implementing a Directed Acyclic Graph, basically a way to ink up properties so that when the results of one property changed, any dependent properties on the DAG would automatically recalculate. This was explicitly implemented so that we could directly translate the logic from Excel sheets into Python code.

A lot of our projects were built to accept an excel sheet as input, then generate another Excel sheet as the output. The idea was the logic and data manipulation and transformation should be in source control, but the Quants would still have Excel available to build graphs, pivot tables and do statistical analysis on the results.

The DAG also turned out to be pretty handy for implementing web applications and all sorts of other apps though.

Re: I was wrong about spreadsheets (2017)

#123

Post by hef19898 »

halfeatenpie wrote: post #26

I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…

So true, I used to run European cross-docking ops, planning, inventory, forecasting, scheduling, in Excel back the day. Successfully scaled by 100% YoY for three years. Was quite a lot of fun.

Issue I have with that approach is not using excel per se. It is using Eycel in addition to what ever system is being used the first place (SAP for example is a pretty popular thing to circumvent with Excel sheets). That and over engineering spreadsheets to the point where nobody but the creator can use them anymore. At that point you data just gets suspect. And circumventing existing systems with local offline spreadsheets just screws up everything. Now combine these two.

But that has less to do with spreadsheets themselves, see my fist point regarding cross docking, and more with the application. Used correctly spreadsheets can be incredibly powerful. And at least for my purposes I have yet to encounter an analysis issue I failed to dig through using Excel. Sure, something like Python might have worked better but consider me to be an empowered user (read: I have no clue about Python or SQL or...). And at least I knew I could trust the way the analysis was done as intended.

Re: I was wrong about spreadsheets (2017)

#124

Post by hef19898 »

mirimir wrote: post #93
MooMooMilkParty wrote: post #55

Earlier quoted context omitted.

You're blowing my mind here a little bit. I don't use MySQL but do use Excel and the R/Python data science stacks quite a bit. I regularly open data with tens or even hundreds of thousands of columns in the latter by using lazy computations. Is this not the case with database approaches? What kind of hardware are we talking? Presumably Excel has to load everything into memory at once since it's immediately viewable,…

In MySQL, maximum row size for non-text columns is 65,535 bytes. MySQL is limited to 4096 columns per table, and 1017 columns per table if the InnoDB engine is being used. Excel's limits are 16384 columns and 1048576 rows. I'm talking wimpy hardware here, I admit. Basically, VirtualBox VMs on a quad-core i5 box with SSD and 8GB RAM. With the VM having three cores and 6GB RAM. But it was the same wimpy hardware for Wi…

Oh, the day Excel got rid of the 64k line limit! Was a very sweet day!

Re: I was wrong about spreadsheets (2017)

#125

Post by fock »

xvilka wrote: post #53

There is LibreOffice Calc, which allows you to program it with Python[1]. Best of the both worlds. I recently created a suggestion [2] to support also Julia language in it. Hopefully they will consider it. [1] https://wiki.documentfoundation.org/Macros/Python_Design_Gui... [2] https://bugs.documentfoundation.org/show_bug.cgi?id=125728

did you ever "program" in LibreOffice with Python. Last time I checked, the API were veeery thin wrappers around the Java-API itself - I guess that this is hindering adoption (and not not having bindings to X)

Re: I was wrong about spreadsheets (2017)

#126

Post by quibbler »

Maybe it works because it ofyen doesn't matter.

In cases where Data Science is a replacement for astrology ("just tell me some reassuring mumbo jumbo to alleviate the burden of decision making), the inevitable bugs may be irrelevant.

I appreciate the theoretical power of Excel. I just don't see how keeping logic in Spreadsheets bug free could be possible. The logic is hidden away and hard to get to. It is already hard enough to debug classical code. Spreadsheets seem impossible.

Re: I was wrong about spreadsheets (2017)

#127

Post by Xelbair »

halfeatenpie wrote: post #26

I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…

I just hate one single thing about Excel - all function names are localized - and in case of my native language - they are horrible and inconsistent.

Re: I was wrong about spreadsheets (2017)

#128

Post by beefield »

LeonB wrote: post #103
function_seven wrote: post #90

Earlier quoted context omitted.

Unfortunately the answer is "no" to both #1 and #2. It does do something close to those. Column data types are automatically carried over to the next row when inserting or appending rows to an existing table. But you're free to override it if you want for a particular cell Same with formulas. When you enter a formula in one cell, Excel will automatically copy across the whole column. But you're able to undo that auto…

The warning you receive when a formula is not consistent is a nice feature, and an advanced user will quickly learn to be very sensitive to those warnings, will spot them quickly, and avoid causing them when possible. To strictly enforce that the formula must be the same in each cell of the column would not be very excel-like, I can't see it being done.

> advanced user

Yep, the problem is that I have this conspiracy theory that Microsoft has designed Excel to be the ultimate booster of Dunning-Krueger effect so that most people think they are advanced users [1] while they actually have no clue what they are doing. All the while giving no protection whatsoever against those Dunning-Krueger cases.

[1] I am very much afraid I belong to this group.

Re: I was wrong about spreadsheets (2017)

#129

Post by d__k »

kyllo wrote: post #61
mirimir wrote: post #33

Earlier quoted context omitted.

On the same hardware, with the same table, Excel in Windows 7 does a cross-tab faster than MySQL in Ubuntu does. Indeed, MySQL chokes if there are more than a few hundred columns, but Excel just keeps going. And it uses all CPU cores. Edit: In case anyone is wondering why I did that, I wanted a simple visualization of ping-location results for thousands of IPv4 from several hundred ping-probe locations. So that meant…

The other thing people don't realize about Excel is that it has a compressed, in-memory, columnstore analytics database built into it. It was called Power Pivot then Power Query, not sure if it changed names again, but it's actually a SQL Server Analysis Services Tabular Model, the same thing that's in Power BI as well. It's going to perform much better than a row-oriented RDBMS at aggregating columns because that's…

It was a separate data processing engine within Excel developed by another team (SQL server) for the purposes of self service BI (Power Pivot/Query). Most probably, this engine is not used for normal (traditional) calculations.

Re: I was wrong about spreadsheets (2017)

#130

Post by zaarn »

mirimir wrote: post #33
halfeatenpie wrote: post #26

I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…

On the same hardware, with the same table, Excel in Windows 7 does a cross-tab faster than MySQL in Ubuntu does. Indeed, MySQL chokes if there are more than a few hundred columns, but Excel just keeps going. And it uses all CPU cores. Edit: In case anyone is wondering why I did that, I wanted a simple visualization of ping-location results for thousands of IPv4 from several hundred ping-probe locations. So that meant…

Might be worth a try to look into Postgres. In case of the column problem, PG has a limit of 8 Kilobytes for Column Width (instead of a count of columns). You can circumvent some of the limits by using arrays, types, json or hstore. It also has pl/Python which lets you code up queries containing python fragments for data processing.
Post reply on HN