Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

Post reply on HN
Print view View original

201–210 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#201

Post by etothepii »

raverbashing wrote: post #176
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

Those are fair criticisms but I think under Windows, Docker+pip is the worse way of managing it 2 and 4 are surprising, it would be interesting to do a benchmark and maybe figure out the best way to do stuff in Python for your case About 3, I suppose that's why developers should break up complex expressions (and not only in Python)

Its absolutely true that an untested 1000 line python function is no better than a 1000 line untested VBA function.

Re: Ditching Excel for Python in a legacy industry

#202

Post by fnord123 »

etothepii wrote: post #183
Mauricebranagh wrote: post #179

Earlier quoted context omitted.

And you find a bug that costs you $$$$, I worked for a company that used an opaque excel spread sheet as a part of its accounting system - turns out there where bugs and we found a massive short fall one of the contributing factors in the collapse of the company.

This is the real problem. It's not with excel per se, but the complete lack of automated testing and source control.

Python works with source control and tests so this would indeed be a problem with Excel per se.

Re: Ditching Excel for Python in a legacy industry

#203

Post by arendtio »

> We used an Excel spreadsheet-based model with dozens of tabs containing complex formulas, endless pivot tables and unintelligible VBA code.

> The tangled mess of VBA was re-written into independent Python modules, each of which performs a distinct function.

Hardly a valid argument. I mean, VBA supports using modules too, so it basically comes down to an increased skill of the programmer and maybe more knowledge about the actual scope as it is a rewrite, but there is no reason why it could not be done with VBA.

Re: Ditching Excel for Python in a legacy industry

#204

Post by fock »

macintux wrote: post #83
techphys_91 wrote: post #60

Earlier quoted context omitted.

Is this really an issue for the use case being described? Environment management is obviously a significant consideration for software developers who need to keep track of versions etc, but it sounds as if these users primarily want to use the fundamental numpy functions. They could install one of the scientific python stacks (e.g. anaconda) or just install packages globally with pip.

Things always break over time. I have a very technical co-worker, a systems admin, with a broken Python stack on Windows. No idea what's wrong.

so he did a chmod -w on their folder and it broke. Well, that's a feat I still have to work out!

Re: Ditching Excel for Python in a legacy industry

#205

Post by noir_lord »

trimbo wrote: post #63
SoSoRoCoCo wrote: post #30

Earlier quoted context omitted.

It is great that you respond with succinct reasons. People who have used Python for some years seem to forget just how clunky it really is. I've been using Excel since 1990, and sure, it has its own warts, but Python is a very rudimentary tool compared to Excel. Python is a machine shop. Excel is a car. It may be a lemon, but it's a functional car. This is a great example of programmers not being able to see the fore…

> Reminds me of the "Once Linux gets a desktop it will take over the world" debate from circa 1997-today. Linux did take over the world, just not on the desktop. It was on servers and mobile, which now have more users than desktops or laptops (edit: servers via the web). Technology gets its warts fixed when it grows along an explosive new market, especially if the market ends up being larger than the last. Python is…

> Linux did take over the world, just not on the desktop.

I'm honestly not sad that it never happened.

It took over my desktop around ~1998 and I wonder if massive adoption of Linux on the desktop would have benefited me or been worse (from my perspective).

As it stands literally every single tool I want/need to do my job is already available for Linux and indeed many of those tools are simply better on Linux (docker on a mac is horrible, I have a work issued current gen macbook pro, I use it purely for testing docker set-ups and then it goes back in its case).

Its going to sound elitist but not dumbing down the platform for the average user is a benefit to me.

Re: Ditching Excel for Python in a legacy industry

#206

Post by bash-j »

ineedasername wrote: post #96

I work somewhere that was stuck in even more simple excel usage, like sorting a list and counting rows for each type of category instead of using a pivot table. I've done some more advanced work with python and xgboost for some modeling, but the biggest improvements in terms of both time saving and regular use of data for informed decision making has been implementing basic reports and dashboards. So much so that som…

I know your pain all too well. I hate when people say stuff like 'Did you go to Harvard?' after I help them print a document. I knew managers who didn't know what meant. They think creating a pivot table is genius work. I actually heard someone say you don't have to put .au on the end of the email address if you live in Australia. I sat next to one guy in a meeting who was typing away on his laptop keyboard without looking at the screen for about 5 minutes and then used spell check to fix every second word. These are the people running these old corporations.

Re: Ditching Excel for Python in a legacy industry

#207

Post by unixhero »

sam_bristow wrote: post #113
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

Funny story about Excel on corporate machines. A couple of years ago the company I work for got boight by an Italian company. When we finally migrated the Windows users over to the corporate Office installs a bunch of people found that Excel wouldn't work for them. Things like sum(A1:A20) were syntax errors. After a bunch of digging i worked out that the localisation from corporate meant they suddenly had Italian fun…

Localisation is where you first have to debug Excel formula errors. Is it colon, or semicolon; it responds to, is my goto method to approach the issue.

Re: Ditching Excel for Python in a legacy industry

#208

Post by 1337shadow »

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

I thought the point was to make a web interface in Python to query some database and process to calculations on a powerful server, in which case the database and server for sure are going to be faster than excel, including the WITH RECURSIVE SQL statement, and only a browser is necessary for users which makes the solution not only multi platform but also remote-friendly. As such, your post really makes me wonder what they are trying to do at all.

Re: Ditching Excel for Python in a legacy industry

#209

Post by nojito »

Foobar8568 wrote: post #180
gerdesj wrote: post #16

Earlier quoted context omitted.

"analysts" - lol! I know a HR director at a multi-national. He'd had enough of Excel and liked the look of this Python thing. I showed him R as well for balance but he wanted Python. I showed him how to install a Python distro and MS Code on his Windows machine, wired them up and off he went a few months back. The board are in awe of his presentations. He is not an IT bod at all but a Uni. degree in Psycho. involves…

And I have seen a board member not wanted anything but linear regression from his own data scientist team (If I remember well, they were like 5~ PHD or master in stats) because he couldn't understand anything. And that was in one of the largest organization in the world.

Data scientists are useless if they don’t have the right communication skills to empower decision makers.

This is why linear regression will always be king and people who know how to turn complex problems into linear problems are worth millions.

Re: Ditching Excel for Python in a legacy industry

#210

Post by dataflow »

sam_bristow wrote: post #163
mehrdadn wrote: post #152

Earlier quoted context omitted.

> There are ways of building spreadsheets that are reliable and auditable. Then there's how 95% of people do it Do you have any pointers to learning materials on how to do this? Would be interested in reading more on it.

This Twitter thread is a good start: https://mobile.twitter.com/keith_ng/status/13079610874515251... There also Joel Spolsky video I linked in another comment: https://youtu.be/0nbkaYsR94c I don't actually use Excel much so others might have better resources.

Oh I've already seen the Spolsky video, but I seem to recall for the most part (except for using tables, which is great) it's about how to use Excel effectively, not about how to produce maintainable sheets. The Twitter thread looks good though. Thanks!
Post reply on HN