Ditching Excel for Python in a legacy industry • Hacker News Board

Re: Ditching Excel for Python in a legacy industry

#211

Post by everling » Thu, Dec 31, 2020, 12:10 PM UTC

mch82 wrote: post #49

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

> problem #1: Environment management Great observation. Python environment management is getting simpler, but is off putting for people without a software background. Unclear even CS majors get enough classroom exposure to package & dependency management to utilize Python efficiently. I’m more optimistic about an on-prem deployment of Jupyter Notebooks or Sage Math Cloud as a way to hide a lot of the setup complexity…

Quant here, at my firm we've deployed a JupyterHub server which provides users with a production docker image, so that analysts and portfolio managers can perform analyses without installing python, dependencies and sql drivers locally. It is working well and spurs interest in Python across the wider org - so we let everyone use it.

Similar to in OP's case, I think the real selling point of Python over Excel is advancing the capabilities and the scale of the business. Talks of different programming languages falls on flat ears in finance - show what can be done instead. With Python, Zipline and notebooks I can manage a global equity portfolio, continuously adding active strategies and adapting to real-world changes and constraints. And backtest! Excel is great, but there is an upper bound to what can be reasonably done without a thriving open source community.

Re: Ditching Excel for Python in a legacy industry

#212

Post by quietbritishjim » Thu, Dec 31, 2020, 12:11 PM UTC

ogre_codes wrote: post #35

quietbritishjim wrote: post #25

Earlier quoted context omitted.

In Excel there's a toolbar button to toggle showing formulae rather than their results. I realise this doesn't counter your overall objection, but it does mean chasing down logic isn't quite as bad as having to select individual cells one at a time.

I mean... sure? There are a fair number of ways you can mitigate these issues, but the way spreadsheets are structured does not lend itself to structured/ well managed code.

I mean... sure? I never said they did. I even specifically said I wasn't disagreeing with your overall point, in the vain hope of avoiding this redundant discussion. I was just correcting one specific inaccuracy.

Re: Ditching Excel for Python in a legacy industry

#213

Post by keithalewis » Thu, Dec 31, 2020, 12:16 PM UTC

This reminds me of Bill Gates' comment about secretaries writing VBA. Thankfully, that didn't happen. My friend rides herd on all the python code written at BigBank. He said they now have over 1MM python scripts, "Most of them written by people who had no business writing code in the first place." Python certainly lowers the barrier to writing code, but I seriously doubt many Excel power users will jump on the Python bandwagon. If you are not up to speed with the latest BI tools for Excel, you are missing out. Power Pivot is brilliant and dynamic ranges are the cat's meow. With LAMBDA there is no need to try cramming Python into Excel. Don't underestimate how much large corporations prefer to use products written by professionals instead of relying on a menagerie of ad hoc packages cooked up by well-meaning amateurs.

Re: Ditching Excel for Python in a legacy industry

#214

Post by nojito » Thu, Dec 31, 2020, 12:17 PM UTC

x87678r wrote: post #7

I do a lot of both. Excel really is great for data where there are less than say 100k rows. Its just so easy to see exactly what you're doing and what the data looks like. If you have millions of records Python really does better but I still find it frustrating to find a way to keep peeking behind the curtain. Ideally I'd have a type safe language which can embed data the way excel does. If Excel had dotnet languages…

Excel can easily handle tens of millions to hundreds of millions of rows of data.

Check out power query and power pivot.

Re: Ditching Excel for Python in a legacy industry

#215

Post by andylynch » Thu, Dec 31, 2020, 12:26 PM UTC

zinekeller wrote: post #48

mslate wrote: post #29

Earlier quoted context omitted.

From an actuary's perspective: is Google Sheets ever entertained as an Excel alternative?

Used GSuite (Google Workspace?): It's fine, it is improving but lack of shortcuts even for basic tasks (I can change the font on Word with just a keyboard, try that with Docs without a mouse). Dealbreaker is the 5 million cell (not row nor column) limit, which is even lower than Microsoft's old limits (more than 15 million cells, which was increased in 2007 to you-have-a- serious -problem-if-you-somehow-fill-this-lim…

Not to mention that if you're using Power Query, Excel's 1 million row limit doesn't apply in the workbook queries either.

Re: Ditching Excel for Python in a legacy industry

#216

Post by [deleted] » Thu, Dec 31, 2020, 12:28 PM UTC

wendyshu wrote: post #23

gerdesj wrote: post #16

Earlier quoted context omitted.

"analysts" - lol! I know a HR director at a multi-national. He'd had enough of Excel and liked the look of this Python thing. I showed him R as well for balance but he wanted Python. I showed him how to install a Python distro and MS Code on his Windows machine, wired them up and off he went a few months back. The board are in awe of his presentations. He is not an IT bod at all but a Uni. degree in Psycho. involves…

What's a bod?

[deleted]

Re: Ditching Excel for Python in a legacy industry

#217

Post by grvdrm » Thu, Dec 31, 2020, 12:29 PM UTC

hermitcrab wrote: post #174

Lots of discussion here of Python and R as alternatives to Excel. But no mention of drag and drop data processing tools such as Alteryx, Knime or Easy Data Transform. Are these not a more natural alternative to Excel for people who don't have a programming background?

I used Knime in my last company. Very interesting software. But I think the initial introduction can be tough to carry forward. You need a champion or two in your org to help everyone else. Excel is a tool enough people know to both work independently and be dangerous.

Re: Ditching Excel for Python in a legacy industry

#218

Post by vimy » Thu, Dec 31, 2020, 12:50 PM UTC

SoSoRoCoCo wrote: post #114

nl wrote: post #102

Earlier quoted context omitted.

I think the 2.x vs 3.x issues have mostly been resolved by now. I don't think I've hit one for a long time, and even StackOverflow answers are more likely to be Python 3 now.

macOS still ships with Py2.7 and has dependencies, and npm-gyp only recently switched to 3.x. same with python SDR. it depends what you use: less popular packages are still languishing. but that discounts the tens of thousands of projects that are already out there that are in use and need conversion. it'll take probably 3-5 years for it to really go away.

As of macOS 11.0 there is no python anymore. You have to download it.

Re: Ditching Excel for Python in a legacy industry

#219

Post by etothepii » Thu, Dec 31, 2020, 1:14 PM UTC

fnord123 wrote: post #202

etothepii wrote: post #183

Earlier quoted context omitted.

This is the real problem. It's not with excel per se, but the complete lack of automated testing and source control.

Python works with source control and tests so this would indeed be a problem with Excel per se.

But Python doesn't have to, I've seen plenty of python with 1000 line functions, no tests and no source control. It's particularly common in Jupyter Notebooks.

Re: Ditching Excel for Python in a legacy industry

#220

Post by m101 » Thu, Dec 31, 2020, 1:25 PM UTC

shp0ngle wrote: post #154

This is actually about using a paid, closed-source add-on PyXLL, that integrates python into Excel, but only to the Windows version. And costs 25 USD per month (but has a free trial). Excel is never actually ditched. edit: oh, that's only step 4. Step 5 is actually ditching Excel.

I've used xlwings at work but I'm mired by packaging issues and things like the CMD window disappearing for some users but not others.