Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

Post reply on HN
Print view View original

51–60 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#51

Post by yold__ »

davidu wrote: post #37
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

This is the argument made against all software stack advancements. Nothing to do with industry. But when the benefits outweigh the the hurdles, change happens. And if I was starting a new insurance company (which I've considered) I'd be doing our work in code not xls, and probably python. Having RCS, Numpy, unlimited compute, unlimited storage, all gives me an advantage over my competition. :-) As to the memoization,…

"As to the memoization, that is not hard to manage in Python."

Yes it is. Recursive calls for financial calculations easily go hundreds of thousands of calls deep. This is why high-end actuarial modeling software either decomposes it into a dependency graph and unrolls function calls where possible, or just "brute-forces" it by being a thin wrapper over c++, i.e. using operator overloading on ::operator().

I've seen ill-fated efforts of capable software developers attempting to unroll the recursive function calls, and ending up with 2000 line functions that are impossible to maintain.

Re: Ditching Excel for Python in a legacy industry

#52

Post by fock »

maxerickson wrote: post #31
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

Would the people doing the implementation need to be able to choose and manage dependencies? Or could they do the work inside a prepared environment (comparable to Excel in some sense).

that costs millions!

Re: Ditching Excel for Python in a legacy industry

#53

Post by shnock »

yold__ wrote: post #45
fock wrote: post #41

Earlier quoted context omitted.

I wonder what the problem is with standarizing companywide around python@3.x, numpy@1.1x and pandas@1.x. At this point these can all be considered mature and why on earth would an org, which is not developing these packages, nor heavily consuming outside code (because they didn't with excel either in a sensible way?) decide to jump on the "but we need rrrrrollling release"-fad bandwagon?

The problem isn't feasibility, it's resources. Building and rolling out a standardized environment, and maintaining it, will cost millions of dollars. It shouldn't, but it does. And for what added benefit? The end-users don't want it, you'd have to spend another couple million for a lateral move at best. More than likely, you'll end up with a pile of Python spaghetti code that runs slower than the spreadsheet (see po…

Why does building and rolling out a standardized environment cost so much? Could you break down the requisite steps and resources required to achieve this?

Thank you, I appreciate it :)

Re: Ditching Excel for Python in a legacy industry

#54

Post by klelatti »

davidu wrote: post #37
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

This is the argument made against all software stack advancements. Nothing to do with industry. But when the benefits outweigh the the hurdles, change happens. And if I was starting a new insurance company (which I've considered) I'd be doing our work in code not xls, and probably python. Having RCS, Numpy, unlimited compute, unlimited storage, all gives me an advantage over my competition. :-) As to the memoization,…

Agree 100% with this. Better analytics can be a key competitive advantage for insurers and modern tools / cloud offer potential to be much better than Excel.

I've been working on a product that turns JupyterLab into an IDE for life insurance calculations - Python API wrapped around an optimised C / GPU computation layer underneath, all integrated with key open source libraries.

Re: Ditching Excel for Python in a legacy industry

#55

Post by alexpetralia »

ogre_codes wrote: post #20

The big problem I've always had with "programming" in a spreadsheet is by nature everything is obfuscated and difficult to trace. Yes, you can inspect a cell and see what the source for that cell is, but that might be 10 other cells and you can only really review one cell at a time. It's like a programming language where you only see one line of code at a time. Worse, those references usually aren't named. What does…

You can F9 on "A1 + SomeOtherTab:B2" (just that selection) and it'll calculate for you. You can do this for all the dependencies under Trace Dependencies.

I admit it's not perfect but I have found it much easier historically to follow a calculation through Excel than through untested pandas code (people inner join and drop rows; they groupby and lose null groups plus related rows; they filter string data without case insensitive matching, etc.).

Re: Ditching Excel for Python in a legacy industry

#56

Post by [deleted] »

ogre_codes wrote: post #35
quietbritishjim wrote: post #25

Earlier quoted context omitted.

In Excel there's a toolbar button to toggle showing formulae rather than their results. I realise this doesn't counter your overall objection, but it does mean chasing down logic isn't quite as bad as having to select individual cells one at a time.

I mean... sure? There are a fair number of ways you can mitigate these issues, but the way spreadsheets are structured does not lend itself to structured/ well managed code.

[deleted]

Re: Ditching Excel for Python in a legacy industry

#57

Post by klelatti »

meztez wrote: post #50

I'm pushing for our actuarial team to transition to more R + Git. After 3 years of preaching, most of the actuaries now use RStudio + git as their primary work tool. It is happening. What we did : 1) Provide documentation on everything from install to using internal R libraries for ETL. 2) Provide mostly problem free, always updated VMs with RStudio Server/ Shiny Server. 3) Establish an hotline channel for instant he…

I'm an actuary with a strong interest in this area - would be very interested to hear more especially on your R vs Python experience.

Re: Ditching Excel for Python in a legacy industry

#58

Post by yold__ »

fock wrote: post #52
maxerickson wrote: post #31

Earlier quoted context omitted.

Would the people doing the implementation need to be able to choose and manage dependencies? Or could they do the work inside a prepared environment (comparable to Excel in some sense).

that costs millions!

I think you are mocking me, but I'll bite.

Insurance companies are contractor heavy. They bill at $150 an hour. That's $300K annually per head. Won't take long to get a million, when you add PM overhead, information security oversight and governance, etc. Again, it shouldn't cost that much, but it does.

Re: Ditching Excel for Python in a legacy industry

#59

Post by yold__ »

shnock wrote: post #53
yold__ wrote: post #45

Earlier quoted context omitted.

The problem isn't feasibility, it's resources. Building and rolling out a standardized environment, and maintaining it, will cost millions of dollars. It shouldn't, but it does. And for what added benefit? The end-users don't want it, you'd have to spend another couple million for a lateral move at best. More than likely, you'll end up with a pile of Python spaghetti code that runs slower than the spreadsheet (see po…

Why does building and rolling out a standardized environment cost so much? Could you break down the requisite steps and resources required to achieve this? Thank you, I appreciate it :)

Up-front costs (mostly salaries, but all I.T. projects are "billable") 1.) Getting buy-in from solutions architect, software architecture, information security, I.T. management. This will be a 6 month process. 2.) Getting buy-in from actuarial management and audit. Another 6 month process.

Recurring annual costs (over 10 years) 3.) Contractor at $150 an hour = $300K annually 4.) Contractor PM at $50 an hour = $100K annually 5.) Information security compliance hoops, getting it to play nicely with the myriad of endpoint security tools, etc 6.) Ongoing maintenance and support (failed rollouts and upgrades, user desktop support, user training)

Re: Ditching Excel for Python in a legacy industry

#60

Post by techphys_91 »

mch82 wrote: post #49
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

> problem #1: Environment management Great observation. Python environment management is getting simpler, but is off putting for people without a software background. Unclear even CS majors get enough classroom exposure to package & dependency management to utilize Python efficiently. I’m more optimistic about an on-prem deployment of Jupyter Notebooks or Sage Math Cloud as a way to hide a lot of the setup complexity…

Is this really an issue for the use case being described? Environment management is obviously a significant consideration for software developers who need to keep track of versions etc, but it sounds as if these users primarily want to use the fundamental numpy functions.

They could install one of the scientific python stacks (e.g. anaconda) or just install packages globally with pip.

Post reply on HN