Ditching Excel for Python in a legacy industry • Hacker News Board

Re: Ditching Excel for Python in a legacy industry

#41

Post by fock » Thu, Dec 31, 2020, 12:37 AM UTC

SoSoRoCoCo wrote: post #30

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

It is great that you respond with succinct reasons. People who have used Python for some years seem to forget just how clunky it really is. I've been using Excel since 1990, and sure, it has its own warts, but Python is a very rudimentary tool compared to Excel. Python is a machine shop. Excel is a car. It may be a lemon, but it's a functional car. This is a great example of programmers not being able to see the fore…

I wonder what the problem is with standarizing companywide around python@3.x, numpy@1.1x and pandas@1.x. At this point these can all be considered mature and why on earth would an org, which is not developing these packages, nor heavily consuming outside code (because they didn't with excel either in a sensible way?) decide to jump on the "but we need rrrrrollling release"-fad bandwagon?

Re: Ditching Excel for Python in a legacy industry

#42

Post by yold__ » Thu, Dec 31, 2020, 12:38 AM UTC

mslate wrote: post #29

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

From an actuary's perspective: is Google Sheets ever entertained as an Excel alternative?

It's been a couple years since I've used it, and I didn't feel it was a comparable alternative. It's decent for about 80% of spreadsheet users, but the keyboard shortcuts were lacking and it was missing some functions that I rely on.

For keyboard shortcuts, most Excel power users don't use the mouse, so while it sound trivial, it's really hard to feel productive when you have to hunt around for the right button to click.

From an enterprise perspective, Excel is so entrenched it would be a 5-10 year effort to port existing spreadsheets to sheets. Practically speaking, most companies wouldn't see the benefit.

Re: Ditching Excel for Python in a legacy industry

#43

Post by Closi » Thu, Dec 31, 2020, 12:40 AM UTC

konjin wrote: post #39

Closi wrote: post #33

Earlier quoted context omitted.

How would the conversion rate between EUR and USD be determined?

My first job was cleaning up the mess that was caused by hard coded yahoo finance urls in spreadsheets. One of them died, no one noticed and it cost the company millions of dollars in bad trades over three months.

You could automatically import a table from the web into the sheet, or alternatively create some custom data types for currency conversion.

Re: Ditching Excel for Python in a legacy industry

#44

Post by bernardv » Thu, Dec 31, 2020, 12:45 AM UTC

Loved the post. I spent more than a year trying to pull a prominent reinsurer on the rock, out of spreadsheet hell, and into the modern age. Baby step #1 would have been to transfer critical data to a database environment and baby step #2 would have been to extract business logic and very-poorly written VBA code into an external code library (Python) that is source-controlled and auditable... I can still hear the Chikadees laughing at me. Last I heard said firm was still deep in spreadsheet hell.

Re: Ditching Excel for Python in a legacy industry

#45

Post by yold__ » Thu, Dec 31, 2020, 12:45 AM UTC

fock wrote: post #41

SoSoRoCoCo wrote: post #30

Earlier quoted context omitted.

It is great that you respond with succinct reasons. People who have used Python for some years seem to forget just how clunky it really is. I've been using Excel since 1990, and sure, it has its own warts, but Python is a very rudimentary tool compared to Excel. Python is a machine shop. Excel is a car. It may be a lemon, but it's a functional car. This is a great example of programmers not being able to see the fore…

I wonder what the problem is with standarizing companywide around python@3.x, numpy@1.1x and pandas@1.x. At this point these can all be considered mature and why on earth would an org, which is not developing these packages, nor heavily consuming outside code (because they didn't with excel either in a sensible way?) decide to jump on the "but we need rrrrrollling release"-fad bandwagon?

The problem isn't feasibility, it's resources. Building and rolling out a standardized environment, and maintaining it, will cost millions of dollars. It shouldn't, but it does. And for what added benefit? The end-users don't want it, you'd have to spend another couple million for a lateral move at best. More than likely, you'll end up with a pile of Python spaghetti code that runs slower than the spreadsheet (see point #4 about massively recursive calcs).

Re: Ditching Excel for Python in a legacy industry

#46

Post by fock » Thu, Dec 31, 2020, 12:47 AM UTC

kqvamxurcagg wrote: post #27

bash-j wrote: post #19

Earlier quoted context omitted.

This has frustrated me as a python user for the last 7 years, working as the only python user in business environments dominated by Excel. People will say things like, "if you leave, who can support this report you made in python?" Well I say, who can support the bloated 40mb spreadsheet that would take forever to unpick and figure out how to update with new data? No one can, because I've seen people would rather reb…

I've used excel and python in lots of business contexts. For most tasks involving domain experts, excel usually wins hands down. An excel spreadsheet is usually easily auditable. The visual presentation and layout lends itself to review by others. You can click and point at values. Python and other programming languages require an environment and tooling that can't be easily supported across the enterprise. It requir…

how does Python require source control? I guess you can just drop your files on a sharedrive... just like ... EXCEL!

Also you reference an existing report? What's exactly the problem of serializing your data after a run of your python program? Actually, if you think this through, you would probably establish some pipeline architecture to just continously integrate your results.

Re: Ditching Excel for Python in a legacy industry

#47

Post by konjin » Thu, Dec 31, 2020, 12:48 AM UTC

Closi wrote: post #43

konjin wrote: post #39

Earlier quoted context omitted.

My first job was cleaning up the mess that was caused by hard coded yahoo finance urls in spreadsheets. One of them died, no one noticed and it cost the company millions of dollars in bad trades over three months.

You could automatically import a table from the web into the sheet, or alternatively create some custom data types for currency conversion.

Or you could act like you're responsible for tens of millions of dollars and hand it over to someone who can make sure it doesn't blow up.

"We don't need to hire an electrician to wire up the office, I did my garage using uninsulated wires and it works perfectly!"

Re: Ditching Excel for Python in a legacy industry

#48

Post by zinekeller » Thu, Dec 31, 2020, 12:49 AM UTC

mslate wrote: post #29

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

From an actuary's perspective: is Google Sheets ever entertained as an Excel alternative?

Used GSuite (Google Workspace?): It's fine, it is improving but lack of shortcuts even for basic tasks (I can change the font on Word with just a keyboard, try that with Docs without a mouse). Dealbreaker is the 5 million cell (not row nor column) limit, which is even lower than Microsoft's old limits (more than 15 million cells, which was increased in 2007 to you-have-a-serious-problem-if-you-somehow-fill-this-limit cells).

Re: Ditching Excel for Python in a legacy industry

#49

Post by mch82 » Thu, Dec 31, 2020, 12:51 AM UTC

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

> problem #1: Environment management

Great observation. Python environment management is getting simpler, but is off putting for people without a software background. Unclear even CS majors get enough classroom exposure to package & dependency management to utilize Python efficiently.

I’m more optimistic about an on-prem deployment of Jupyter Notebooks or Sage Math Cloud as a way to hide a lot of the setup complexity. More like a wiki for math. Curious if anyone has stories/tips to share (good or bad)?

Re: Ditching Excel for Python in a legacy industry

#50

Post by meztez » Thu, Dec 31, 2020, 12:51 AM UTC

I'm pushing for our actuarial team to transition to more R + Git. After 3 years of preaching, most of the actuaries now use RStudio + git as their primary work tool. It is happening.

What we did :

1) Provide documentation on everything from install to using internal R libraries for ETL.

2) Provide mostly problem free, always updated VMs with RStudio Server/ Shiny Server.

3) Establish an hotline channel for instant help on R or git.

4) A couple members on the team developed really close working relationship with IT and we have great respect for each other work.

What we provide is way better and by being active, we built users trust in the tools.

We are phasing out SAS and proprietary modeling tools. Python never took hold even if we bought Anaconda entreprise. Excel is there to stay for sure but since actuarial student learn R in school, it is easier to onboard new hire.

If you want to go down this path and have a chat, hit me up. I'm in P&C. We use R both in development and production environments. We use it for pricing, spatial contractual obligation, claims assignment and a couple more models.