Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

71–80 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#71

Earlier quoted context omitted.

I'm an actuary with a strong interest in this area - would be very interested to hear more especially on your R vs Python experience.

I'm on mobile, but do also consider https://JuliaActuary.org (something that I personally have contributed to).

Looks really interesting thanks. I've seen some interesting insurance projects using Julia e.g.

https://www.youtube.com/watch?v=__gMirBBNXY

Re: Ditching Excel for Python in a legacy industry

#72
post #38

> The development environment is not user friendly, the syntax confusing, there’s no support for unit testing – I could go on. The vba ide is pretty good imo. Lack of unit test frameworks is valid but doesn't stop you from rolling your own.

Loved VB and VBA but it is too limiting when needing advanced numerical capabilities. Back in the day had to create add-ins making use of compiled Matlab code to get access to decent numerical routines. Eventually moved to Python and never looked back, although I still use Excel for certain tasks.

However I do miss the VBA GUI editor built into Excel. It allowed for relatively polished interfaces in record time.

Re: Ditching Excel for Python in a legacy industry

#73

I work in a small R&D team within a larger engineering organization. I use Python, and it has spread to the rest of my team. However, I've tried to share tools that I've written in Python with the engineers. The problem is that I have to hand-hold them through the process of getting Python working on their computer at the level of detail of: Here is how you find the Python editor. Double click on it. Click on "open."…

Have you tried pyinstaller? Works pretty well for simple cli tools or tkinter apps. There's already predefined docker images with all of the necessary wine hacks that just consume your code + requirements.txt and spit out a exe.

Re: Ditching Excel for Python in a legacy industry

#74
post #42
post #29

Earlier quoted context omitted.

From an actuary's perspective: is Google Sheets ever entertained as an Excel alternative?

It's been a couple years since I've used it, and I didn't feel it was a comparable alternative. It's decent for about 80% of spreadsheet users, but the keyboard shortcuts were lacking and it was missing some functions that I rely on. For keyboard shortcuts, most Excel power users don't use the mouse, so while it sound trivial, it's really hard to feel productive when you have to hunt around for the right button to cl…

> From an enterprise perspective, Excel is so entrenched it would be a 5-10 year effort to port existing spreadsheets to sheets. Practically speaking, most companies wouldn't see the benefit.

And at the end of the day, it would have worse performance than Excel both in calculation speed and _much_ worse UI. One of the reasons Excel is so much better than Sheets is speed. Insurance companies spend hundreds of thousands of dollars a year on actuaries. Even if Excel cost them $500/year/user, it would be easily worth it for actuarial departments.

Re: Ditching Excel for Python in a legacy industry

#75
post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

Agree completely on 1. but not sure on 2. and 4. - I think if you're using Python and need performance then you will be using Numpy etc - would be interested to hear if there are instances where this doesn't work.

numpy is great for vectorizable calculations, but many calcs (particularly for long-term life contingent risks, i.e. reserves), are not vectorizable except in the most simplistic cases.

Re: Ditching Excel for Python in a legacy industry

#76
post #50

I'm pushing for our actuarial team to transition to more R + Git. After 3 years of preaching, most of the actuaries now use RStudio + git as their primary work tool. It is happening. What we did : 1) Provide documentation on everything from install to using internal R libraries for ETL. 2) Provide mostly problem free, always updated VMs with RStudio Server/ Shiny Server. 3) Establish an hotline channel for instant he…

I'm an actuary with a strong interest in this area - would be very interested to hear more especially on your R vs Python experience.

I'm a CPA. When I started learning code, I looked for whatever was most like a spreadsheet. R for the bill, with built-in frames.

Re: Ditching Excel for Python in a legacy industry

#77
If the industry needs a better data analysis solution, that's fine, find one or maybe start to build one. But saying "I'll just throw some Python at it" isn't a good solution. It's like saying you're gonna replace a lawnmower with tool steel and a welding torch. Not only is it not guaranteed to end up as a better solution, but you're setting yourself up for a lot more work than you think.

Re: Ditching Excel for Python in a legacy industry

#78
post #51
post #37

Earlier quoted context omitted.

This is the argument made against all software stack advancements. Nothing to do with industry. But when the benefits outweigh the the hurdles, change happens. And if I was starting a new insurance company (which I've considered) I'd be doing our work in code not xls, and probably python. Having RCS, Numpy, unlimited compute, unlimited storage, all gives me an advantage over my competition. :-) As to the memoization,…

"As to the memoization, that is not hard to manage in Python." Yes it is. Recursive calls for financial calculations easily go hundreds of thousands of calls deep. This is why high-end actuarial modeling software either decomposes it into a dependency graph and unrolls function calls where possible, or just "brute-forces" it by being a thin wrapper over c++, i.e. using operator overloading on ::operator(). I've seen…

Why doesn't annotating these functions with @functools.lru_cache(10000000) work?

Re: Ditching Excel for Python in a legacy industry

#79
XLL lets you also write .NET code, basically anything that can compile to a DLL, to make custom functions you can expose in Excel - and it is a pretty old supported integration method by Microsoft with Excel. COM enabled DLLs were another way to do this, but they ran slower.

Not that I have any issue with getting Python in my Excel, but people seem to forget that .NET is also an option.

Getting these capabilities enabled in a locked down corporate IT environment traditionally was difficult but I suspect that is changing.

I have also lived the whole, turning a model in Excel into an app exercise. At the time, we rewrote a fairly complex demand planning app from Excel/VBA to C# since the other dev team members were C# devs and could support the app.

However, during the project, I did a demo of how one could build a Winforms app in VB.NET also, to the developer who was the Excel/VBA guru. He'd had no idea that coding in VB.NET and Winforms was close enough that he nearly could have been doing that instead.

The compiled C# version of the model we built, went from running a single instance of the model in 1 hour, to under 1 minute. We could re-run their model for tens of thousands of instances daily, without breaking a sweat.

Ironically, the rewritten version in C# never saw the light of day as the project was canceled (corporate politics and wisdom). However, the simple optimizations we identified in the rewrite were given to the Excel guru who actually made improvements to his tool that let it run in more like 10 minutes.. and it was even object oriented and modular! He learned he could do a lot more in Excel/VBA that he didn't even know about.

Coding is coding.. just some tools make the jobs easier or harder.

Re: Ditching Excel for Python in a legacy industry

#80
post #50

I'm pushing for our actuarial team to transition to more R + Git. After 3 years of preaching, most of the actuaries now use RStudio + git as their primary work tool. It is happening. What we did : 1) Provide documentation on everything from install to using internal R libraries for ETL. 2) Provide mostly problem free, always updated VMs with RStudio Server/ Shiny Server. 3) Establish an hotline channel for instant he…

I’ll second klelatti’s question about R vs Python. From my perspective Python is just as practical for actuarial calcs and better for building general purpose tools. Is there a reason Anaconda didn’t click?
Post reply on HN