Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

221–230 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#221
The problems with excel IMO are:

- large datasets are an issue

- it doesn't have some libraries without extra cost (and in my instance long winded approvals). I'm specifically thinking about linear programming libraries here.

- VBA is less easy to code in

Otherwise excel is great.

Side note - is anyone exploring Jai? This seems to try to be solving the installation and compatibility issues that mires coding these days.

Re: Ditching Excel for Python in a legacy industry

#223

Earlier quoted context omitted.

That's the thing, most Excel workbooks start out as a one-off then gradually get adapted and extended until they're load-bearing. They're often built by specialists in another dept who definitely wouldn't consider themselves programmers. Doing it 'properly' would probably mean having to spec put the problem, get a budget, maybe wait a few months for someone to look at it. And the same thing every time the requirement…

> until they're load-bearing This cannot be stressed enough. I've outlived generations of finance teams at many startups, and I've seen firsthand the masterpieces/abominations left behind in Excel. Imagine a dozen sheets with ad-hoc queried data copy/pasted from System A/B/C/D into Excel, with formulas that feed formulas that feed formulas. Sometimes columns are inputs (seasonality adjustments for monthly forecasts),…

Someone who does not use Excel nice will probably make spaghetti code as well.

Re: Ditching Excel for Python in a legacy industry

#224
post #29
post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

From an actuary's perspective: is Google Sheets ever entertained as an Excel alternative?

Uploading propertiary data to a cloud is a big no.

Re: Ditching Excel for Python in a legacy industry

#225
post #171
post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

I think these are all fair points and a good reflection of the downside of Python. But there are also some pretty huge upsides. My view is that for any model of significant complexity, the pros of Python outweigh the cons from a technical point of view. - Abstraction. It's very difficult to effectively abstract parts of a model in Excel. It's a bit like a doctor having to 'model' a human being as a collection of atom…

> I appreciate some of the above is also possible in VBA, but if you're writing an entire model in code and not really using Excel at all, my view is it's better to use a more sophisticated programming language.

Which is why many VBA experts eventually adopt VB.NET instead of jumping into a complete foreign language, with the benefit that is actually compiled to native code (JIT/NGEN), if performance is ever an issue.

Re: Ditching Excel for Python in a legacy industry

#226

Earlier quoted context omitted.

Interesting, my comment was solely in response to your complaint about naming convention... you've expanded your criticism quite a bit. I already anticipated your issue with naming being that few people used named cells/ranges -- again, it's not even in a menu or ribbon, it's present at all times, what else do you want? Not "the way it's taught"? Well, blame your teacher. > You have to have variable names in Python s…

If the code was stored in source control with a separation between dev and production the intern might be able to raise such a Pull Request but it would never get approved without oversight from someone more senior. This Code Review process is almost completely impossible in Excel.

That is a big if.

Office has source control management via SharePoint integration.

Re: Ditching Excel for Python in a legacy industry

#227
post #38

> The development environment is not user friendly, the syntax confusing, there’s no support for unit testing – I could go on. The vba ide is pretty good imo. Lack of unit test frameworks is valid but doesn't stop you from rolling your own.

And a stepping stone into VB.NET in case more power is needed.

Re: Ditching Excel for Python in a legacy industry

#228
post #75

Earlier quoted context omitted.

Agree completely on 1. but not sure on 2. and 4. - I think if you're using Python and need performance then you will be using Numpy etc - would be interested to hear if there are instances where this doesn't work.

numpy is great for vectorizable calculations, but many calcs (particularly for long-term life contingent risks, i.e. reserves), are not vectorizable except in the most simplistic cases.

Numba can usually speed up non-vectorizable things by JIT compiling and potentially parallelizing.

I mean, in an ideal world you'd use Julia or a Cython extension, but if you already have something in Python/numpy, numba only requires you add a decorator to your function and it gets jitted.

Re: Ditching Excel for Python in a legacy industry

#229
post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

It is great that you respond with succinct reasons. People who have used Python for some years seem to forget just how clunky it really is. I've been using Excel since 1990, and sure, it has its own warts, but Python is a very rudimentary tool compared to Excel. Python is a machine shop. Excel is a car. It may be a lemon, but it's a functional car. This is a great example of programmers not being able to see the fore…

> "Once Linux gets a desktop it will take over the world" debate from circa 1997-today.

Pretty much all servers run on linux.

Linux also dominates smartphones in the form of Android phones.

Chrome OS is also linux, and it's market share is currently at around 6%.

I mean, other than desktops Linux pretty much is everywhere.

Re: Ditching Excel for Python in a legacy industry

#230
I dont claim that Excel is the best tool or that it should be used in reinsurance, but it seems that the author does not know how to use it properly.

It sounds more as if they knew Python so they use Python. If author would know Java they would write that Java is better.

For example combining data with logic does not need to happen in Excel. Someone who does this might make the same bad practice in Python too.

Also there are no thoughts that there are many bad Excels, but there are also many bad programs (in Python and every other language). There is some sort of magic thinking that a rookie who switched from Excel to Python will somehow not produce spaghetti code. What is not true at all. Those Excels are much easier to debug by the business side. Any program is a black box.

Author makes empty claims that "excel forumals are long" or that models have many tabs. If the Python program gets as big it might also become a mess.

How many times have you heard that the new programmer looks on old code and says that it is spaghetti? Nearly every time. Nealry every time they want a rewrite too. And Excel just works.

I doubt author used Python programs made by others. There are no comments on that. Also eas authors code reviewed by a real programmer? Author is self thought so odds are that they create some really awful code and dont even know it.

Post reply on HN