Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

Post reply on HN
Print view View original

191–200 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#191

Post by etothepii »

indymike wrote: post #4
RcouF1uZ4gsC wrote: post #2

> At the end of the day, making a change in an Excel sheet is easy; understanding formulas is achievable; but learning to code is hard. This is the crux of the matter. I would guess that there are more than an order of magnitude Excel users than Python programmers. Python is great if you already know programming, but expecting domain experts to learn Python in large numbers is going to be a daunting barrier.

I think there's a realization that some computation has become too complex or too risky to put in a spreadsheet. The reasons the author gave for moving to python less about difficulty of coding, but that other problems were bigger than learning to code.

This is exactly the point, do you know of any companies that have had this realisation. If they are based in Bermuda I'd love to work with them.

Re: Ditching Excel for Python in a legacy industry

#192

Post by etothepii »

listenallyall wrote: post #110
bash-j wrote: post #19

Earlier quoted context omitted.

This has frustrated me as a python user for the last 7 years, working as the only python user in business environments dominated by Excel. People will say things like, "if you leave, who can support this report you made in python?" Well I say, who can support the bloated 40mb spreadsheet that would take forever to unpick and figure out how to update with new data? No one can, because I've seen people would rather reb…

> You can't even count on the data being in the same place over the 52 weeks, since they would have added and removed data points over time. > I can just save it as a .py file and run it as a scheduled task on a virtual computer forever. This is rather naive and short-sighted. Do you think the spreadsheet guy is moving data points around because s/he's bored at work and is screwing around with no purpose? No, the bus…

I will grant that tracking vacation days in a 5 person company might be a valid use of excel.

Re: Ditching Excel for Python in a legacy industry

#193

Post by lmns »

kqvamxurcagg wrote: post #27
bash-j wrote: post #19

Earlier quoted context omitted.

This has frustrated me as a python user for the last 7 years, working as the only python user in business environments dominated by Excel. People will say things like, "if you leave, who can support this report you made in python?" Well I say, who can support the bloated 40mb spreadsheet that would take forever to unpick and figure out how to update with new data? No one can, because I've seen people would rather reb…

I've used excel and python in lots of business contexts. For most tasks involving domain experts, excel usually wins hands down. An excel spreadsheet is usually easily auditable. The visual presentation and layout lends itself to review by others. You can click and point at values. Python and other programming languages require an environment and tooling that can't be easily supported across the enterprise. It requir…

> An excel spreadsheet is usually easily auditable. The visual presentation and layout lends itself to review by others.

Tell that the next poor soul who has to edit some organically grown spreadsheets powered by VBA and malformed CSV files which generate your company's financial reports.

Re: Ditching Excel for Python in a legacy industry

#194

Post by wodenokoto »

disgruntledphd2 wrote: post #189
wodenokoto wrote: post #175

Earlier quoted context omitted.

The current top comment (sibling to the one I’m replying to) argues that keeping Python environments across actuaries/users computers up to date is too difficult. This is nicely solved by using R server. I’ve worked in an R server shop, and the experience is really nice. You log on to the server in chrome or Firefox and the browser window basically becomes RStudio and all calculations are done on the server and all c…

The big problem with notebooks is that you don't have a real REPL. This prevents one from single step debugging and tracing. This is one area where RStudio is much, much better. The trouble is that so many of the younger DS people are focused on Python, that it makes financial sense to just deal with all its problems. There's also a lot more programming tools (though less statistical modelling tools).

You do have access to a repl when using jupyter notebooks.

You can hook a notebook or a repl to an existing kernel. I always have a command line attached to my notebooks. When using jupyter lab I attach the build-in terminal and place it at the bottom. When using notebooks I attach it from my terminal.

The experience in Rstudio is still better imho. It’s also a more mature text editor and ide than jupyter.

Re: Ditching Excel for Python in a legacy industry

#195

Post by etothepii »

kqvamxurcagg wrote: post #27
bash-j wrote: post #19

Earlier quoted context omitted.

This has frustrated me as a python user for the last 7 years, working as the only python user in business environments dominated by Excel. People will say things like, "if you leave, who can support this report you made in python?" Well I say, who can support the bloated 40mb spreadsheet that would take forever to unpick and figure out how to update with new data? No one can, because I've seen people would rather reb…

I've used excel and python in lots of business contexts. For most tasks involving domain experts, excel usually wins hands down. An excel spreadsheet is usually easily auditable. The visual presentation and layout lends itself to review by others. You can click and point at values. Python and other programming languages require an environment and tooling that can't be easily supported across the enterprise. It requir…

> An excel spreadsheet is usually easily auditable.

What does this sentence mean? Virtually no part of excel is easily auditable, at best you can see "this file was changed by xyz at a:b:c" identifying the cells that have changed between versions wouldn't even be easy.

Re: Ditching Excel for Python in a legacy industry

#196

Post by newdude116 »

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

From the submitted link:

"The desire to price increasingly complex deals with increasingly large datasets"

Bingo! Most people use Excel when they actually should use a database. I am sure you can use Excel with a database like MS Access, but then again, who does?

To your arguments: 1. " and good luck getting I.T. to support Docker on Windows desktops." Yah. Great experience to work with Excel on Linux.

2. You can always link compiled code for stuff that needs to be fast. But in the end most people wont use neither python nor Excel for HFT

3. " it's relatively easy to figure out what a mangled and convoluted formula is doing"

https://www.sciencemag.org/news/2016/08/one-five-genetics-pa...

https://www.washingtonpost.com/news/wonk/wp/2016/08/26/an-al...

https://www.sciencealert.com/excel-is-responsible-for-20-per...

4. Maybe. Not sure it is really an issue.

Re: Ditching Excel for Python in a legacy industry

#197

Post by etothepii »

jbjbjbjb wrote: post #10

I like the idea but it’s a bit concerning that everyone would be expected to build models in Python. You’d think there would be enough reuse and structure that it wouldn’t be needed or an application could be built to simplify the construction of the models. If it’s so complex that it needs to be coded up in Python and everyone is doing that bespoke each time it feels like alarm bells should be going off.

I think the OP is hoping that by using python one makes reuse of robust libraries and solutions easier so that the ad-hoc reimplementation of common tasks happens less frequently not just "in python".

Re: Ditching Excel for Python in a legacy industry

#198

Post by etothepii »

bernardv wrote: post #44

Loved the post. I spent more than a year trying to pull a prominent reinsurer on the rock, out of spreadsheet hell, and into the modern age. Baby step #1 would have been to transfer critical data to a database environment and baby step #2 would have been to extract business logic and very-poorly written VBA code into an external code library (Python) that is source-controlled and auditable... I can still hear the Chi…

Are you still on the rock? Send me an email (my address is my profile and we should get coffee.)

Re: Ditching Excel for Python in a legacy industry

#199

Post by etothepii »

listenallyall wrote: post #157
ogre_codes wrote: post #148

Earlier quoted context omitted.

> Excel offers the ability to name any cell or range of cells Except almost nobody does. It's not intuitive or the way it's taught. > Lots of Python programmers don't use comments or descriptive variable names either. You have to have variable names in Python. If you want to give them shitty names, that's your bag, but unlike Excel, it's not an extra step. You also have to deal with the fact that every cell in a rang…

Interesting, my comment was solely in response to your complaint about naming convention... you've expanded your criticism quite a bit. I already anticipated your issue with naming being that few people used named cells/ranges -- again, it's not even in a menu or ribbon, it's present at all times, what else do you want? Not "the way it's taught"? Well, blame your teacher. > You have to have variable names in Python s…

If the code was stored in source control with a separation between dev and production the intern might be able to raise such a Pull Request but it would never get approved without oversight from someone more senior. This Code Review process is almost completely impossible in Excel.

Re: Ditching Excel for Python in a legacy industry

#200

Post by etothepii »

_the_inflator wrote: post #162

The author pretty accurately describes a business model for a startup in an enterprise company: offering a service that was once hidden in some excel sheets. As a developer at heart turned Senior Manager, I find this article especially interesting. I stumble a lot over complaints like these in the enterprise company I am working for and truth to be told, I voiced many of these before myself. Problems I see: - What is…

You can reach out to her (or me) on email. Her email is all over her blog and mine is in my profile.

In my opinion the project that inspired this article was some of the most valuable work we did together and it was made more valuable by working directly in a pair (trio?)-programming context with the Underwriter the model was actually for.

Post reply on HN