Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

Post reply on HN
Print view View original

161–170 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#161

Post by atoav »

SoSoRoCoCo wrote: post #30
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

It is great that you respond with succinct reasons. People who have used Python for some years seem to forget just how clunky it really is. I've been using Excel since 1990, and sure, it has its own warts, but Python is a very rudimentary tool compared to Excel. Python is a machine shop. Excel is a car. It may be a lemon, but it's a functional car. This is a great example of programmers not being able to see the fore…

Depends really on your proficiency at python and the task at hand. For many "small fry" tasks Excel is perfectly fine and faster. Good when the table data is meant to be in foreground.

For things where data exploration and formulas should be in the foreground or where data and formulas should be strictly separated python (or a python/jupyter notebook) has tangible benefits (e.g. really good list syntax).

Re: Ditching Excel for Python in a legacy industry

#162

Post by _the_inflator »

The author pretty accurately describes a business model for a startup in an enterprise company: offering a service that was once hidden in some excel sheets.

As a developer at heart turned Senior Manager, I find this article especially interesting. I stumble a lot over complaints like these in the enterprise company I am working for and truth to be told, I voiced many of these before myself.

Problems I see:

- What is the business problem the author is trying to solve? How does a tool - Python - can help do specifically do what better?

- There are no specific measurements mentioned. How big is the data the author mentioned, how long does an analysis cycle take, how large are the teams, the affected people? What about maintaining the software stack? How many requests are there per year?

-What about cost savings? How could they help us compete with other companies? Lead cycles of even weeks may bother a developer but not the business.

It is not that I don't believe his suggestions. It is just that I don't get to the point other than "my favorite tool could do it, too." We could easily substitute Python with R, for example.

"The spreadsheet took 30+ seconds to open" I know this is an annoyance, but how often do you open it? One time a day? 20 times an hour?

"The new model logic is testable and can be upgraded independently" this is one of the most valuable points here, as long as you work in a larger environment. So context is needed here as well.

I know a colleague of mine who is extremely well versed in Excel who has put a decent amount of magic into her sheets. However, even losing her and starting all over again is from a business perspective way cheaper than trying to put her solution behind a cloud service.

It would be fun, to have a conversation with the author.

Re: Ditching Excel for Python in a legacy industry

#163

Post by sam_bristow »

mehrdadn wrote: post #152
sam_bristow wrote: post #113

Earlier quoted context omitted.

Funny story about Excel on corporate machines. A couple of years ago the company I work for got boight by an Italian company. When we finally migrated the Windows users over to the corporate Office installs a bunch of people found that Excel wouldn't work for them. Things like sum(A1:A20) were syntax errors. After a bunch of digging i worked out that the localisation from corporate meant they suddenly had Italian fun…

> There are ways of building spreadsheets that are reliable and auditable. Then there's how 95% of people do it Do you have any pointers to learning materials on how to do this? Would be interested in reading more on it.

This Twitter thread is a good start: https://mobile.twitter.com/keith_ng/status/13079610874515251...

There also Joel Spolsky video I linked in another comment: https://youtu.be/0nbkaYsR94c

I don't actually use Excel much so others might have better resources.

Re: Ditching Excel for Python in a legacy industry

#164

Post by kfk »

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

Hi, this is a late reply but I am actually pulling this off. Your points above all make sense but are around the desktop paradigm. If you move to the cloud paradigm not only most of the points go away but you gain a lot by having data all in one place (S3) and strict collaboration (github). Specifically the Python env problem goes away if you ask analysts to work online with notebooks (ie jupyter hub).

Re: Ditching Excel for Python in a legacy industry

#165

Post by RMPR »

krick wrote: post #86

I'm way more proficient as a programmer, than an Excel user, so it was my assumption that all these marketing guys that constantly work with Excel can do wonders with it. I mean, they probably can, but recently I tried to use it (actually, it was LibreOffice Calc, so there might be my problem, but I don't know if the difference really is this big) instead of writing a python or bash script (as I would usually do) and…

> Python/Julia/R when you don't indent

This typo made me chuckle

> So my question is, is this really the state of art for visual working with data-sheets, semi-manual data editing and such?

I guess Excel is used because it's well established, and migrating will be very costly not to mention finding people in that field knowing Python/Julia/R

Re: Ditching Excel for Python in a legacy industry

#166

Post by harha »

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

For 1. I would say R is a good option. It works relatively well everywhere and has an ok IDE, lots of packages that make life easier (tidyverse). I also wouldn’t recommend python for exactly that reason.

Re: Ditching Excel for Python in a legacy industry

#167

Post by RMPR »

hcrad wrote: post #92
analog31 wrote: post #61

I work in a small R&D team within a larger engineering organization. I use Python, and it has spread to the rest of my team. However, I've tried to share tools that I've written in Python with the engineers. The problem is that I have to hand-hold them through the process of getting Python working on their computer at the level of detail of: Here is how you find the Python editor. Double click on it. Click on "open."…

Pyinstaller + Gooey has been my go-to combo for sharing executable python with simple/intuitive UIs. https://github.com/chriskiehl/Gooey

> https://github.com/chriskiehl/Gooey

Now that is great

Re: Ditching Excel for Python in a legacy industry

#168

Post by kfk »

meztez wrote: post #50

I'm pushing for our actuarial team to transition to more R + Git. After 3 years of preaching, most of the actuaries now use RStudio + git as their primary work tool. It is happening. What we did : 1) Provide documentation on everything from install to using internal R libraries for ETL. 2) Provide mostly problem free, always updated VMs with RStudio Server/ Shiny Server. 3) Establish an hotline channel for instant he…

I am putting together course aimed at Python beginners in enterprise, I too have experience in Finance. If someone is interested I would love some early feedback, you can contact me my email is in this profile bio.

Re: Ditching Excel for Python in a legacy industry

#169

Post by RMPR »

analog31 wrote: post #90
shmoogy wrote: post #64

Earlier quoted context omitted.

Solution for this is to make flask or django apps. Easier to make user interfaces, and solves packaging / user experience problems.

I've had some success with WinPython, where I just install the whole kit and kaboodle on their computer. Before I share anything, I try running my code on a fresh install of WinPython. Learning to distribute Python code is on my to-do list for next year. We now have a younger programmer on the team who is up to date on this stuff, and has agreed to train me.

> Learning to distribute Python code

Found pyinstaller very handy, in fact that's what I use to create releases for my side project[0]. And if you're creating CLIs, a sibling comment mentioned Gooey.

0: github.com/rmpr/atbswp

Re: Ditching Excel for Python in a legacy industry

#170

Post by alexhutcheson »

st1ck wrote: post #134
hnracer wrote: post #123

Earlier quoted context omitted.

Does R have robust CSV parsing? I remember using the default and it'd be extremely finicky about getting the header and index flags right and wouldn't typecast numeric columns properly (instead they'd end up as factors and not play nice)

Python version of data.table has very fast CSV parsing (compared to Pandas), and it didn't have issues like those you mention. Even if data.table had issues with CSV parsing, you could probably use Apache Arrow to parse CSV into arrow table and then convert it to data.table (but that is probably suboptimal).

https://readr.tidyverse.org/
Post reply on HN