Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

Post reply on HN
Print view View original

131–140 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#131

Post by etothepii »

sam_bristow wrote: post #113
yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

Funny story about Excel on corporate machines. A couple of years ago the company I work for got boight by an Italian company. When we finally migrated the Windows users over to the corporate Office installs a bunch of people found that Excel wouldn't work for them. Things like sum(A1:A20) were syntax errors. After a bunch of digging i worked out that the localisation from corporate meant they suddenly had Italian fun…

It must be easier to build an auditable and reliable solution using a high-level language programming language and concepts like source control and automated testing.

Excel is only easier if you aren't interested in building something auditable and reliable solution that might have some hope of being maintained after you have left the company.

Re: Ditching Excel for Python in a legacy industry

#132

Post by racl101 »

I wish there was just a desktop program like Excel that actually fucking works well on Mac and PC. I like Google Sheets but it's way too integrated in the cloud for my tastes.

Excel has so many good features, but the core of it is so fucking buggy.

It sucks that I gotta bust out Jupyter and use Pandas to double check my work, especially dates, because I can't trust Excel.

Re: Ditching Excel for Python in a legacy industry

#133

Post by nl »

SoSoRoCoCo wrote: post #114
nl wrote: post #102

Earlier quoted context omitted.

I think the 2.x vs 3.x issues have mostly been resolved by now. I don't think I've hit one for a long time, and even StackOverflow answers are more likely to be Python 3 now.

macOS still ships with Py2.7 and has dependencies, and npm-gyp only recently switched to 3.x. same with python SDR. it depends what you use: less popular packages are still languishing. but that discounts the tens of thousands of projects that are already out there that are in use and need conversion. it'll take probably 3-5 years for it to really go away.

Yes, that's true. And aren't some versions of RedHat still on 2.7 too?

But I class this as a packaging issue more than a 2.7 vs 3.x issue: you see the same problems with (as a random example...) different versions of OpenCV - people not using virtual environments have problems even if they are all on 3.7.

When I think of the "2.x vs 3.x problems" I was thinking more of the language and core libray level incompatibilities.

Re: Ditching Excel for Python in a legacy industry

#134

Post by st1ck »

hnracer wrote: post #123
meztez wrote: post #120

Earlier quoted context omitted.

R has data.table. It is the game changer as I agree base R data.frame do not cut it for performance. tibble will come close once they incorporate more of the data.table performance tricks. https://h2oai.github.io/db-benchmark/

Does R have robust CSV parsing? I remember using the default and it'd be extremely finicky about getting the header and index flags right and wouldn't typecast numeric columns properly (instead they'd end up as factors and not play nice)

Python version of data.table has very fast CSV parsing (compared to Pandas), and it didn't have issues like those you mention. Even if data.table had issues with CSV parsing, you could probably use Apache Arrow to parse CSV into arrow table and then convert it to data.table (but that is probably suboptimal).

Re: Ditching Excel for Python in a legacy industry

#135

Post by listenallyall »

konjin wrote: post #39
Closi wrote: post #33

Earlier quoted context omitted.

How would the conversion rate between EUR and USD be determined?

My first job was cleaning up the mess that was caused by hard coded yahoo finance urls in spreadsheets. One of them died, no one noticed and it cost the company millions of dollars in bad trades over three months.

are you suggesting programmers don't hardcode URLs in quick-and-dirty (and sometimes, even production) Python scripts/code?

Re: Ditching Excel for Python in a legacy industry

#136

Post by jandrewrogers »

yold__ wrote: post #26

I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…

I use both Excel and Python, and like both. They solve different kinds of problems, even within the same context.

Excel is fantastic for what I would describe as linear modeling, building a graph of effects in single data models. I reach for Python when I need to fundamentally transform the data model at points to answer the desired question. That is difficult to the point of being impractical in Excel, especially if the data model is large or exploratory. Python is more programmable in this regard but also lacks the strong static typing that would be useful in such work.

I can’t imagine not using either.

Re: Ditching Excel for Python in a legacy industry

#137

Post by sam_bristow »

etothepii wrote: post #131
sam_bristow wrote: post #113

Earlier quoted context omitted.

Funny story about Excel on corporate machines. A couple of years ago the company I work for got boight by an Italian company. When we finally migrated the Windows users over to the corporate Office installs a bunch of people found that Excel wouldn't work for them. Things like sum(A1:A20) were syntax errors. After a bunch of digging i worked out that the localisation from corporate meant they suddenly had Italian fun…

It must be easier to build an auditable and reliable solution using a high-level language programming language and concepts like source control and automated testing. Excel is only easier if you aren't interested in building something auditable and reliable solution that might have some hope of being maintained after you have left the company.

That's the thing, most Excel workbooks start out as a one-off then gradually get adapted and extended until they're load-bearing.

They're often built by specialists in another dept who definitely wouldn't consider themselves programmers.

Doing it 'properly' would probably mean having to spec put the problem, get a budget, maybe wait a few months for someone to look at it. And the same thing every time the requirements change.

Excel is available today and they can get started solving their immediate problem straight away.

After it's been in use for a couple of years and shown value someone takes a look and sees the Lovecraftian horror it's become.

Re: Ditching Excel for Python in a legacy industry

#138

Post by lldbg »

yold__ wrote: post #87
stochastastic wrote: post #70

Earlier quoted context omitted.

Hey fellow reinsurance actuary! I totally agree that Excel has its place in modeling, especially one-offs, and your criticisms make sense. That said, we have been moving a lot of our calculations to Python. We have had way too many rickety tools to move files or send emails (“first you open this spreadsheet and click this button, then you open this spreadsheet and click this button, then...”), and way too many versio…

One reason why Python is so successful is that it places very nicely with C code. Many of Python's libraries are thin wrappers around native DLLs. For example, numpy is a wrapper around a BLAS DLL (e.g. Intel MKL). Pipenv manages the python side of things, but don't exert control over the system DLLs (like Docker does). Anaconda gets very close to what Docker does (by managing DLLs). Have not used poetry, so can't co…

numpy is much more than a wrapper around a BLAS dll. BLAS implements three sets of operations: Level 1: unary and binary vector vector operations, one transform. Level 2: Matrix vector operations. Level 3: Matrix matrix operations (most famously the dgemm routine).

Perhaps some blas implementations offer more features, but that would defeat the purpose of a standard interface.

Re: Ditching Excel for Python in a legacy industry

#140

Post by Der_Einzige »

solresol wrote: post #78
yold__ wrote: post #51

Earlier quoted context omitted.

"As to the memoization, that is not hard to manage in Python." Yes it is. Recursive calls for financial calculations easily go hundreds of thousands of calls deep. This is why high-end actuarial modeling software either decomposes it into a dependency graph and unrolls function calls where possible, or just "brute-forces" it by being a thin wrapper over c++, i.e. using operator overloading on ::operator(). I've seen…

Why doesn't annotating these functions with @functools.lru_cache(10000000) work?

Not sure why this is being downvoted since I don't think the OP has done a good job of showing evidence that this doesn't work for recursion. Your computer almost certainly WILL have enough space for all the stack frames necessary.
Post reply on HN