Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ditching Excel for Python in a legacy industry

amypeniston.com

Post reply on HN
Print view View original

141–150 of 289 posts

Re: Ditching Excel for Python in a legacy industry

#141

Post by tomnipotent »

yold__ wrote: post #59
shnock wrote: post #53

Earlier quoted context omitted.

Why does building and rolling out a standardized environment cost so much? Could you break down the requisite steps and resources required to achieve this? Thank you, I appreciate it :)

Up-front costs (mostly salaries, but all I.T. projects are "billable") 1.) Getting buy-in from solutions architect, software architecture, information security, I.T. management. This will be a 6 month process. 2.) Getting buy-in from actuarial management and audit. Another 6 month process. Recurring annual costs (over 10 years) 3.) Contractor at $150 an hour = $300K annually 4.) Contractor PM at $50 an hour = $100K a…

This isn't realistic at all. Assuming actuaries know Python like they know Excel, the only other added cost is someone technical to slap together a working environment, which isn't particularly hard when compared to the state of what Excel offers in a collaborative environment.

You just don't jump from "humans in Excel" to "CI/CD perfected pipeline" overnight, nor do you need it.

Excel shops still have costly expenses rewriting entire workflows/re-doing Excel files constantly as people come and go, it's not like there isn't already maintenance cost with the current method.

Re: Ditching Excel for Python in a legacy industry

#142

Post by Vaslo »

I work in an excel heavy function (Corporate Finance). And while this sounds very exciting and fresh I am just not seeing it take hold in Fortune 500s I have/am working for. A few reasons:

1) biggest gripe: I don’t have time to maintain and fix models after I move to a new role. If it’s a Python based model I build, no one can seem to fix it when some tiny thing breaks 6 months after due to a change in the data. I’ve had to work weekends to help colleagues fix models that I don’t use anymore. I can hand Excel to a young or old worker and they can always seem to figure it out and take it over.

2) The tools seem limited when directly doing Python in Excel like the one mentioned nothing the article. VBA kind of sucks in 2020 but until Excel natively accepts Python as part of its base, I don’t love being dependent on these 3rd party tools. VBA always works.

3). I’ve recently complete an MS in Data Sci so I am very familiar with Python and R. My company doesn’t need that level of model for most things. We are a best in class in our industry and we get by using lots of Excel models. I mentioned in my first point that I have built a few things with Python. When I had to fix I just rebuilt in Excel and that was all I needed. When I kept fixing the Python code I always felt like I let folks down if I couldn’t fix their stuff right away. Yet our business makes money and we continue to do well without much Python.

I love Python. But until others start to see its value and a critical mass of individuals knows/supports/can implement Python, I will put emphasis on learning Excel tools or SQL first because those will always be supported.

Re: Ditching Excel for Python in a legacy industry

#143

Post by buryat »

SoSoRoCoCo wrote: post #114
nl wrote: post #102

Earlier quoted context omitted.

I think the 2.x vs 3.x issues have mostly been resolved by now. I don't think I've hit one for a long time, and even StackOverflow answers are more likely to be Python 3 now.

macOS still ships with Py2.7 and has dependencies, and npm-gyp only recently switched to 3.x. same with python SDR. it depends what you use: less popular packages are still languishing. but that discounts the tens of thousands of projects that are already out there that are in use and need conversion. it'll take probably 3-5 years for it to really go away.

macOS ships with both py2.7 and 3.7, `python` would call 2.7, and `python3` would 3.7

Re: Ditching Excel for Python in a legacy industry

#144

Post by buryat »

shnock wrote: post #53
yold__ wrote: post #45

Earlier quoted context omitted.

The problem isn't feasibility, it's resources. Building and rolling out a standardized environment, and maintaining it, will cost millions of dollars. It shouldn't, but it does. And for what added benefit? The end-users don't want it, you'd have to spend another couple million for a lateral move at best. More than likely, you'll end up with a pile of Python spaghetti code that runs slower than the spreadsheet (see po…

Why does building and rolling out a standardized environment cost so much? Could you break down the requisite steps and resources required to achieve this? Thank you, I appreciate it :)

reminds me of standardization of the shipping container, not sure of all the details of how that push had happened though

Re: Ditching Excel for Python in a legacy industry

#145

Post by carlmr »

Vaslo wrote: post #142

I work in an excel heavy function (Corporate Finance). And while this sounds very exciting and fresh I am just not seeing it take hold in Fortune 500s I have/am working for. A few reasons: 1) biggest gripe: I don’t have time to maintain and fix models after I move to a new role. If it’s a Python based model I build, no one can seem to fix it when some tiny thing breaks 6 months after due to a change in the data. I’ve…

>When I kept fixing the Python code I always felt like I let folks down if I couldn’t fix their stuff right away. Yet our business makes money and we continue to do well without much Python.

I'm all too familiar with this. I think you need to let go of those Python models. You need to let others fix them themselves, maybe with minimal guidance. That's the only way they have a chance to learn.

Re: Ditching Excel for Python in a legacy industry

#147

Post by tomnipotent »

sam_bristow wrote: post #137
etothepii wrote: post #131

Earlier quoted context omitted.

It must be easier to build an auditable and reliable solution using a high-level language programming language and concepts like source control and automated testing. Excel is only easier if you aren't interested in building something auditable and reliable solution that might have some hope of being maintained after you have left the company.

That's the thing, most Excel workbooks start out as a one-off then gradually get adapted and extended until they're load-bearing. They're often built by specialists in another dept who definitely wouldn't consider themselves programmers. Doing it 'properly' would probably mean having to spec put the problem, get a budget, maybe wait a few months for someone to look at it. And the same thing every time the requirement…

> until they're load-bearing

This cannot be stressed enough. I've outlived generations of finance teams at many startups, and I've seen firsthand the masterpieces/abominations left behind in Excel. Imagine a dozen sheets with ad-hoc queried data copy/pasted from System A/B/C/D into Excel, with formulas that feed formulas that feed formulas. Sometimes columns are inputs (seasonality adjustments for monthly forecasts), sometimes their outputs (modeled growth * last year * seasonality adjustment) and more often than not their right next to each other and maybe they have different cell background colors or a black separator line. Maybe.

And this is just finance. For many e-commerce businesses, planning is done in Excel with equal zeal.

Re: Ditching Excel for Python in a legacy industry

#148

Post by ogre_codes »

listenallyall wrote: post #104
ogre_codes wrote: post #20

The big problem I've always had with "programming" in a spreadsheet is by nature everything is obfuscated and difficult to trace. Yes, you can inspect a cell and see what the source for that cell is, but that might be 10 other cells and you can only really review one cell at a time. It's like a programming language where you only see one line of code at a time. Worse, those references usually aren't named. What does…

> What does "A1 + SomeOtherTab:B2" mean? Excel offers the ability to name any cell or range of cells. Don't even have to search through menus or the ribbon, it's right there to the left of the formula bar. I'm well aware that the vast majority of Excel spreadsheets don't use named cells/ranges, but you can't really blame Excel for that. It couldn't be too much easier. Lots of Python programmers don't use comments or…

> Excel offers the ability to name any cell or range of cells

Except almost nobody does. It's not intuitive or the way it's taught.

> Lots of Python programmers don't use comments or descriptive variable names either.

You have to have variable names in Python. If you want to give them shitty names, that's your bag, but unlike Excel, it's not an extra step.

You also have to deal with the fact that every cell in a range has its own unique formula. It's like you have a special function for each and every cell. You have nice conveniences for it like copy/ paste and dragging, but ultimately you are copying formulas all over the place. And it's super easy to update all but one of those when you make a change.

Yes, you can create custom functions, but much like named ranges, it's not the default behavior, takes extra steps, and it isn't the way Excel is taught.

Spreadsheets are amazing for small to moderately complex things, but beyond a certain point, they are just an unmanageable mess regardless of who creates them.

Re: Ditching Excel for Python in a legacy industry

#150

Post by BuuQu9hu2 »

yold__ wrote: post #124
andi999 wrote: post #117

Earlier quoted context omitted.

Although at fist glance this formula is written recursively, one doesn't have to (and shouldn't) implement using recursion, does one? Just making f, g, q, d arrays and then loop over t should be good, or is there more to this formula?

Appreciate the curiosity. In this small trivial case, yes that works. But what happens when something in the logic changes? You wind up needing to know the order of calculations since things are no longer lazily evaluated via recursion. This is a problem when you have dozens of "columns" (i.e. recursive functions or arrays as you are suggesting). Often times, the value in the array is NULL (or worse, leftover from a…

It is common knowledge that all recursive functions can be re-written using iteration (e.g. loops). See “ Recursion versus iteration” here https://en.m.wikipedia.org/wiki/Recursion_(computer_science). The assumption that only trivial calculations can occur using iteration, or that recursion alone allows for supportable code, I believe are very flawed assumptions.
Post reply on HN