Ditching Excel for Python in a legacy industry
amypeniston.com
Ditching Excel for Python in a legacy industry
1–10 of 289 posts
Re: Ditching Excel for Python in a legacy industry
#2> At the end of the day, making a change in an Excel sheet is easy; understanding formulas is achievable; but learning to code is hard.
This is the crux of the matter. I would guess that there are more than an order of magnitude Excel users than Python programmers. Python is great if you already know programming, but expecting domain experts to learn Python in large numbers is going to be a daunting barrier.
Re: Ditching Excel for Python in a legacy industry
#3This is a really interesting article on some of the challenges facing enthusiasts for technologies such as Python in a "traditional" corporate environment.
I'm a really strong advocate for these technologies - and have been developing a notebook based product for use in the insurance sector - but there is a lot of resistance.
- There is very little awareness of the power of open source tools to do traditional data manipulation tasks;
- In addition to Excel there are both legacy and newer proprietary systems backed by consulting firms that have a strong hold over parts of the market.
On the other hand there are some areas where there is increasing adoption of Python and (especially I think) R to do statistical analyses that are difficult / impossible in Excel. Also DataScience tools and techniques are now being taught as part of standard actuarial courses.
Finally, firms are increasingly acutely aware of the risks of relying on Excel and are looking for tools with better control / testing environments.
Re: Ditching Excel for Python in a legacy industry
#4> At the end of the day, making a change in an Excel sheet is easy; understanding formulas is achievable; but learning to code is hard. This is the crux of the matter. I would guess that there are more than an order of magnitude Excel users than Python programmers. Python is great if you already know programming, but expecting domain experts to learn Python in large numbers is going to be a daunting barrier.
I think there's a realization that some computation has become too complex or too risky to put in a spreadsheet. The reasons the author gave for moving to python less about difficulty of coding, but that other problems were bigger than learning to code.
Re: Ditching Excel for Python in a legacy industry
#5As the author states - this is an issue for some really complex models - where the complexity, reusability and iteration challenges approach code.
Most models do NOT take that many tabs, you can build a toy model near instantly - the production line from finished model and output to publishable material is a few shortcuts away.
Having an analyst, write that same thing using Jupyter? From an accounts perspective? Man, I’d want to see it in a spread sheet. It’s just simpler, or more familiar, to debug accounting information in a spread sheet.
The idea that we are going to see all those analysts pick up code - over excel - is possible, but I’d say less likely.
I’d suspect that the idea of python inside of excel, is a winner. But given that excel is working with its own data model and data tools with power BI, or with their new Lambda function, I’d say they are also working to keep people happy within the excel ecosystem.
Interestingly, this is a version of the Bloomberg terminal debate - the terminal does everything, any upstart can only do a small part of the BB offering, allowing BB to always be relevant if not dominant.
Re: Ditching Excel for Python in a legacy industry
#6That was a neat detour into reinsurance. I wonder how the main thrust of the article holds up generally. My experience in trading/finance is that Excel remains extremely preferred for last-mile use (i.e. for writing and reading reports). Whereas the data science ecosystem has been adopted to develop research platforms and manage the data into (what ultimately becomes, for most users in the organization) an Excel sheet.
I don't see this changing any time soon. I think Excel was always strongest for last-mile use. Excel is extremely powerful when you know how to use it correctly, and I routinely see people match or exceed the productivity of programmers using it for specialized use cases.
Re: Ditching Excel for Python in a legacy industry
#7I do a lot of both. Excel really is great for data where there are less than say 100k rows. Its just so easy to see exactly what you're doing and what the data looks like. If you have millions of records Python really does better but I still find it frustrating to find a way to keep peeking behind the curtain.
Ideally I'd have a type safe language which can embed data the way excel does. If Excel had dotnet languages instead of VBA and could store data arrays in XLBs it'd slay.
Re: Ditching Excel for Python in a legacy industry
#8Blockpad (http://www.blockpad.net) tries to ease some of these pain points but more in the engineering space.
Re: Ditching Excel for Python in a legacy industry
#9We have a few customers in reinsurance, and for the most part the goal is to do the opposite of what the python solutions try to do. Instead of integrating foreign stuff into existing workbooks, the goal is to retain the existing worksheets as source of truth and build modern tools around the files. The most common use case is building out a web interface to replicate the Excel formula engine.
In the python space, there are libraries like openpyxl and xlrd, but the real hurdle is introducing python into an ecosystem which otherwise has no natural knowledge. JavaScript is the language of choice for modern Excel addins as Excel provides an actual API for it https://docs.microsoft.com/en-us/office/dev/add-ins/referenc...
Re: Ditching Excel for Python in a legacy industry
#10I like the idea but it’s a bit concerning that everyone would be expected to build models in Python. You’d think there would be enough reuse and structure that it wouldn’t be needed or an application could be built to simplify the construction of the models.
If it’s so complex that it needs to be coded up in Python and everyone is doing that bespoke each time it feels like alarm bells should be going off.