Why use Python when MS built in JS?
Ditching Excel for Python in a legacy industry
81–90 of 289 posts
Re: Ditching Excel for Python in a legacy industry
#82Does your name rhyme with "tennis tin"?
Re: Ditching Excel for Python in a legacy industry
#83Earlier quoted context omitted.
> problem #1: Environment management Great observation. Python environment management is getting simpler, but is off putting for people without a software background. Unclear even CS majors get enough classroom exposure to package & dependency management to utilize Python efficiently. I’m more optimistic about an on-prem deployment of Jupyter Notebooks or Sage Math Cloud as a way to hide a lot of the setup complexity…
Is this really an issue for the use case being described? Environment management is obviously a significant consideration for software developers who need to keep track of versions etc, but it sounds as if these users primarily want to use the fundamental numpy functions. They could install one of the scientific python stacks (e.g. anaconda) or just install packages globally with pip.
Things always break over time. I have a very technical co-worker, a systems admin, with a broken Python stack on Windows. No idea what's wrong.
Re: Ditching Excel for Python in a legacy industry
#84Earlier quoted context omitted.
Agree completely on 1. but not sure on 2. and 4. - I think if you're using Python and need performance then you will be using Numpy etc - would be interested to hear if there are instances where this doesn't work.
numpy is great for vectorizable calculations, but many calcs (particularly for long-term life contingent risks, i.e. reserves), are not vectorizable except in the most simplistic cases.
Thanks - sorry I'm struggling a bit - wouldn't they be vectorizable across the portfolio or across scenario for stochastic calculations. Maybe it's because of different backgrounds (mine in UK) but I'm can't recall seeing the deeply nested function calls that you're alluding to.
Re: Ditching Excel for Python in a legacy industry
#85Earlier quoted context omitted.
It is great that you respond with succinct reasons. People who have used Python for some years seem to forget just how clunky it really is. I've been using Excel since 1990, and sure, it has its own warts, but Python is a very rudimentary tool compared to Excel. Python is a machine shop. Excel is a car. It may be a lemon, but it's a functional car. This is a great example of programmers not being able to see the fore…
> Reminds me of the "Once Linux gets a desktop it will take over the world" debate from circa 1997-today. Linux did take over the world, just not on the desktop. It was on servers and mobile, which now have more users than desktops or laptops (edit: servers via the web). Technology gets its warts fixed when it grows along an explosive new market, especially if the market ends up being larger than the last. Python is…
Absolutely. I worked at Intel and our distributed computing pools went from a combination of SunOS and AIX machines to Linux in about 3 months, essentially overnight, (back in the 90's). It was an astonishingly fast deployment.
Linux dominates the server world AND the entertainment device world (hello busybox & gstreamer!)
[1] Regarding clunkiness of Python: mostly it is the packages, installation, and 2.x vs 3.x nightmare that persists. Everyone seems to forget the initial pain getting the Python env to work, esp. when it comes to cython native compilation issues / arch wheels, unsupported packages, etc. The only issue I have with python is it is extremely challenging to make cross-platform deployments for single-executables. I've tried three different approaches and they were all trainwrecks. Once that is ironed out, I'll be switching from Electron to whatever Python offers.
Re: Ditching Excel for Python in a legacy industry
#86I'm way more proficient as a programmer, than an Excel user, so it was my assumption that all these marketing guys that constantly work with Excel can do wonders with it. I mean, they probably can, but recently I tried to use it (actually, it was LibreOffice Calc, so there might be my problem, but I don't know if the difference really is this big) instead of writing a python or bash script (as I would usually do) and was unpleasantly surprised by how complicated the stuff that I would consider standard is, like concatenating columns, grouping/counting unique values, dirty data semi-automated cleanup and such. Everything would require either multiple clicks in multiple menus to perform something that appears to be very ill-composable actions in Calc, or would require writing 50-line Basic script (on kinda ugly APIs) for something that I can do by simply converting all of that to csv and writing 5 lines of Python (or sometimes even shell text-utils, literally). My general impression was that it is not really a tool made with power-user efficiency in mind, ending up being not very efficient for anyone, since it isn't very intuitive software to use for an excel-noob anyway.
So my question is, is this really the state of art for visual working with data-sheets, semi-manual data editing and such?
I was assuming that running a Jupyter/Pluto/RStudio and doing stuff in Python/Julia/R when you don't indent to do actual data-analysis/learning, but only something that seems like basic stuff/preprocessing is more of a bad habit, because Excel was actually made to work with tabular (DataFrame-like) structures, but I ended up feeling like there's no way I would actually prefer Excel for that.
Re: Ditching Excel for Python in a legacy industry
#87I'm a research actuary working in reinsurance. Here is why I think Python creates more problems than it solves from the standpoint of most insurance business users: 1.) Environment management. There are many solutions for managing python dependencies, my favorite is Docker + pip. Good luck getting actuaries and underwriters to write Dockerfiles etc, and good luck getting I.T. to support Docker on Windows desktops. Li…
Hey fellow reinsurance actuary! I totally agree that Excel has its place in modeling, especially one-offs, and your criticisms make sense. That said, we have been moving a lot of our calculations to Python. We have had way too many rickety tools to move files or send emails (“first you open this spreadsheet and click this button, then you open this spreadsheet and click this button, then...”), and way too many versio…
One reason why Python is so successful is that it places very nicely with C code. Many of Python's libraries are thin wrappers around native DLLs.
For example, numpy is a wrapper around a BLAS DLL (e.g. Intel MKL). Pipenv manages the python side of things, but don't exert control over the system DLLs (like Docker does). Anaconda gets very close to what Docker does (by managing DLLs). Have not used poetry, so can't comment.
Ultimately, like most dependency management issues, lacking a stable DLL environment won't be a problem until it is :)
Re: Ditching Excel for Python in a legacy industry
#88I'm pushing for our actuarial team to transition to more R + Git. After 3 years of preaching, most of the actuaries now use RStudio + git as their primary work tool. It is happening. What we did : 1) Provide documentation on everything from install to using internal R libraries for ETL. 2) Provide mostly problem free, always updated VMs with RStudio Server/ Shiny Server. 3) Establish an hotline channel for instant he…
I’ll second klelatti’s question about R vs Python. From my perspective Python is just as practical for actuarial calcs and better for building general purpose tools. Is there a reason Anaconda didn’t click?
GPU integration was broken for a long time. Managing VMs / Environments. The absolutely horrible integration with git/Github.
Having to rebuild your environment from scratch when your workspace crashed. Imagine starting a notebook with a 45 minutes compile time. No go.
One click deploy, let's just forget about it.
Re: Ditching Excel for Python in a legacy industry
#89As the author states - this is an issue for some really complex models - where the complexity, reusability and iteration challenges approach code. Most models do NOT take that many tabs, you can build a toy model near instantly - the production line from finished model and output to publishable material is a few shortcuts away. Having an analyst, write that same thing using Jupyter? From an accounts perspective? Man,…
Last time I checked, Libre Office Calc had Python as an option for scripting language, why aren't more people using it?
Re: Ditching Excel for Python in a legacy industry
#90I work in a small R&D team within a larger engineering organization. I use Python, and it has spread to the rest of my team. However, I've tried to share tools that I've written in Python with the engineers. The problem is that I have to hand-hold them through the process of getting Python working on their computer at the level of detail of: Here is how you find the Python editor. Double click on it. Click on "open."…
Solution for this is to make flask or django apps. Easier to make user interfaces, and solves packaging / user experience problems.
I've had some success with WinPython, where I just install the whole kit and kaboodle on their computer. Before I share anything, I try running my code on a fresh install of WinPython.
Learning to distribute Python code is on my to-do list for next year. We now have a younger programmer on the team who is up to date on this stuff, and has agreed to train me.