People have been integrating web services with excel for two decades. Previously you had to write a C XLL to do it properly, but COM add-ins have been supported for a long time, although have oddities, and now you can load .net assemblies into excel. You can use kerberos baked into windows for authentication, or just use http basic auth, and control access to data sets just like any other thing. So I guess, I'm confused by the article because a lot of the claims about how you have to do things in excel don't seem to be quite right.
Ditching Excel for Python in a legacy industry
271–280 of 289 posts
Re: Ditching Excel for Python in a legacy industry
#272Earlier quoted context omitted.
That's really interesting. I've been working on a JupyterHub / JupyterLab / Python based product for the insurance industry - choosing this for all the reasons you've cited. Would be really interested if there are any points you can share e.g are you using Kubernetes and if so how have you found it?
We use Kubernetes, according to our IT/devops guys it was pretty straightforward to deploy in Azure with Jupyterhub's KubeSpawner module + documentation. A few people are quite eager to learn Python / code in general, so we try to make it convenient. One common use case would be to work with existing excel spreadsheets, so the notebook volume storage should be mountable in Windows. The file upload/management in noteb…
Thanks especially for the Streamlit suggestion.
We've tested JH with K8 in GCP which was straightforward also. With a small team though tending towards a single VM deploy (based on "The Littleist JupyterHub") which looks a lot easier to maintain.
Re: Ditching Excel for Python in a legacy industry
#273Earlier quoted context omitted.
> Reminds me of the "Once Linux gets a desktop it will take over the world" debate from circa 1997-today. Linux did take over the world, just not on the desktop. It was on servers and mobile, which now have more users than desktops or laptops (edit: servers via the web). Technology gets its warts fixed when it grows along an explosive new market, especially if the market ends up being larger than the last. Python is…
> Linux did take over the world, just not on the desktop. I'm honestly not sad that it never happened. It took over my desktop around ~1998 and I wonder if massive adoption of Linux on the desktop would have benefited me or been worse (from my perspective). As it stands literally every single tool I want/need to do my job is already available for Linux and indeed many of those tools are simply better on Linux (docker…
Well, it would certainly make may day live easier if we had Microsoft Office, the Affinity Suite, and Lightroom on Linux, which would probably have happened if Linux was the dominant desktop.
Re: Ditching Excel for Python in a legacy industry
#274Earlier quoted context omitted.
logs of what? if a programmer hard-codes URLs, you really think they are following best practices elsewhere? further, how do logs help get back the millions of dollars that were lost?
Yes, logging with email alerts means that when a catastrophic error is encountered you'd know about it, instead of it being hidden for months.
LOL! as I already stated, a weak coder isn't going to implement best practices for logging. At best, he hard-coded his own email address to receive an alert, however, this also goes unnoticed, since, as you stated initially, he's dead.
Look, whatever company this was, was relying on YAHOO as its data source for making million+ dollar trades -- not Reuters, or Bloomberg, or JP Morgan, but YAHOO -- and then, for MONTHS -- not hours, or days, but MONTHS -- nobody in its finance department or trading desk or whatever happened to notice that the incoming data feeds were not matching up with quotes from counterparties, market makers, CNBC, colleagues, Wall St Journal? Does this company not have auditors? A CFO? Any IT oversight whatsoever? I'm sorry to say that this particular company's problems are rooted much, much deeper than the loss of a particular Excel guy.
Re: Ditching Excel for Python in a legacy industry
#275Earlier quoted context omitted.
> You can't even count on the data being in the same place over the 52 weeks, since they would have added and removed data points over time. > I can just save it as a .py file and run it as a scheduled task on a virtual computer forever. This is rather naive and short-sighted. Do you think the spreadsheet guy is moving data points around because s/he's bored at work and is screwing around with no purpose? No, the bus…
It sounds like you missed the point of GP's post. He was talking about avoiding manual weekly data copy-paste errors by writing code to do it in a predictable format. I think you assumed that they meant the code would never have to be changed again, when they were actually talking about being able to standardize the data update process. I don't think they said anything about never having to change the code, just that…
I'll stick with bash-j's own words, where he bragged about running a python script "forever", and not your interpretation of his words. I'm also going to dismiss what you "think [I] assumed that they meant"
However, if you, or bash-j, believe Excel is incapable of avoiding manual data updates, that's simply not correct. Excel provides at least 3 ways of bypassing manual data entry in favor of automated imports from an external data source -- VBA code, ODC (not to be confused with ODBC) data connection definitions, and PowerQuery (I think that is the current name, haven't used it in a long time).
Re: Ditching Excel for Python in a legacy industry
#276Earlier quoted context omitted.
> until they're load-bearing This cannot be stressed enough. I've outlived generations of finance teams at many startups, and I've seen firsthand the masterpieces/abominations left behind in Excel. Imagine a dozen sheets with ad-hoc queried data copy/pasted from System A/B/C/D into Excel, with formulas that feed formulas that feed formulas. Sometimes columns are inputs (seasonality adjustments for monthly forecasts),…
Someone who does not use Excel nice will probably make spaghetti code as well.
I'd rather reverse engineer spaghetti code than a spaghetti Excel spreadsheet.
Re: Ditching Excel for Python in a legacy industry
#277Earlier quoted context omitted.
All good anecdotes, but I’m still kind of stuck on the whole you got to work at a pie factory thing
(Sorry, misty eyed recollection alert) I should point out that "pies" in the UK is a rather generic term. MBOs (mince beef and onion), sausage rolls, pasties, pork pies and quiche was made in this south Devon based factory, near Plymouth. It was a good corp citizen thing to attend the 1100 "taste panel" which was part of the quality process. Obviously Product Dev, QA and the line crews could not mark their own work s…
I helped to sell System 36s (the big ones that took half a room) back in early 1980s.
I remember one demo when a potential customer asked "If it's this slow with one user how slow is it with six?"
The person doing the demo "improvised" with "It's dynamic load time balancing." Which I'd never heard of before. Turns out neither had anyone else involved with the System 36.
I later came across a whole insurance company that was run on a S/36. It was replaced by a single 386 PC.
Re: Ditching Excel for Python in a legacy industry
#278Re: Ditching Excel for Python in a legacy industry
#279Earlier quoted context omitted.
I'd rather reverse engineer spaghetti code than a spaghetti Excel spreadsheet.
You’ve been lucky with your spaghetti code experiences, then.
At least with spaghetti code I can set a breakpoint or add print statements and get an understanding of the execution path.
Re: Ditching Excel for Python in a legacy industry
#280Earlier quoted context omitted.
You do have access to a repl when using jupyter notebooks. You can hook a notebook or a repl to an existing kernel. I always have a command line attached to my notebooks. When using jupyter lab I attach the build-in terminal and place it at the bottom. When using notebooks I attach it from my terminal. The experience in Rstudio is still better imho. It’s also a more mature text editor and ide than jupyter.
Ok fair enough, I only used notebooks when I can't avoid it. I'm pretty sure you don't get a repl by default though, is there an involved set up in jupyter?
jupyter console --existing
https://stackoverflow.com/questions/22447572/connect-termina...
For jupyter lab, you just choose to start a repl from the gui and choose an existing kernel.