> You can achieve a awful lot with Excel: more, even, than some programmers can achieve without it.
I've heard this a lot, but have never really used Excel. What can it do that a programmer can't ?
An oral history of Bank Python
161–170 of 333 posts
Re: An oral history of Bank Python
#162I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based). Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their m…
Does everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ? I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL). What does the devops there look like ? Snapshot every minute ?
I use Pickle quite a lot for caching, a file read is almost always faster than a DB query.
For long-term persistent data ? Seems very dangerous to me, even reading a pickle from say PyPy vs a Cython intepreter corrupts the damn thing.
Re: An oral history of Bank Python
#163Earlier quoted context omitted.
> Dependency graphs are an elegant solution to risk management and pricing etc. Dependency graphs are not a solution to risk and pricing. They are, in certain circumstances, a very useful tool. That's all. They also scale notoriously painfully. Putting a dependency graph as a mandatory component in your risk system was one of the worst technical decisions I've come across (and I've been doing this lark a long time).
Wouldn't an observer pattern work better? The graph itself could even be used to instantiate subscriptions in a pub/sub system where changes in underlying pricing could be dealt with via an event queue. Compaction and debouncing could be applied on top of the queue to avoid lots and lots of redundant execution.
> Wouldn't an observer pattern work better?
Better as a solution to what problem? In some cases a dependency graph is an excellent solution. In some cases it's not. In some cases it's fine for small graphs but scales poorly as it can be very hard to reason about (as attested by pretty much anyone who's supported a really big spreadsheet).
But that's the point; it's a really useful tool. Sometimes.
Re: An oral history of Bank Python
#164I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based). Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their m…
> python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages I'm not sure how to react to that, but these features in Python are miles ahead of what many other languages have (or actually don't have).
Am I the only one who has never had enough headache over the years to say its awful? I do think dependency management is a somewhat difficult problem to solve and a lot of systems have pros/cons but I never have had huge issues with Python's.
Re: An oral history of Bank Python
#165This immediately sprang out at me: > Investment banks have a one-way approach to open source software: (some of) it can come in, but none of it can go out. I wonder how well this plays with the various open source software licenses?
Well, with GPL particularly but open source licenses more generally, the user is allowed to do whatever they want with the code. It is only when the code is redistributed that the source must be provided. With AGPL the source must be provided also to anyone using a service running AGPL licensed code.
So it plays perfectly with the licenses. This is the sort of thing free software was designed for, allowing everyone that uses a codebase to own it 100%.
Re: An oral history of Bank Python
#166It sounds like there may not be any tests-written-in-code at all, let alone CI-style testing?
Re: An oral history of Bank Python
#167Earlier quoted context omitted.
There was also a giant investment bank system written in Smalltalk, so that may be a direct influence: http://www.esug.org/data/ESUG2004/ValueOfSmalltalk.pdf
Can anyone confirm whether JP Morgan were able to decommission Kapital when they went to Athena? I've seen so many cases in banks where the old system is still running years and years after it was 'replaced'. And Kapital was used in so many, different, parts of the business.
The answer is no. Kapital persisted past Athena for many years and was not (seriously) considered for shutdown
Re: An oral history of Bank Python
#168Honestly, this sounds a lot less insane than a lot of "conventional" stacks.
Re: An oral history of Bank Python
#169Earlier quoted context omitted.
Could be a misdirection because all of the rest fits Quartz to a tee. The Quartz database is called Sandra (referred to as Barbara here). The Quartz directed acyclic graph is called Dag (referred to as Dagger here) The Quartz job runner is called Bob (referred to as Walpole here which is a reference to Robert Warpole whose shortname is..Bob) These and the horrible proprietary IDE make it obvious which particular syst…
How are the Barbara databases synchronized - as multiple nodes are mentioned ? The description makes it sound like it's just a large set of pickles in something like a Berkley DB?
Each server in a ring has a complete copy of the data for that ring. Each ring consists of a network of servers which may have nodes in different geographies. They're called rings, but are actually acyclic networks (IIRC).
Replication occurs automatically so you need to manage consistency in your app architecture. For example if you have instances of an app running in different geographies, the specific data for those instances should be in different folders.
Re: An oral history of Bank Python
#170> You can achieve a awful lot with Excel: more, even, than some programmers can achieve without it. I've heard this a lot, but have never really used Excel. What can it do that a programmer can't ?
By virtue of Turing completeness there's nothing you can do in Excel that you can't do in a program. It's all a matter of speed.
Having seen Excel wizards work their magic before, the dizzying ways they can slice and dice their data with the help of a combination of GUI affordances, formulas, and hot keys is truly astounding. Often times a person could build out a full set of data and charts in half an hour that might be something like > 100 lines of equivalent Python/Pandas code.
And crucially often the report would have less bugs than the equivalent code because the analyst could see all the data in front of them as they were manipulating it and would naturally do spot checks along the way.
Now note the "some" in that statement. A Python/Pandas master could also probably whip up the equivalent > 100 lines in half an hour. But it was really astounding just how fast Excel experts worked.