An oral history of Bank Python • Hacker News Board

Re: An oral history of Bank Python

#271

Post by Maven911 » Fri, Nov 05, 2021, 6:14 AM UTC

seanhunter wrote: post #76

simonh wrote: post #31

Earlier quoted context omitted.

I worked on Quartz for 3 years and loved it. Some devs grumbled about various aspects of it, but I come from an application support background and taught myself python, so I suppose I had fewer developer habits to un-learn. From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based. The lore is that SecDB was instrumental in Goldman being ab…

I know Kirat really well. Fun fact, one two-week dev cycle we had 667 distinct developers commit to the secdb code base which Kirat's boss described to me as "The number of the beast.... plus Kirat" Second fun thing. Kirat was advocating for lisp for secdb for a long time and used to rag on me for liking python when it's so slow.

It's amazing how a few people have left such a big mark on a part of the investment banking industry. I missed Kirat right before exiting BAML but met all his "disciples" and Dubno..including his miniature dinosaur and telescope in his office :). Very much felt like tech religion where no open debate on merits and drawbacks could be discussed. And a lot has changed in terms of engineering innovation with turnover since that era...

Re: An oral history of Bank Python

#272

Post by shrike21 » Fri, Nov 05, 2021, 6:15 AM UTC

seanhunter wrote: post #21

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember) Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. O…

Hello Sean,

I remember Slang when I first saw the code, a parse tree based evaluator in 1997. Come on folks. Separate parsing from evaluation. Opaque types with message passing. Inference anyone? Clearly no one read hindly milner.

Add parse time optimizations, add locals, hey globals and locals weren't handled properly. Python in the 90s anyone?

Shitty KV store with 32K object limits related to some random OS2 btree limits. Add huge pages.

Deal with random rubbish related to inconsistent non transactional indices.

Figure out you should layer nodes in a dag. A dag is topologically 2 dimensions, fairly limiting.

Figure out that's somewhat similar to layering databases, it's just another dimension.

Hmm bottom up evaluators, you actually need top down as well, create a turing complete specification. well, limit it a bit.

Ah KLL points out that layers on top of dimensions end up being combinatorial, but you can actually cache the meta traversal and it's small n.

Lots of people point out category theory parallels. Haskell is pretty but completely unusable. I'm a math guy, and it's still unusable, I don't like feeling smart when I do simple things.

But interestingly creating imperative functions with pure input/outputs with implied function calls is pretty interesting. You can create an OOP paradigm with self and args as known roots aka linda tuple spaces.

Ah and each tuple space point can be scheduled independently, some issues with serialization and locality...

Go to another bank and choose to use python, foolishly decide to rely on pickle. Do that twice. Bad idea.

But write a much better petabyte scale multi-master optimistic locking database with 4 operations. Insert, Upsert, Update, delete. WAL as a first class API.

Finally decide that writing a coding scheme to convert python objects to json is not really hard. And of course cloud native and cloud agnostic is really the only way to go nowadays.

I'm always confused why people complain about Athena/Quartz, hell we wrote it all, fix it if you don't like it. Open source it if you want other people to contribute. If we made stupid decisions on pickling data, well there's a version id, add json serialization, it's not hard, don't take things as given.

Re: An oral history of Bank Python

#273

Post by Maven911 » Fri, Nov 05, 2021, 6:22 AM UTC

mirko22 wrote: post #206

seanhunter wrote: post #21

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember) Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. O…

Oh wow, I remember Quartz at BAML… Though this was several years after initial deployment and when core devs left. One day I will sit down and write a small poem about the insanity of software development based on my experience with Quartz. It will be an intriguing story of love and hate being told through a sensual dance between sales and engineering. The battles will not be epic but the consequences of one’s action…

Tell me more about what worked and didn't..I recall the pain of watching QzDesktop load and Bob/HUGS jobs failing..but what else, and what did you enjoy

Re: An oral history of Bank Python

#274

Post by esonoi » Fri, Nov 05, 2021, 6:30 AM UTC

markus_zhang wrote: post #107

alecco wrote: post #86

I worked at one of the largest of these systems. It seems to be the one referred by the post. The global distributed store of pickled python objects using Event Sourcing was one of the most horrible and expensive database systems I've ever heard of. It runs on THOUSANDS of expensive servers with all data stored in-memory. To get the state of a single deal you had to open, decompress, deserialize, and merge hundreds i…

But it was good for the first generation who worked there. Big dollars, total control and even the opportunity to create a proprietary IDE...

That proprietary IDE was a piece of crap. Monkey patching was prevalent, exponentially increasing startup time depending on the last time you opened it, and an “online” source tree where one could easily modify the source code in someone else’s ‘private’ workspace.

PyCharm was a move in the right direction, but the way it worked was absurd- it would run the internal IDE in the background and sync to the file system. Given the proprietary IDE took up more resources than PyCharm, you constantly had to shut down apps so the machine had enough memory.

IDEs should not be a requirement- they are tools… but you had no choice but to use it and their totally flawed code completion. Their measure of success was tantamount to having a Jupyter notebook- write code and get back results immediately.

Re: An oral history of Bank Python

#275

Post by esonoi » Fri, Nov 05, 2021, 6:52 AM UTC

streamofdigits wrote: post #69

> Investment banks have a one-way approach to open source software: (some of) it can come in, but none of it can go out I'd say thats how they see society and their role in it more generally. Doing God's work is a surprisingly directed graph. It applies also to the broader banking world, but investment banks being the most lucrative (when not bailed out) attract talented people that in principle can close the loop an…

Not entirely true. Two things to consider: 1. Public sentiment. When Goldman Sachs open sourced their collections library on GitHub, it gained marginal traction (opinion: seemed more about PR to attract tech talent). When it was adopted by the Eclipse Foundation, usage rose by a non-trivial amount (based on usage stats from mvnrepository that aren’t other eclipse projects). 2. There was a massive hiring frenzy, and their due diligence regarding IP was non-existent. Garden leave doesn’t compensate for ‘strategic’ systems. Apart from “competitive advantage”, when you have someone as tenured as he who shall not be named, you mitigate the risk of being sued by not making it obvious you’re cloning a system developed for another firm.

Bulge brackets are more risk avoidant than smaller firms, like hedge funds. Today, we have LMAX disruptor and Real Logic’s Aeron (basis for Akka Remote) due to their liberal policy towards open source.

Re: An oral history of Bank Python

#276

Post by seanhunter » Fri, Nov 05, 2021, 7:04 AM UTC

keithalewis wrote: post #237

seanhunter wrote: post #212

Earlier quoted context omitted.

> From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based. The lore is that SecDB was instrumental in Goldman being able to rapidly figure out what their exposure was during the 2008 crisis. Correct. We used python for a bunch of infrastructure stuff (eg distributing all of secdb to all of the places it needed to go). The actual pricing a…

My grandpappy always told me to cut out the middleman. Modern C++ was heavily influenced by the need to make it simple to use directly. If you are in the business of writing code instead of reminiscing, you can now leverage move semantics, lambdas, and smarter pointers to create software that is close to the silicon. Python might be great, but it sure is slow. Its success is founded on smart people making it easier f…

A big force multiplier in the old GS secdb model was simply the speed of the dev cycle vs speed of the code. As a strat you could push slang changes to pricing and risk literally in minutes with full testing, backout, object audit logging etc.

C++ changes went out in a 2 week release cycle so changes were still fast by most standards but much slower. But yeah we had 20m + lines of C++ code so it was extensively used.

Re: An oral history of Bank Python

#277

Post by rbanffy » Fri, Nov 05, 2021, 7:20 AM UTC

shrike21 wrote: post #272

seanhunter wrote: post #21

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember) Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. O…

Hello Sean, I remember Slang when I first saw the code, a parse tree based evaluator in 1997. Come on folks. Separate parsing from evaluation. Opaque types with message passing. Inference anyone? Clearly no one read hindly milner. Add parse time optimizations, add locals, hey globals and locals weren't handled properly. Python in the 90s anyone? Shitty KV store with 32K object limits related to some random OS2 btree…

> foolishly decide to rely on pickle.

This scared me a bit TBH. It’s one of those decisions that come back to bite us repeatedly.

Re: An oral history of Bank Python

#278

Post by AmericanBlarney » Fri, Nov 05, 2021, 7:25 AM UTC

barrkel wrote: post #230

smallnamespace wrote: post #219

Earlier quoted context omitted.

Armen was very passionate about the value of "strats" (GS's own term for "quants", and later broadened to include software engineers and data scientists). A favorite quip of his: At GS, I'm like an arms dealer. When a desk has a problem, I send in the strats, and they blow away all the competition! Also, SecDB's core idea is not just tight integration between the backend and development environment, but that all obje…

It's as old as VisiCalc, it's how spreadsheets work. I built a similarly reactive system for web UI binding back in 2004, running binding expressions on the back end with cached UI state to compute the minimal update to return to the front end, in the form of attributes to set on named elements.

Yes, although doing it in distributed fashion at the scale of SecDB or Athena introduces quite a few more complexities.

Re: An oral history of Bank Python

#279

Post by rbanffy » Fri, Nov 05, 2021, 7:37 AM UTC

miohtama wrote: post #239

sandGorgon wrote: post #145

Earlier quoted context omitted.

Does everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ? I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL). What does the devops there look like ? Snapshot every minute ?

ZODB is the object oriented database as a giant pickle dump. Surprisingly, it works and scales wuite well. The downside is that non-Python tools cannot access it all. https://zodb.org/en/latest/

I remember explaining our tech stack (Python and Zope) to clients.

“Where is the code for that page?”

“It’s in the database”

“Oh… Like MySQL?”

“No. It’s an object database”

“???”

I called it “Martian Technology Syndrome”. But it worked. At later stages we paid the price and had to serialize the datastore for migrations, but that’s what you get for relying on pickles.

Re: An oral history of Bank Python

#280

Post by AmericanBlarney » Fri, Nov 05, 2021, 7:43 AM UTC

qwhelan wrote: post #160

hdjjhhvvhga wrote: post #114

Earlier quoted context omitted.

> but a bank is definitely not one of them Investment banks are basically risk-management shops. The partner made an assessment and evaluated the potential benefits as higher than risks. Note the word "probably".

Also worth mentioning that "unapproved software" on bank infrastructure is what an aggressive prosecutor would call "felony bank hacking".

Almost definitely false. Provided you weren't doing anything intentionally malicious with it, the risk would be that regulators might fine the bank for inadequate controls. As such, the bank might fire you for doing something that could lead to such a situation, but I don't see a criminal charge. There was actually quite a decent bit of "unapproved" software in use at one of the banks I worked in - mostly stuff that was in the process for approval, but that could take forever, so it was reasonably common for teams to run through the checks themselves (security scan, license review, etc) and move forward while the official review confirmed no issues.