An oral history of Bank Python • Hacker News Board

Re: An oral history of Bank Python

#311

Post by harel » Fri, Nov 05, 2021, 10:15 PM UTC

bantatoes wrote: post #304

harel wrote: post #44

I worked on Quartz for a while as a contractor. Hated every second of it. Python version was old (2.4 if I remember correctly when 3.x was already the popular version). But that wasn't it. It was the proprietary version of everything in the stack that got me. Proprietary ide, source control, libs etc. I noted that none of the others who have been there for years have any transferable skills that can cary them out of…

I think this would have depended on the teams/individuals you worked with. I recall some devs being completely clueless about environments - my code (that depends on dozens of other files) in uat is producing different outputs compared to my code in dev, why? Some didn't know how to debug. Many went on to FAANG as senior engineers (including Uber when it was the next big thing), hedge funds (citadel, 2s), startups (t…

I have no doubt the people who BUILT Quartz are top notch. Users of it... Mileage varies i suppose.

Re: An oral history of Bank Python

#312

Post by dolmen » Fri, Nov 05, 2021, 10:25 PM UTC

misja111 wrote: post #59

rich_sasha wrote: post #24

Earlier quoted context omitted.

I think it’s mostly true. The complaint with this kind of “industrial global Python code base”, be it at banks or elsewhere, is that often they are hastily cobbled together and depend on extreme user care to not flop over all the time. I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. When something does break in the end, someone senior just sho…

> I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. It depends which department you are in, but in general: absolutely not. Actually the reverse is true. Banks have huge risks to manage: just imagine for instance what damage a hack of their account system could cause. Or a crash of their payment system. Therefore it is of the utmost importance t…

And that's why many old banks still use COBOL.

Re: An oral history of Bank Python

#313

Post by dolmen » Fri, Nov 05, 2021, 10:42 PM UTC

excitom wrote: post #244

sam0x17 wrote: post #190

Earlier quoted context omitted.

Very true. I often prototype algorithms and things in google sheets. One time I had backpropagation working in there, with a little button to process the next "row" of training samples.

I used to work for a very successful company that produced mobile games. The entire logic of the game, the rules, etc. was all in Excel

So Excel spreadsheets were deployed on mobile devices? What about the runtime?

Re: An oral history of Bank Python

#314

Post by markus_zhang » Fri, Nov 05, 2021, 11:01 PM UTC

esonoi wrote: post #274

markus_zhang wrote: post #107

Earlier quoted context omitted.

But it was good for the first generation who worked there. Big dollars, total control and even the opportunity to create a proprietary IDE...

That proprietary IDE was a piece of crap. Monkey patching was prevalent, exponentially increasing startup time depending on the last time you opened it, and an “online” source tree where one could easily modify the source code in someone else’s ‘private’ workspace. PyCharm was a move in the right direction, but the way it worked was absurd- it would run the internal IDE in the background and sync to the file system.…

I guess it was originally created as a productivity tool as there was not any good Python IDE back before the year 2010. But after a while it became a monstrosity and new tools emerged but it was rooted so deeply that it was impossible to implement the new tools.

But as said I'd really love to work on those projects. Most of the people don't get the prestige to own one's own baby in a big corporation. 99% of the job is maintaining a shit mountain of code and piling new shits on top of it. It really took a lot of luck to be able to make it happen.

I mean even for people who get to work for Jetbrain or Microsoft Visual Studio team, they don't get to create new IDEs, they are buried deeply in a shit mountain of code and JIRA issues.

Plus the pay and vacation is really good for the banks.

Re: An oral history of Bank Python

#315

Post by grvdrm » Fri, Nov 05, 2021, 11:52 PM UTC

3maj wrote: post #301

grvdrm wrote: post #109

I'm curious about experiences in other similar orgs: I work as a portfolio manager for a large reinsurance/insurance company but spend significant time in SQL, Python, Excel (not unexpected I'm sure). The Wapole platform in the article struck a chord with me. We built something roughly similar - call it Trek - that handles jobs. Jobs encompass lots of tasks - reading/writing from Excel, executing SQL, running Python,…

I work in a Technical PM role at a large North American Insurance company and used to work at one of the largest Banks as a Sr. Business Analyst (or Sr.Systems Analyst depending where you are). >lots of our software devs write bad code Ultimately, you get what you pay for, all our full stack devs are making 6 figures... While it might not be FAANG money they also almost never work overtime and the stress levels are r…

> Ultimately, you get what you pay for, all our full stack devs are making 6 figures... While it might not be FAANG money they also almost never work overtime and the stress levels are relatively low.

Yes, it's the blessing and the curse. They are lovely people. They have lives. They aren't working 24/7. But there is misalignment between senior folks who want to innovate and build internal tech around core IP and the talent level of the folks tasked with actually getting that done. Insurance hardly unique in that regard, but an acute issue nonetheless. Firms like GS, JPM, etc have fatter margins (I think) and can afford to pay devs/strats/etc.

Interesting to hear that you went from corporate finance to technical PM. Quite the journey. Would love to dig more into that if you're willing.

Re: An oral history of Bank Python

#316

Post by grvdrm » Fri, Nov 05, 2021, 11:54 PM UTC

3maj wrote: post #303

grvdrm wrote: post #117

Earlier quoted context omitted.

Certainly - let me try to share succinct version germane to my day-to-day - We regularly perform group/segment level risk roll-ups. Involves running computationally expensive (by insurance standards) in-house and third-party models that estimate loss from hurricanes, earthquakes, etc. A lot of our insurance data is unsurprisingly stored in disparate systems that don't talk to each other, and in some cases, don't have…

>read CSV file from another third-party Generally, if you're getting a CSV file from a client that indicates to me that theres a good chance they're exporting info from their systems and sending you the CSV. Have you considered developing a client facing API that can be used to send and digest data?

We have, but that's not something my team (actuarial) can accomplish without help/blessing/oversight from IT.

I don't know all the details but we do have connections to some of our MGAs through XML dumps or perhaps real-time feeds (I'm doubtful). But that data is often missing some of the details as I need. It's useful for policy admin - not for all the other stuff.

We've also explored portals but those are fraught with concerns about double-entry.

Re: An oral history of Bank Python

#317

Post by lmm » Sat, Nov 06, 2021, 12:18 AM UTC

sandGorgon wrote: post #269

lmm wrote: post #256

Earlier quoted context omitted.

It's not a single giant pickle dump; each individual object gets pickled and stored in Minerva (which works more or less like Cassandra or something). It's a pretty similar high level design to what the likes of Google or Facebook do do where you store everything as protobufs in BigTable - the bank uses pickle rather than protobuf because they put a higher priority on being able to store arbitrary objects and deal wi…

So Minerva is like a distributed datastore, specifically for python object storage ? Interesting. Do you think you would do this today with a Cassandra/Hbase? Can it be done - let's say take python 3.10 and the latest Cassandra (or even better - something like Firebase or Cloud Spanner). Just curious that in a post AWS/Firebase world, can something like Minerva be built, without investing in writing the db store grou…

The incarnation of Minerva I worked on actually used Cassandra as its storage backend. But it's something that's not particularly useful piecemeal; the great value of Minerva is that all the bank's data is there and it's all temporal, all access-controlled and all the rest. The most fragile and cumbersome parts of Minerva are the parts where it integrates with an external/legacy datastore - but if you tried to introduce a Minerva-style datastore as a small piece in a system that was otherwise using a "normal" technology stack, those integrations would be most of what you made.

Re: An oral history of Bank Python

#318

Post by collinmanderson » Sat, Nov 06, 2021, 1:53 AM UTC

govg wrote: post #133

seanhunter wrote: post #21

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember) Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. O…

For some context (as an ex-Goldman employee myself), "Armen" in the quote is most probably https://www.goldmansachs.com/insights/outlook/bios/armen-ava... , who has quite a legendary reputation within the firm for the work he's done. He was also one of the first to be hired as a "strat", which used to be how Goldman referred to its quants who sat between front office and tech systems and worked with both sides.

> Prior to joining the firm, Armen was a member of the technical staff at Bell Laboratories in Murray Hill.

Wow. Bell Labs too.

Re: An oral history of Bank Python

#319

Post by 3maj » Sat, Nov 06, 2021, 4:08 AM UTC

grvdrm wrote: post #315

3maj wrote: post #301

Earlier quoted context omitted.

I work in a Technical PM role at a large North American Insurance company and used to work at one of the largest Banks as a Sr. Business Analyst (or Sr.Systems Analyst depending where you are). >lots of our software devs write bad code Ultimately, you get what you pay for, all our full stack devs are making 6 figures... While it might not be FAANG money they also almost never work overtime and the stress levels are r…

> Ultimately, you get what you pay for, all our full stack devs are making 6 figures... While it might not be FAANG money they also almost never work overtime and the stress levels are relatively low. Yes, it's the blessing and the curse. They are lovely people. They have lives. They aren't working 24/7. But there is misalignment between senior folks who want to innovate and build internal tech around core IP and the…

>Firms like GS, JPM, etc have fatter margins (I think) and can afford to pay devs/strats/etc.

From what I've seen on my end the Insurance firms have started paying somewhat of a premium to compensate for the lack of "excitement" that is associated with the insurance industry as a whole.

>Would love to dig more into that if you're willing.

Always willing to chat!

Re: An oral history of Bank Python

#320

Post by mirko22 » Sat, Nov 06, 2021, 4:39 AM UTC

Maven911 wrote: post #273

mirko22 wrote: post #206

Earlier quoted context omitted.

Oh wow, I remember Quartz at BAML… Though this was several years after initial deployment and when core devs left. One day I will sit down and write a small poem about the insanity of software development based on my experience with Quartz. It will be an intriguing story of love and hate being told through a sensual dance between sales and engineering. The battles will not be epic but the consequences of one’s action…

Tell me more about what worked and didn't..I recall the pain of watching QzDesktop load and Bob/HUGS jobs failing..but what else, and what did you enjoy

All great stories use mundane circumstances in life to convey much deeper and abstract ideas.

One such story, I humbly hope, would be my poem.

Yes the b00bs were all over the place as they are in real life, and in my opinion how they got there is as interesting as real life.

Original people solving the original problem had a good understanding of what they are dealing with.

However with subsequent generations it became a monkey problem https://m.youtube.com/watch?v=5QuwPeH9P7Y

When all you have is a DAG all solutions end up with sales person coming at your door problem. But years down the line we didn’t have traveling salesman problem but out of date libraries problem.

The monkeys tried to make their risk taking more safe by introducing all kinda of random constraints not realising that what gave power to the whole idea was actually risk taking.

And no, they didn’t try to manage risk but have completely fail to understand that risk is what made product good in the first place.

And this is how every great idea in human history fell apart, the original people had different view of a current problem however the following generations have only understood the simplified problem and down the line the the solution to original problem had became the actual problem.

I still believe the poem would make greater justice to the whole idea, but what I tried to explain here is that the simple things we have all witnessed actually hide much deeper truths about life in general.

And from there I assume QzDesktop loading times where not the issue at the beginning of this deterministic chaotic system, but simply one of the possible generational products.

I am yet to understand how to solve the monkey issue.