An oral history of Bank Python • Hacker News Board

Re: An oral history of Bank Python

#121

Post by zzzeek » Thu, Nov 04, 2021, 1:21 PM UTC

rich_sasha wrote: post #3

Compared to one major IB bank Python system, this is all extremely clean and neat. Consider a Python API that is a thin wrapper on COM calls intended to be used from Excel. Want to request some data? Fill in a 2D virtual Excel table. Want to pull some data? Query it and parse a text-dump of a table excerpt (remembering to parse #NA! Etc as nans). Want to automate a job? Enter it as a new row to a global spreadsheet.…

this is the approach I'm familiar with, but back when I did it in Perl (yes Win32 perl with COM bindings) and Java (we wrote native Java plugins to do COM to Excel). all the important stuff is Excel VBA code that they spent years developing and can never replace, so any front end type of thing had to somehow get back to the Excel models.

We eventually did rewrite the Excel models in Java, released something, and then the whole project probably got cancelled or something, 9/11 happened a few years later and the whole building in which all this code was written had to be demolished.

Re: An oral history of Bank Python

#122

Post by seanhunter » Thu, Nov 04, 2021, 1:23 PM UTC

vba wrote: post #119

seanhunter wrote: post #21

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember) Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. O…

What year was this out of interest?

I think 2002 or 3. I worked there from 2001 to about 2010 and it was pretty early in my time there.

Re: An oral history of Bank Python

#123

Post by simonh » Thu, Nov 04, 2021, 1:23 PM UTC

odiroot wrote: post #98

urban_winter wrote: post #53

BAML Quartz was conceived by a bunch of front-office quants who had not the first idea about the software needs of a big bank beyond the front office. There was an arrogant assumption that front office software is obviously the most complicated/difficult variety of software within a bank and therefore any system designed with front office requirements at the forefront would, of course, be perfect for universal use. T…

Can you have a career in finance as an engineer with Python and without C/C++ (professional) experience? Your post really made think it, it's an attractive area to work in.

Also consider Application Support. I know it's not sexy rockstar dev stuff, but if you can get into App Support on the Quartz (or Athena I suppose) environments you get a dev account and access to all the tools. You can view all the code, config and running systems. If you have a good relationship with your dev team you can submit patches e.g. to improve logging. The live log files of all your applications are just a URL away.

If you're up for it, you'll spend a significant amount of time in the Quartz IDE. There are teams within App Support that develop monitoring and compliance reporting tools in Qz and do about 50% development. I know because I ran one. One of my team transferred into our dev team.

Re: An oral history of Bank Python

#124

Post by pmontra » Thu, Nov 04, 2021, 1:29 PM UTC

Icathian wrote: post #115

danmur wrote: post #92

Earlier quoted context omitted.

I think there are too many options, or not enough direction for busy people. Once you understand how it all works and pick / build the right tools it all works pretty well.

I would agree for all but deployment. I know my way around python reasonably well, but pyinstaller and friends still make me have bad days pretty regularly.

Four Python projects, same customer, five different deployment systems. Docker, a Capistano look-a-like I coded in bash, git pull (their former standard), git format-patch plus scp, zip archives. Yes, python file.zip works if it contains the right files. Probably the latter is the easiest way, except it doesn't address the dependencies.

Re: An oral history of Bank Python

#125

Post by ttyprintk » Thu, Nov 04, 2021, 1:33 PM UTC

lmm wrote: post #20

sfgweilr4f wrote: post #5

I can see the benefits of this collection of tools within an all-in-one monolith. Ease of deployment is a big benefit. I can also see the costs. As a stack its probably better in some ways than how a lot of other businesses operate as well as worse. There's probably a lot both ways. The mainframe mindset might be a factor here as well. The giant mainframe where all the magic happens is still a thing to behold and thi…

> The part about programming languages frequently not having tables is interesting. The closest as mentioned is the hash, but you lose so much in that abstraction eg the relational aspects. The counter argument then becomes the obvious: why aren't you using a database library, or in a pinch, sqlite? Rightly so. Why would you add relational tables to python rather than have a generic python database spec or a collecti…

The first-blush conversion from Excel to this ecosystem only needs lookup tables. Excel has some static database I/O, but people who only know Excel use it as dat input for lookup tables.

The Python results of that first conversion need to test against Excel, so it’ll have identical lookup tables.

Re: An oral history of Bank Python

#126

Post by mumblemumble » Thu, Nov 04, 2021, 1:34 PM UTC

> There is an uncharitable view (sometimes expressed internally too) that Minerva as a whole is a grand exercise in NIH syndrome.

My brief experience with this (in an adjacent area - proprietary trading) was that the more charitable view is that these firms need to be able to fully own their software stacks, and have the resources to pay for that luxury.

Reading these descriptions from the article, I can't help drawing a connection to the Smalltalk ecosystem. It sounds like, to at least some extent, what these banks have built is a system that exhibits many of the more interesting characteristics of an enterprise Smalltalk system, only on top of a tech stack that they could own from top to bottom.

Re: An oral history of Bank Python

#127

Post by ttyprintk » Thu, Nov 04, 2021, 1:35 PM UTC

kwertyoowiyop wrote: post #108

captainmuon wrote: post #18

Reminds me of what we used at the ATLAS experiment at CERN* . Python was tightly integrated with the application framework, Athena (which I just realize has the same name as JPM's Python framework!). You could use it as a job description language, and you would compose computation steps from classes you could write in C++. I think there was a separate `athena` executable that was just python with some packages pre-lo…

The Greek and Roman gods have always been a go-to for project names, LOL. We need to give some other cultures a shot!

With particularly poor timing, I chose the Egyptian goddess Isis.

Re: An oral history of Bank Python

#128

Post by Chris2048 » Thu, Nov 04, 2021, 1:42 PM UTC

simonh wrote: post #41

rich_sasha wrote: post #24

Earlier quoted context omitted.

I think it’s mostly true. The complaint with this kind of “industrial global Python code base”, be it at banks or elsewhere, is that often they are hastily cobbled together and depend on extreme user care to not flop over all the time. I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. When something does break in the end, someone senior just sho…

You could not be more wrong. Quartz has first class documentation, solid tooling, very well thought out and rigorous code review and access controls. Banks are regulated up to the eyeballs, so everything has to be audited and justified in detail. It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but t…

> Quartz has ... solid tooling

Is that how you'd describe the IDE and (integrated) VCS?

Re: An oral history of Bank Python

#129

Post by Chris2048 » Thu, Nov 04, 2021, 1:51 PM UTC

harel wrote: post #44

I worked on Quartz for a while as a contractor. Hated every second of it. Python version was old (2.4 if I remember correctly when 3.x was already the popular version). But that wasn't it. It was the proprietary version of everything in the stack that got me. Proprietary ide, source control, libs etc. I noted that none of the others who have been there for years have any transferable skills that can cary them out of…

> Proprietary ide, source control, libs etc

The reason this was a problem was that it meant investment was needed for each of these things, and as such fell behind.

The IDE fell behind most modern IDEs, presumably because it didn't get the budget for it. The source control/ libs where usually modified versions of existing libs, but now needing to be maintained internally to remain compatible with the mainstream versions (which, again, they did not, and so fell out of compatibility).

> any transferable skills

It puts you in a position of arguing that you are familiar with , just a modified proprietary version of it.. Makes those conversations a bit more difficult..

> All were good devs but only knew quartz

To be fair - this is their own fault. It's difficult providing proof in terms of "what you worked on in your last job"; but there's no reason a professional python dev couldn't become familiar with the popular versions of things on their own, given they are fairly close in functionality. Many of the quartz devs I knew already had backgrounds in Python, attended pycons etc; so knew more than Quartz.

Re: An oral history of Bank Python

#130

Post by klelatti » Thu, Nov 04, 2021, 1:52 PM UTC

grvdrm wrote: post #117

klelatti wrote: post #110

Earlier quoted context omitted.

Really interesting comments. I'd guess that you have proprietary (e.g. valuation / capital calculation) systems that need to interface with Trek in some way. Could you share how you've approached that at all? Also not clear why R couldn't be added to Trek alongside Python and C#?

Certainly - let me try to share succinct version germane to my day-to-day - We regularly perform group/segment level risk roll-ups. Involves running computationally expensive (by insurance standards) in-house and third-party models that estimate loss from hurricanes, earthquakes, etc. A lot of our insurance data is unsurprisingly stored in disparate systems that don't talk to each other, and in some cases, don't have…

Thanks for a really comprehensive reply. Enjoyed your comment on PowerBI.

My background is mainly life which is dominated (at least in Europe) by computationally demanding proprietary liability modelling systems but I think Python / R is getting a foothold in capital calculation / aggregation.

My perception that there is a lot more use of in-house models in the GI / property-casualty worlds so more Python etc but sounds like you still have to interface with proprietary modelling systems.