Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An oral history of Bank Python

calpaterson.com

Post reply on HN
Print view View original

251–260 of 333 posts

Re: An oral history of Bank Python

#251

Post by joshuamorton »

sandGorgon wrote: post #145
lmm wrote: post #19

I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based). Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their m…

Does everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ? I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL). What does the devops there look like ? Snapshot every minute ?

This works until...

Specifically, until you realize that pickle changes based on python version, so updating from py3.x to py3.x+1 will prevent your application from reading previously stored data.

Re: An oral history of Bank Python

#252

Post by detaro »

joshuamorton wrote: post #251
sandGorgon wrote: post #145

Earlier quoted context omitted.

Does everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ? I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL). What does the devops there look like ? Snapshot every minute ?

This works until... Specifically, until you realize that pickle changes based on python version, so updating from py3.x to py3.x+1 will prevent your application from reading previously stored data.

This is wrong. pickle can read old files just fine and lets you generate files in old pickle format versions if you require backwards compatibility further than when the current protocol was introduced (it does not get increased with each python version).

Re: An oral history of Bank Python

#253

Post by lmm »

pphysch wrote: post #180
lmm wrote: post #20

Earlier quoted context omitted.

> The part about programming languages frequently not having tables is interesting. The closest as mentioned is the hash, but you lose so much in that abstraction eg the relational aspects. The counter argument then becomes the obvious: why aren't you using a database library, or in a pinch, sqlite? Rightly so. Why would you add relational tables to python rather than have a generic python database spec or a collecti…

> The separate datastore is the problem to be solved here - databases, especially relational databases, are extremely poorly integrated into programming languages and this makes it really painful to develop anything that uses them. Hence "Active Record" ORMs like Rails and Django being highly successful. They functionally embed the RDBMS into the language/app (almost literally if using SQlite), which is a huge boon f…

> Hence "Active Record" ORMs like Rails and Django being highly successful. They functionally embed the RDBMS into the language/app (almost literally if using SQlite), which is a huge boon for developer productivity...

Right, those are a step in the right direction, but still a lot more cumbersome than properly integrating your datastore with your application.

Re: An oral history of Bank Python

#254

Post by lifeisstillgood »

airstrike wrote: post #201
lifeisstillgood wrote: post #194

Earlier quoted context omitted.

Sean ? Spectre ? Sexism ?

Sean would be a weird codename and Spectre sounds a bit too ominous... We'll have to settle for Sexism

Welcome to your first day on Project Sexism.

Our PERT charts are really pert!

We burn story points not bras round here !

Agility is a core value ... nudge nudge ;-)

... oh well, The 1950s want their jokes back.

Re: An oral history of Bank Python

#255

Post by tim333 »

sizzle wrote: post #247
simonh wrote: post #226

Earlier quoted context omitted.

This is only my opinion, but I think the reason Armen said it like he did was because by not making it an order he's giving Sean the option of not doing it, if he's not up for accepting the risk. However the risk was both of them could have got fired. Armen must have known people would know Python had been put on these machines and that he authorised it, in fact what's the point of putting it on them if nobody knows…

Exactly, so what is this Armen character getting out of this other than a potentially big amount of liability and unarticulated risk. The OP said he told people openly that Armen told him he could do it when asked. This makes no sense to me, what’s the upside to Armen? If he is business savvy, he needs to be gaining something in exchange for having his name thrown around by OP as signing off on this.

Guess the Python was useful? From an FT article:

>In 2011 Goldman Sachs put its top computer wizard, Armen Avanessians, in charge of the division. He has helped turn round its fortunes. The arm’s assets under management reached a nadir of $38bn in 2012, but it now manages $91.8bn...

Re: An oral history of Bank Python

#256

Post by lmm »

sandGorgon wrote: post #145
lmm wrote: post #19

I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based). Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their m…

Does everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ? I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL). What does the devops there look like ? Snapshot every minute ?

It's not a single giant pickle dump; each individual object gets pickled and stored in Minerva (which works more or less like Cassandra or something). It's a pretty similar high level design to what the likes of Google or Facebook do do where you store everything as protobufs in BigTable - the bank uses pickle rather than protobuf because they put a higher priority on being able to store arbitrary objects and deal with robustness/compatibility later, rather than having to write a proto definition and a bunch of mapping code up front. You wouldn't want to use a relational database because they're not properly distributed (and, frankly, kind of bad and overrated).

The Minerva I worked on was temporal and append-only, like a HBase that never did compactions (so "delete" actually just writes a tombstone row at a particular timestamp - there was an "obliterate" command but you needed special authorization to use that), and it was distributed (with availability zones even) so you didn't really worry about losing data; loading data as-of a particular timestamp was part of every query (and implemented efficiently). There were probably regular dumps somewhere too but I never needed to encounter those.

Re: An oral history of Bank Python

#257

Post by luord »

I have been a developer for eight years and yet I still get shocked about the places where Python will be used. I mean, it is my favorite language, but in the communities I gravitate towards (basically communities like HN) it has so many detractors for being dynamically typed, not being functional enough, being slow, etc, that sometimes I'm tempted to think maybe it's actually a guilty pleasure of mine, and that I should look for better pastures.

Then I read articles like this and I remember why I like it: it gets the job done, and quickly (for the developers at least). It's why it's so widely used and keeps climbing. Of course, nothing wrong with learning other languages and I do try to keep up, but Python will remain my go-to for the time being.

Re: An oral history of Bank Python

#258

Post by jb_s »

smallnamespace wrote: post #219
seanhunter wrote: post #21

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember) Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. O…

Armen was very passionate about the value of "strats" (GS's own term for "quants", and later broadened to include software engineers and data scientists). A favorite quip of his: At GS, I'm like an arms dealer. When a desk has a problem, I send in the strats, and they blow away all the competition! Also, SecDB's core idea is not just tight integration between the backend and development environment, but that all obje…

This immediately struck me when I was reading this article.

To be honest, this whole paradigm seems absurdly fucking efficient for the developers. But I wonder about stuff like

* What happens if the data model needs to change? If you need to move something from db["some/path"]?

* How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?

Re: An oral history of Bank Python

#259

Post by mst »

luord wrote: post #257

I have been a developer for eight years and yet I still get shocked about the places where Python will be used. I mean, it is my favorite language, but in the communities I gravitate towards (basically communities like HN) it has so many detractors for being dynamically typed, not being functional enough, being slow, etc, that sometimes I'm tempted to think maybe it's actually a guilty pleasure of mine, and that I sh…

People who make complaints like that are privileging their own personal aesthetics over pragmatism.

Same mistake as the people who keep talking about perl being "dead" while they're deploying their production platforms on debian or red hat based systems and ignoring the fact that the packaging and release QA work for those distros is substantially dependent on - actively maintained by the distros in question - perl projects.

Re: An oral history of Bank Python

#260

Post by lmm »

andrewshadura wrote: post #81
lmm wrote: post #19

I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based). Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their m…

> python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages I'm not sure how to react to that, but these features in Python are miles ahead of what many other languages have (or actually don't have).

Python is literally decades behind. Dependency resolution is nondeterministic by default. The way you run a build is still not remotely standard (e.g.: I've downloaded the source of one of the top 20 packages on PyPI. How do I run the tests? Perl had a standard way to do that back in the '90s). Deployment is so bad that people recommend using containers as a substitute for something like fat jars.
Post reply on HN