Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Bank with Kubernetes [slides]

community.monzo.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Building a Bank with Kubernetes [slides]

#51

Post by UK-AL »

lgieron wrote: post #46
webscaleizfun wrote: post #35

Earlier quoted context omitted.

It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base. People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to…

I'd say the reliability the bank is looking for is much higher than what's acceptable for web companies (i.e. web companies are fine with eventual consistency, which would is obviously unacceptable in a banking system outside of trivial non-core features).

I would assume its pretty hard to make something ACID style when doing stuff across Microservices, which i'm assuming mondo is doing.

Re: Building a Bank with Kubernetes [slides]

#52

Post by nvarsj »

obeattie wrote: post #10

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

What kind of resource usage do you see with linkerd?

It looks good, but I'm concerned about the resource overhead of a JVM based proxy on every node / pod.

edit: Ok answered myself - found https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniqu....

So with some work it can be reduced from 500Mi->100Mi per instance. It'd be interesting to see the kind of CPU time it uses under load, though.

Re: Building a Bank with Kubernetes [slides]

#53

Post by user5994461 »

nstj wrote: post #17
KirinDave wrote: post #14

Earlier quoted context omitted.

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce vol…

> I tend to think (from experience) that the reason banks don't embrace modern tech is inertia

A lot of inertia. A lot of new-hype-tech is unreliable undocumented shit not ready for production. A lot of new-tech-is-old-tech that's been done for many years but with a new name.

Whenever someone asks "why don't you use Docker for xxx?"

Reply with "When is the last time you had an issue with Docker? Tell me about it."

You'll LOVE the horror stories <3

Re: Building a Bank with Kubernetes [slides]

#54

Post by riskable »

obeattie wrote: post #19
KirinDave wrote: post #14

Earlier quoted context omitted.

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you…

I work for a huge bank and the system of record thing is spot-on. I don't think there's any law that says we must operate systems of record in certain ways or keep records of every little thing but remember that this is banking. Keeping records of every little thing is practically the religion of banking!

So for a long time now the big banks have been trying (and often failing) to keep track--centrally--of every little server or device that pops up on their networks. So there was a big push recently (few years ago) at the big banks to improve their systems of record. The goal being mostly related to better financial (asset) tracking. So they can figure out which internal teams were using the most/least resources as well as figure out who's not upgrading their stuff on a regular basis (technical debt builders).

So they spent all this money improving their systems of record and in walks Docker. It practically turns the entire concept of having a central place to track "systems" on its head!

To give you an example of the difficulties: We have loads of policies that say things like, "all systems/applications must be registered in ." Sounds simple enough: Just make sure that wherever a Docker container comes up we create a new record in the system of record and remove it when it comes down.

Except it's not that simple for many reasons the most obviously problematic of which is that the "system of record" works in batch. As in, you submit your request to add a new record and then maybe 8 hours later it'll show up.

Did I mention that there's also policies that say you can't put any system into production until it shows up in the system of record? =)

That's just scratching the surface though. Because the system of record at most financial institutions doesn't just allow you to delete records. Once you create one it is there forever. It merely gets marked as "retired" (or similar) and most banks require phases as well. For example, before a production system can be marked as retired it must first go through a mandatory, "waning" period (what it's called depends on the bank) that can often be weeks.

I can go on and on about all the zillions of ways in which systems of record (and the policies that go with them) are anathema to usage of Docker but I think everyone reading should "get the picture" at this point. If not, just imagine the havoc that would entail when you have thousands of Docker containers coming up and down every second. Or the entire concept of a container only being up for a few seconds to perform a single batch operation (banks love batch, remember!).

If you think the system of record requirements make adoption of Docker difficult you should know that the security policies are worse! Imagine a policy that states that all systems must undergo a (excruciatingly slow) security scan before being put into production. That's just one of the headaches, sigh.

Re: Building a Bank with Kubernetes [slides]

#55

Post by riskable »

raesene6 wrote: post #39
nstj wrote: post #17

Earlier quoted context omitted.

> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce vol…

I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems. The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong ap…

This is nonsense. Not every system in banks needs to be highly available. That's just silly.

Like the print server on the 3rd floor needs an active standby! Haha.

No, just like any organization banks have "critical" systems and "everything else." Docker is mostly being "sold" as a means to replace and improve the non-critical stuff. Like that internal web app everyone uses to look up . Or the system that generates daily reports on .

Just like most organizations, banks have a few critical systems and everything else is less so (to varying degrees).

Re: Building a Bank with Kubernetes [slides]

#57

Post by jerf »

lgieron wrote: post #46
webscaleizfun wrote: post #35

Earlier quoted context omitted.

It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base. People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to…

I'd say the reliability the bank is looking for is much higher than what's acceptable for web companies (i.e. web companies are fine with eventual consistency, which would is obviously unacceptable in a banking system outside of trivial non-core features).

I'd also suggest that it's not just scale; the kind of reliability Facebook needs is fundamentally different than what a bank needs. Broadly speaking, Facebook needs the site to keep working as well as possible even if some subservice fails, and a bank needs a subservice not to fail. I'm summarizing here and I know it; clearly neither of them is actually on the absolute extreme end, as Facebook needs authentication to work and a bank may not care if the interest rate display widget on their customer banking app fails to load a couple of times. But I'd still suggest there's enough difference between the requirements to be a fundamentally different domain.

Even in "the cloud" things differ between services. A social media app has very different reliability requirements than a backup cloud.

Re: Building a Bank with Kubernetes [slides]

#58

Post by riskable »

kchoudhu wrote: post #34
KirinDave wrote: post #14

Earlier quoted context omitted.

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. Let me put this even more bluntly: if your bank's technology plan isn't 99.5% about dealing with government regulation and maintaining core record integrity and auditability, you are not a serious player in the space.

Man, it's like people think banks are special when it comes to IT. They're not!

You have lots of extra regulations for sure but most of them are about retention of financial records. If a system isn't processing/storing financial records or "privileged" information nobody gives a damn.

The "technology plan" is 99.5% about making or saving money. That remaining .5%? Yeah, that's compliance. Because that's all it costs. Unless you think central logging systems are going to take up some large percentage of a multi-billion dollar quarterly budget?

People love to complain about "the costs of regulation" but you know what? In finance it really doesn't amount much in terms of "how much we spend." How much "it holds back the market" is a different debate entirely.

Aside: Without those regulations we'd just repeat all the same financial disasters throughout history.

Re: Building a Bank with Kubernetes [slides]

#59

Post by raesene6 »

riskable wrote: post #55
raesene6 wrote: post #39

Earlier quoted context omitted.

I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems. The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong ap…

This is nonsense. Not every system in banks needs to be highly available. That's just silly. Like the print server on the 3rd floor needs an active standby! Haha. No, just like any organization banks have "critical" systems and "everything else." Docker is mostly being "sold" as a means to replace and improve the non-critical stuff. Like that internal web app everyone uses to look up . Or the system that generates da…

yes and the topic of discussion in this thread is ..... core banking systems.... which do have to be highly available. If the topic had been bank print servers and I had made my comment yours may have made more sense

My comment didn't say "banks print servers need to be highly available" anywhere.. at all..

Re: Building a Bank with Kubernetes [slides]

#60

Post by alexandre_m »

agibsonccc wrote: post #7
avitzurel wrote: post #5

It can be called: "How we built 'x' with Kubernetes". Really the only thing that is specific to a bank (as I see it) is that they use separate linkerd in order to do the secure stuff. Which is essentially what banks have been doing for ages. I commented before on how Kube has just taken over and beat mesos/marathon stack. This talk is an example to that. You can see how many people jumped on the Kube stack and runnin…

Disclosure: I mainly use DC/OS mesos myself. I've evaluated k8s for our use case and didn't find it was quite what we were looking for. Our customers and stack are mainly JVM based. We do on prem deployments not cloud where GCE is already doing pretty well. We also mainly work with the microsoft side of things (azure,enterprise stuff) Not convinced of this. Direct mesos and yarn integration with spark (not to mention…

What about running K8s on top of DC/OS?

I find the idea of using Mesos as the resources scheduler much more interesting, especially for multi-tenancy where each tenant launch their own k8s cluster on shared infrastructure.

Post reply on HN