Live data from Hacker News

Building a Bank with Kubernetes [slides]

community.monzo.com

71–80 of 106 posts

Re: Building a Bank with Kubernetes [slides]

#71
post #46

Earlier quoted context omitted.

I'd say the reliability the bank is looking for is much higher than what's acceptable for web companies (i.e. web companies are fine with eventual consistency, which would is obviously unacceptable in a banking system outside of trivial non-core features).

Why would eventual consistency be unacceptable in a banking system? In my experience people interact with social media on far shorter time scales than their banks. When they post a new Instagram photo, they expect that their friends will see it basically instantaneously. In comparison, when people use their debit card at CVS, they're not expecting anyone to log into their bank account seconds later and see the charge…

You're actually somewhat right! ATMs are an (sometimes) an example of eventual consistency. If an ATM is offline, it'll often allow you to make a withdraw anyways and once it's back on the network report back. That could mean an overdraft for you. Caveat here is that these are often low-traffic ATMs on the periphery, ones in the city are usually making calls home to check balances.

However, the buck (no pun intended) has to stop somewhere. Overdraft limits have to be consistently applied. Even that is somewhat up in the air. Take this with a grain of salt as it's second-party information, but my wife works in fraud prevention at a smaller credit union. She says that transactions are collected throughout the day and overdrafts are only applied at the end of the day to allow for bills to drain your account beyond its capacity and then payroll to land without applying overdrafts unless you're in the red afterwards. In some sense, that's even "eventual consistency" on the scale of 24 hours.

The most important thing in banks is that at the end of the day, the balance sheet, well, balances. And they limit their liability by preventing too much overdraft and applying daily limits to ATM withdrawals. I pose that general eventual consistency fits that pretty well, as long as "eventual" isn't "hours" for the most part.

A little more on eventual consistency in general as I understand it, eventual consistency systems come in many forms. In a leader/follower setup (think MySQL w/ async replication), usually "important" calls are made to the leader in a consistent fashion and changes are asynchronously replicated to the followers for general read fanout. There are a lot of different kinds of systems with different guarantees. In a dynamo-style system, writes/reads are usually done to a quorum of replicas (e.x. 2/3 replicas), and only if the read from the two replicas disagree are the values on all three replicas "repaired" via last-write-wins. Facebook has a model they call causal consistency[1] which models causal relationships (e.x. B depends on A, therefore B isn't visible until A is also replicated).

You can consider any system with a queue or log in it that doesn't provide some token to check for operation completion to be eventual. For example, imagine you fronted DB writes with Kafka. Lag between writing to Kafka and commit into the DB may only be 100ms, but that's "eventual". However, if you provided back a "FYI, your write is offset 1234 on partition 5", you could use that as a part of a read pipeline that checked that the DB writer was beyond offset 1234 on partition 5 before allowing the read to proceed. That'd be consistent.

[1] http://queue.acm.org/detail.cfm?id=2610533

Re: Building a Bank with Kubernetes [slides]

#72
post #57
post #46

Earlier quoted context omitted.

I'd say the reliability the bank is looking for is much higher than what's acceptable for web companies (i.e. web companies are fine with eventual consistency, which would is obviously unacceptable in a banking system outside of trivial non-core features).

I'd also suggest that it's not just scale; the kind of reliability Facebook needs is fundamentally different than what a bank needs. Broadly speaking, Facebook needs the site to keep working as well as possible even if some subservice fails, and a bank needs a subservice not to fail. I'm summarizing here and I know it; clearly neither of them is actually on the absolute extreme end, as Facebook needs authentication t…

"Broadly speaking, Facebook needs the site to keep working as well as possible even if some subservice fails, and a bank needs a subservice not to fail."

One of the reasons many of them stick with mainframes, AS/400's, and NonStop systems for backends. ;)

Re: Building a Bank with Kubernetes [slides]

#73
post #68

Earlier quoted context omitted.

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. Let me put this even more bluntly: if your bank's technology plan isn't 99.5% about dealing with government regulation and maintaining core record integrity and auditability, you are not a serious player in the space.

Matt Levine's comment always stick in my head: 'I say sometimes that the tech industry is about moving fast and breaking things, while "finance is an industry of moving fast, breaking things, being mired in years of litigation, paying 10-digit fines, and ruefully promising to move slower and break fewer things in the future."'

That's a good comment. Applies to most banks. Then there's Goldman Sachs. ;)

Re: Building a Bank with Kubernetes [slides]

#75
post #17

Earlier quoted context omitted.

> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce vol…

> I tend to think (from experience) that the reason banks don't embrace modern tech is inertia A lot of inertia. A lot of new-hype-tech is unreliable undocumented shit not ready for production. A lot of new-tech-is-old-tech that's been done for many years but with a new name. Whenever someone asks "why don't you use Docker for xxx?" Reply with "When is the last time you had an issue with Docker? Tell me about it." Yo…

"You'll LOVE the horror stories It's even better when you're on a site like Hacker News with stronger-than-average, technical people. You get to read endorsements of the latest and greatest with people asking why (insert failure of common action here) is happening... in the same thread. I'll just stick with a hardened, flexible configuration of classic architecture and components that work.

Re: Building a Bank with Kubernetes [slides]

#76

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

If K8S secrets management is storing them in plain-text in etcd how do you go about making it actually secure for a bank?

Re: Building a Bank with Kubernetes [slides]

#77
post #68

Earlier quoted context omitted.

Matt Levine's comment always stick in my head: 'I say sometimes that the tech industry is about moving fast and breaking things, while "finance is an industry of moving fast, breaking things, being mired in years of litigation, paying 10-digit fines, and ruefully promising to move slower and break fewer things in the future."'

That's a good comment. Applies to most banks. Then there's Goldman Sachs. ;)

Oh, what I said was coloured explicitly by my experiences working on regulatory development at Goldman.

Everything was filtered through the lens of "What would the regulators say? And how do we tell the regulators what we're doing?"

Re: Building a Bank with Kubernetes [slides]

#78
post #32

Earlier quoted context omitted.

> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. I'm not sur…

> I'm not sure what your involvement here is … Parent poster is Head of Engineering at the bank (not CTO?), and author of the slides.

Yes but that may or may not mean direct involvement with regulators. For small companies it's really hard to tell where they are in the process.

Re: Building a Bank with Kubernetes [slides]

#79

Earlier quoted context omitted.

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. Let me put this even more bluntly: if your bank's technology plan isn't 99.5% about dealing with government regulation and maintaining core record integrity and auditability, you are not a serious player in the space.

Man, it's like people think banks are special when it comes to IT. They're not! You have lots of extra regulations for sure but most of them are about retention of financial records . If a system isn't processing/storing financial records or "privileged" information nobody gives a damn. The "technology plan" is 99.5% about making or saving money. That remaining .5%? Yeah, that's compliance. Because that's all it cost…

If it's a computer in a bank and it touches risk, trading or treasury, it's fair game for the Fed auditor.

So you tell me: what computer system of any import in a bank doesn't touch one of these three things?

Re: Building a Bank with Kubernetes [slides]

#80
post #17

Earlier quoted context omitted.

> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce vol…

> I tend to think (from experience) that the reason banks don't embrace modern tech is inertia A lot of inertia. A lot of new-hype-tech is unreliable undocumented shit not ready for production. A lot of new-tech-is-old-tech that's been done for many years but with a new name. Whenever someone asks "why don't you use Docker for xxx?" Reply with "When is the last time you had an issue with Docker? Tell me about it." Yo…

Yes I'd hesitate to use Docker in a system I wanted to maintain for 10 years. We can't get consistent interfaces for 2 months!
Post reply on HN