Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Bank with Kubernetes [slides]

community.monzo.com

Post reply on HN
Print view View original

31–40 of 106 posts

Re: Building a Bank with Kubernetes [slides]

#31

Post by obeattie »

lobster_johnson wrote: post #29
obeattie wrote: post #18

Earlier quoted context omitted.

Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a fa…

How good is Cassandra at log-like data? Also, why the split between Cassandra and Logstash? Why not a single solution?

Because of its disk layout, Cassandra is truly excellent at time-series, append-heavy data. In our setup, the data is partitioned by time bucket and n number of labels (one of which is the request ID).

We may unify the two at some point, but there's no immediate need to do so. While the write use-case is quite similar across both, the read use-case is quite different: slog requires reasonably low latency reads soon after the data is written, data can age out after 2-30 days depending on severity, and sometimes dropping events is acceptable. It would be acceptable for reads from the "archival" system to take minutes or even hours, the data should be kept forever (or for a long time), and dropping events is never acceptable.

Re: Building a Bank with Kubernetes [slides]

#32

Post by mjs »

KirinDave wrote: post #30
obeattie wrote: post #19

Earlier quoted context omitted.

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you…

> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. I'm not sur…

> I'm not sure what your involvement here is …

Parent poster is Head of Engineering at the bank (not CTO?), and author of the slides.

Re: Building a Bank with Kubernetes [slides]

#33

Post by runeks »

harshreality wrote: post #4
0xmohit wrote: post #3

Is there a video available elsewhere? Last I checked, it wasn't possible to download those from SkillsMatter.

% youtube-dl 'https://skillsmatter.com/skillscasts/9146-building-a-microservices-with-kubernetes' [generic] 9146-building-a-microservices-with-kubernetes: Requesting header WARNING: Falling back on generic information extractor. [generic] 9146-building-a-microservices-with-kubernetes: Downloading webpage [generic] 9146-building-a-microservices-with-kubernetes: Extracting information [vimeo] 188042022: Downloading web…

Cool! Although, seems like "youtube-dl" may not be the right name for that app anymore.

Re: Building a Bank with Kubernetes [slides]

#34

Post by kchoudhu »

KirinDave wrote: post #14
obeattie wrote: post #10

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies.

Let me put this even more bluntly: if your bank's technology plan isn't 99.5% about dealing with government regulation and maintaining core record integrity and auditability, you are not a serious player in the space.

Re: Building a Bank with Kubernetes [slides]

#35

Post by webscaleizfun »

tluyben2 wrote: post #23
obeattie wrote: post #19

Earlier quoted context omitted.

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you…

> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. Having work…

It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base.

People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to every other major company using these building blocks reliably if your investors, CEO, board or auditors asked why you chose to use PHP & MySQL.

In summary, PHP & MySQL have become the modern equivalent of a "safe" choice for your stack to be built on. Its not necessarily a bad choice either, you get access to a large community of skilled people who can write PHP & write SQL statements, and while everyone likes to hate on PHP, it isn't about to up and disappear any time in the next decade either (unlike COBOL).

Re: Building a Bank with Kubernetes [slides]

#36

Post by davidopp__ »

agibsonccc wrote: post #21
davidopp__ wrote: post #12

Earlier quoted context omitted.

We're working on directly-integrated Spark-on-Kubernetes right now and would love to get input from folks who are interested. The Github issue where we're discussing it is here: https://github.com/kubernetes/kubernetes/issues/34377 Comcast has a prototype of YARN on Kubernetes here: https://github.com/Comcast/kube-yarn (Disclosure: I work on the Kubernetes project at Google.)

The YARN stuff is interesting. I would be curious to see something akin to: https://spark.apache.org/docs/1.6.1/running-on-mesos.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn... My main area of interest with this is: We have a lot of Java native interface code we run. I don't want folks to have to worry about configuring library paths and the like. My support loads get messy quick the second "c co…

Regarding packages, you might be interested in https://github.com/kubernetes/charts

Disclosure: I work on the Kubernetes project at Google.

Re: Building a Bank with Kubernetes [slides]

#37

Post by raesene6 »

obeattie wrote: post #10

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

Hi,

I'd be very interested to hear if you've come across any good documentation on hardening Kubernetes?

From what I've seen so far there's very limited documentation on that when compared to other components like Docker, and the defaults can sometimes not be suitable for a high security environment (e.g. https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co... )

Re: Building a Bank with Kubernetes [slides]

#38

Post by agibsonccc »

davidopp__ wrote: post #36
agibsonccc wrote: post #21

Earlier quoted context omitted.

The YARN stuff is interesting. I would be curious to see something akin to: https://spark.apache.org/docs/1.6.1/running-on-mesos.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn... My main area of interest with this is: We have a lot of Java native interface code we run. I don't want folks to have to worry about configuring library paths and the like. My support loads get messy quick the second "c co…

Regarding packages, you might be interested in https://github.com/kubernetes/charts Disclosure: I work on the Kubernetes project at Google.

Thanks! I'll take a look around. There's a lot more than just "apps" at stake here, but again: I'll keep an eye on the progress. We also do a lot with jvm based microservices and the lightbend/play stack which already standardized on DC/OS with conductr. Datatstax enterprise is also of interest to us. A lot of this stuff is already "baked".

There were a lot of little things that made us pick the stack we did - a lot of this is highly specific to our use cases and customer base. We will likely have a k8s version at some point. When that time comes (or if enough customers ask for it) I'll re evaluate what we need on the stack.

Thanks for the links!

Re: Building a Bank with Kubernetes [slides]

#39

Post by raesene6 »

nstj wrote: post #17
KirinDave wrote: post #14

Earlier quoted context omitted.

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce vol…

I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems.

The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong approach.

This tends to lead to things that save money in the short term (e.g. outsourcing deals) but could well cost money in the longer term, as they make replacing legacy systems harder.

Re: Building a Bank with Kubernetes [slides]

#40

Post by ben_hall »

raesene6 wrote: post #37
obeattie wrote: post #10

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

Hi, I'd be very interested to hear if you've come across any good documentation on hardening Kubernetes? From what I've seen so far there's very limited documentation on that when compared to other components like Docker, and the defaults can sometimes not be suitable for a high security environment (e.g. https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co... )

There is a solution to this coming in v1.5 - https://github.com/kubernetes/kubernetes/pull/32518
Post reply on HN