Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Bank with Kubernetes [slides]

community.monzo.com

Post reply on HN
Print view View original

21–30 of 106 posts

Re: Building a Bank with Kubernetes [slides]

#21

Post by agibsonccc »

davidopp__ wrote: post #12
agibsonccc wrote: post #7

Earlier quoted context omitted.

Disclosure: I mainly use DC/OS mesos myself. I've evaluated k8s for our use case and didn't find it was quite what we were looking for. Our customers and stack are mainly JVM based. We do on prem deployments not cloud where GCE is already doing pretty well. We also mainly work with the microsoft side of things (azure,enterprise stuff) Not convinced of this. Direct mesos and yarn integration with spark (not to mention…

We're working on directly-integrated Spark-on-Kubernetes right now and would love to get input from folks who are interested. The Github issue where we're discussing it is here: https://github.com/kubernetes/kubernetes/issues/34377 Comcast has a prototype of YARN on Kubernetes here: https://github.com/Comcast/kube-yarn (Disclosure: I work on the Kubernetes project at Google.)

The YARN stuff is interesting. I would be curious to see something akin to: https://spark.apache.org/docs/1.6.1/running-on-mesos.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn...

My main area of interest with this is: We have a lot of Java native interface code we run. I don't want folks to have to worry about configuring library paths and the like. My support loads get messy quick the second "c code" comes in the picture.

The ability to hook in to k8s to spin up executors would be pretty neat. The Mesos containerizer has that beat right now.

1 of the main reasons I heavily prefer mesos is also dc/os. dcos package install thing is head and shoulders above just having a docker runtime.

As for yarn: A lot of my customers use yarn and don't know what k8s is. OR if they do have a cluster that uses docker, they call it a "docker cluster" and it's often separate. It's not as hard of a sell for me for the YARN folks to just say: "install my docker daemon as an RPM or cloudera parcel so you can run executors"

vs: "Install k8s in your docker cluster with all this extra stuff in there"

K8s for me is in this limbo of: Not integrated enough or too complex for embedding in commercial use.

You guys have a great story going. Especially on the UX side of things. I know a lot of folks in the k8s ecosystem, but many of them aren't focused on big data or the JVM. Mesos on the other hand was spark's parent project. The integration levels show there. I'll be keeping an eye on it :).

My inherent problem with a lot of this is it's still prototyping. A lot of this support is still very green field and I'm stoked you guys are working on it. If anything because more competition is always good.

Re: Building a Bank with Kubernetes [slides]

#22

Post by agibsonccc »

graffitici wrote: post #6
avitzurel wrote: post #5

It can be called: "How we built 'x' with Kubernetes". Really the only thing that is specific to a bank (as I see it) is that they use separate linkerd in order to do the secure stuff. Which is essentially what banks have been doing for ages. I commented before on how Kube has just taken over and beat mesos/marathon stack. This talk is an example to that. You can see how many people jumped on the Kube stack and runnin…

But what about "Big Data" workloads? Running Spark or Cassandra clusters say? My understanding is that having a custom scheduler makes mesos more attractive for those tasks? Has your experience been different?

On prem aka "non aws" it does tend to matter yes.

Re: Building a Bank with Kubernetes [slides]

#23

Post by tluyben2 »

obeattie wrote: post #19
KirinDave wrote: post #14

Earlier quoted context omitted.

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you…

> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements.

Having worked with banks and insurers solely in Java and connecting to legacy (mostly Cobol) before, I was surprised, in my current position, to see some companies doing their complete banking back end in PHP & MySQL. I knew the how is not part of the regulations, but I did expect the CTOs to pick the 'no one ever got fired for choosing' choice.

Re: Building a Bank with Kubernetes [slides]

#24

Post by sandGorgon »

obeattie wrote: post #18
ktamura wrote: post #16

Earlier quoted context omitted.

What are you using for container/pod logging?

Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a fa…

this is awesome! how do you do this? a new header with a unique id... generated by something like lua+nginx. but then how do you pass this request from one service to another?

Re: Building a Bank with Kubernetes [slides]

#25

Post by amatix »

sandGorgon wrote: post #24
obeattie wrote: post #18

Earlier quoted context omitted.

Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a fa…

this is awesome! how do you do this? a new header with a unique id... generated by something like lua+nginx. but then how do you pass this request from one service to another?

For us, a `X-Request-ID` header is generated by any app if it doesn't receive it from upstream -- but normally nginx or the CDN will generate it. There's a few nginx modules to do it, we use https://github.com/newobj/nginx-x-rid-header

Most languages/logging frameworks have some sort of per-thread context (eg. Filters in Python, MDC in log4j, etc) to be able to tag log messages with. If you're using postgresql, you can call `SET application_name='{requestID}';` and that can be output as part of logs too.

Re: Building a Bank with Kubernetes [slides]

#26

Post by foxylion »

sandGorgon wrote: post #24
obeattie wrote: post #18

Earlier quoted context omitted.

Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a fa…

this is awesome! how do you do this? a new header with a unique id... generated by something like lua+nginx. but then how do you pass this request from one service to another?

We do the same in our setup. A Apache HTTPd assignes a unique id (mod_unique_id) as a http request and response header. So any downstream system will get the request header and can attach it to the logs. (In our case we write json log and one field is the request id)

Re: Building a Bank with Kubernetes [slides]

#27

Post by gdubya »

sandGorgon wrote: post #24
obeattie wrote: post #18

Earlier quoted context omitted.

Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a fa…

this is awesome! how do you do this? a new header with a unique id... generated by something like lua+nginx. but then how do you pass this request from one service to another?

There are quite a few monitoring products being built to solve this problem. Many of them are based around Zipkin (http://zipkin.io/)

Re: Building a Bank with Kubernetes [slides]

#28

Post by lobster_johnson »

olalonde wrote: post #13
obeattie wrote: post #10

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

I hadn't heard of linkerd before. How does it compare to etcd or consul and why did you chose it?

Linkerd is not very similar to Etcd or Consul. It's a proxy that sits between your microservices/apps to act as a middleman for RPC calls. Being a proxy, it can handle things like load balancing, throttling and failure handling, meaning you don't have to build that into each app.

When used together with Kubernetes, Linkerd will use Kubernetes for discovery, so you don't need Etcd or Consul directly (K8s itself relies on Etcd, though).

Re: Building a Bank with Kubernetes [slides]

#29

Post by lobster_johnson »

obeattie wrote: post #18
ktamura wrote: post #16

Earlier quoted context omitted.

What are you using for container/pod logging?

Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a fa…

How good is Cassandra at log-like data? Also, why the split between Cassandra and Logstash? Why not a single solution?

Re: Building a Bank with Kubernetes [slides]

#30

Post by KirinDave »

obeattie wrote: post #19
KirinDave wrote: post #14

Earlier quoted context omitted.

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abou…

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you…

> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements.

I'm not sure what your involvement here is, and mine is limited (thankfully) to a substantial distance. However, I think there is a rather big difference between different FI's experience; because it's a true game of politics.

When investigating if Level Money should be a bank (and learning that almost no one wants to be a bank, it's very hard to make money just being a direct-to-consumer deposit bank unless you're quite scaled), I basically had a surprisingly credulous audience because the CFPB was basically willing to doorbuster anything that they thought would spur the big banks to action. It was... very surprising to ultimately decide that it was impossible for financial reasons to actually succeed at being a bank.

But they will insist on things that every bank should have, like a credible way for analytics to run in an environment where the raw data is not subject to exfiltration by a compromised data scientist's machines without an audit trail a mile wide.

Good luck to you.

Post reply on HN