Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

151–160 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#151
post #136
post #47

Earlier quoted context omitted.

Been there. Simple python django app. Max 20 APIs. Expecting 100 requests per month. But POs and directors wanted shiny DevOps tools. Problems with typical MNCs. Every quarter manager/director comes with some new hype.

If it's a hosted k8s, all you'd do is containerize your application, then create a deployment that pulls that container and exposes it through a loadbalancer. The container is the same if you'd push it to herokus container registry or beanstalk

Heroku I configure external items with some environments variables. Often there is a simple command to do it for me.

Helm. I’m staring at 20 pages of Yaml. Most of it seems arbitrary. Crap ton of interconnections. Plus defining all iam roles.

Doesn’t help that devops team keeps writing custom scripts to interact with it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#152
post #131
post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

Could you share the resources to get started? I'm in a similar boat, 2 person trying to set up K8S in DO using automation (CI/CD).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#153
post #57

We haven't even got to ops stage yet. Small team has spent 6 months trying to repackage our app into openshift and still barely works. I think everyone regrets even looking at it.

What sort of challenges have you run into? What are some of the things the "small team" have done for the last 6 months?

Networking mostly, RMI, JMX. Its our internal rules too, getting egress to work on our segregated networks.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#154
I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics.

Introducing Kubernetes into our org allowed us to do this; we would have never gotten here with our legacy deployment and orchestration Frankenstein. The change has been so positive, that I adopted Kubernetes for my solo projects and I am having a blast.

I understand Coinbase's position, and they need to stick to what works for them. I just wanted to bring up a positive POV for a technology I am becoming a fan of.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#155
post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

Could you share the resources to get started? I'm in a similar boat, 2 person trying to set up K8S in DO using automation (CI/CD).

Sadly I don't have many resources I can refer you to (maybe someone else can add?) but digitaloceans kubernetes guides are excellent so I'd start with that: https://www.digitalocean.com/docs/kubernetes/how-to/

k8s has a loooot of stuff it can do and reading too many blog posts that go into too much detail can be intimidating, so my advice would be:

1. Create a dummy cluster on digitalocean (or docker desktop with k8s support / minikube), then setup kubectl to connect to it.

2. Start with the difference between resource types: What are deployments, what are pods?

3. Create a deployment manifest that just tries to pull a container from some registry, apply it with kubectl -f foo.yml

4. Play with kubectl to inspect things: kubectl get pods, kubectl get deployments, kubectl describe pod xxxxxx, kubectl logs xxxx

5. Learn how kubernetes ties things together through labels: Create a ClusterIP/LoadBalancer service and try to get it to balance to your pods from above through labels (https://www.digitalocean.com/docs/kubernetes/how-to/add-load...)

Deployments/services/(pods) are all you need in the beginning for running containers on k8s and exposing them. Of course then there are things like persistent storage but if your app is made to run ephemeral, you likely have storage/db setup externally already.

For running through the CI, once you have your manifests you could run kubectl apply directly through the CI if you wanted to. We are using terraform in front of k8s with hashicorps hosted state, then run `terraform plan`. If it passes, on dev we automatically apply to the cluster, on prod there is a manual step through the hashicorp admin UI that needs an apply trigger. Then there are more advanced tools like spinnaker that can be used to setup more complex pipelines on what to do on push.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#156
1) Those guys can't imagine their life outside of amazon. this is bad. And yes, amazon k8s was bad, it's getting better, still bad. 2) They said we need a separate team and instead they wrote and maintain their own, sort of lightweight solution. Nothing more about that article. One opinionated team decided to roll out something homemade to add it to CV later on.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#157

Thanks for the article. I've been in tech for the past 10 years, working in or around devops teams for the most part but I don't get all the fuss about k8; yes it's an amazing tool doing a lot more than any other. But it has a big learning curve and setup & maintenance are very costly. I don't understand why most orgs are moving to k8 considering this. When talking to my peers I often see numbers like 6 months to 2 y…

There is a learning curve either way. It’s either an in house container orchestration platform or one of the open source ones. Coinbase seems to have chosen the former.

K8s is simply a good default environment which provides rock solid stability for your applications by outsourcing the distributed systems complexity to your infrastructure team (whether its internal to your company or to a managed one like GKE). Teams are not using it just because it’s “cool” (maybe some are), there is no need to develop in house strategies to deploy and keep an app running and scale it (among other things; this is the lowest common denominator).

It’s the same reason why big data tech has somewhat standardized on a set of tech (spark, airflow etc): once people learn the system, they can focus on building products that provide value rather than building the products and the relevant infrastructure.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#158
post #92

I partially blame keyword driven recruitment in tech for these kinds of responses to a platform/tooling. Kubernetes isn't a magic bullet - it is a platform which solves a very specific set of problems with scaling. And of course it doesn't come for free. You can't just throw in k8s into your existing infrastructure and expect your devs to manage it, in addition to their regular work. And yet, we keep reading about te…

Oh God. This is what is happening at my work. We have an API that has 200 write users and a public front-end that can do reads. None of it is heavy though, with most writes occurring for a month in the Winter and a month in the Fall. In the unlikely event of heavy write loads we could just scale up CPU/RAM for those two months. Any read load be solved with cache or spending time on the worst offenders in SQL. The Lead Dev is gung-ho that it MUST be a micro-service with K8, Kafka, and I'm sure a bunch of other shit we don't need for what is the same application that has been being written for half a century. Data in/out with business logic applied. The entire API has about 8 paths, with your basic HTTP methods for each. Its probably the smallest API I've ever worked on.

The positive for me is I am learning a bunch of stuff during the process. The negative for the project is he expects developers with no previous skillset in this space to design all this new (for us) tech without introducing technical debt. The downside for the client is...well guess.

It's sad. I am rewriting a problematic legacy application and creating a whole new one. I guess I can put K8 and the kitchen sink on my resume when I look for a new job just before this thing implodes on release though. I just wanted to write clean code man, thats all. There is a sadistic part of me that wants to grab some popcorn and see how this explodes in his face...there is also a part of me that wants to be proved wrong...I hate this job.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#159
post #87

imo running k8s in the cloud makes no sense. If you've already setup on prem servers, load balancers, config management, patching, access control, etc to allow developers to run applications on VMs k8s can provide an integrated experience with significantly less work. If you're running k8s in the cloud then just use a hosted service and leverage enterprise support. In the on prem case, you already have a dedicated in…

The most popular cloud product (aws) has a really shitty managed k8s offering. This is perhaps the main reason so many infra teams have spent so much collective time into building k8s. If eks was as good as GKE, what you said is absolutely spot on.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#160
post #136

Earlier quoted context omitted.

If it's a hosted k8s, all you'd do is containerize your application, then create a deployment that pulls that container and exposes it through a loadbalancer. The container is the same if you'd push it to herokus container registry or beanstalk

Heroku I configure external items with some environments variables. Often there is a simple command to do it for me. Helm. I’m staring at 20 pages of Yaml. Most of it seems arbitrary. Crap ton of interconnections. Plus defining all iam roles. Doesn’t help that devops team keeps writing custom scripts to interact with it.

On k8s, you write those env variables into the deployment manifest that describes the container. If they are sensitive, pack them into a k8s secret and add that to the container instead.

You can use the external services the same way if they are already setup with things like DATABASE_URL that holds a postgres connection string to amazon RDS, etc. Running k8s doesn't mean you have to move your entire stack into k8s, you can still use hosted services just fine.

In all my time with k8s, I never used Helm. If you don't need it, don't use it and keep it simple. k8s can do a ton of stuff but in reality, you barely need more than the basic.

Post reply on HN