Container technologies at Coinbase: Why Kubernetes is not part of our stack • Hacker News Board

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#71

Post by rubber_duck » Mon, Jun 08, 2020, 11:11 PM UTC

thrwn_frthr_awy wrote: post #22

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. I've experienced this as well. At the last large company I worked at we had a…

I've used k8s on an early phase project I later moved off from so I don't have much experience with it but I got the impression that simple scenarios worked like documented. You already had a docker based stack so it sounds like hosted k8s shouldn't be that far off from what you were running.

What issues did you encounter ? I had to spend a week working through it to get familiar with everything but I wasn't experienced with docker previously (I've played with it a few times but never had to setup stuff like custom registry, versioning, etc.)

I did end up in situations where the cluster was FUBAR but I eventually figured out that everyone just recommends rebuilding the cluster over diagnosing random stuff that went wrong during development.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#72

Post by snuxoll » Mon, Jun 08, 2020, 11:13 PM UTC

lifeisstillgood wrote: post #46

flowerlad wrote: post #42

Earlier quoted context omitted.

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(

Not the GP, but I honestly couldn't tell you. A lot probably comes down to tooling, the applications you are deploying, security requirements, etc., as well as how familiar you are with k8s itself.

I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applications critical to our billing operations, and developers on the team deploy new applications with little or no support. Any of the issues I've hit are an artifact of migrating legacy applications not designed to run in more-or-less stateless environments, which is why the PCGW Community site still runs on its own server (Invision Community sucks).

I really don't see all the issues people have that aren't due to a mismatch of application design vs target environment (and no, it's not monolith vs microservice - monoliths run just fine on k8s; but you should be designing your application with a 12-factor environment in mind) or a misguided notion that you will be drowning in YAML hell (it's real, but you can manage it - and it's directly related to the complexity of the services you are deploying).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#73

Post by snuxoll » Mon, Jun 08, 2020, 11:16 PM UTC

dvt wrote: post #32

avita1 wrote: post #29

Earlier quoted context omitted.

I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful). How often/how many cron jobs are you running that spawning a new pod per job is a problem ?

The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.

Why use cronjobs at all for this? This is a classic work queue problem.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#74

Post by snuxoll » Mon, Jun 08, 2020, 11:19 PM UTC

rtempaccount1 wrote: post #9

nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

I'd agree that k8s has a lot of functionality built-in, another important thing to realise is what k8s doesn't do. In addition to the well-known integration points (Container Runtime/Network/Storage Interfaces), there's things like the lack of a good built-in user authentication mechanism with Kubernetes, which means you pretty much always need some external authentication service for your clusters. That's not too ba…

> That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem.

Go spin up Keycloak, join it to your user-directory of choice (or not and just use the internal directory), configure it as your authentication provider, done.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#75

Post by zaptheimpaler » Mon, Jun 08, 2020, 11:19 PM UTC

flowerlad wrote: post #42

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun.

I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS without k8s - read their docs, figure out the API.

Maybe i'm an "old man yelling at new tech" but meh i just see very little value in k8s because you inevitably need to understand the layer beneath - linux (k8s is far from a non-leaky abstraction imo), PLUS all the complexity of k8s itself. I do see the value when managing a big and complex infra with 100s of servers or something, but very few people have that problem.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#76

Post by hwbehrens » Mon, Jun 08, 2020, 11:21 PM UTC

joana035 wrote: post #62

ponyous wrote: post #51

Earlier quoted context omitted.

I feel like what 98% companies need is really just barebones linux with some good documentation how to spawn new nodes. To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often. What I really wonder is where did you learn k8s, which parts did you learn the most? It seems hug…

K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc. For people who already know Linux, kubernetes comes naturally because it is pretty obvious. But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.

Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#77

Post by kureikain » Mon, Jun 08, 2020, 11:21 PM UTC

Looks at what they have built:

- container orchestration platform is Odin + AWS ASGs (auto-scaling groups).

- Codeflow (our internal UI for deployments)

- Odin kicks off a step function and begins to deploy your application. New VMs are stood up in AWS and loaded into a new ASG, your software is fetched from various internal locations, a load balancer starts health-checking these new instances, and eventually traffic is cut over in a Blue/Green manner to the new hosts in the new ASG behind the load balancer.

- To handle secrets and configuration management we have built a dynamic configuration service that provides libraries to all internal customers with a p95 of 6m

- re-scheduling/moving of your containers if your VM dies/becomes unhealthy in your ASG

So if your company has all of that, I agree no reason to use Kubernetes. But what if your company doesn't have any of above system? Kubernetes.

I have used bare ec2 vm, deploy with Ansible, Capistrano, Chef, Docker..all of them. I even roll my own autoscaling with Consul, SQS(for termination notice) and I have more downtime than when using Kubernetes.

With Kubernetes, the learning curve is very high, but once you get it. It's painless to bring in new service, with AWS ACM to terminate TLS, and an ingress controller, pretty much as long as you have a Dockerfile, you can run it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#78

Post by nihil75 » Mon, Jun 08, 2020, 11:26 PM UTC

kureikain wrote: post #77

Looks at what they have built: - container orchestration platform is Odin + AWS ASGs (auto-scaling groups). - Codeflow (our internal UI for deployments) - Odin kicks off a step function and begins to deploy your application. New VMs are stood up in AWS and loaded into a new ASG, your software is fetched from various internal locations, a load balancer starts health-checking these new instances, and eventually traffic…

Odin is just Step Function flow that deploys Auto Scaling groups. It's not that complicated, and not so different from your setup.. main difference is they deploy docker images to ec2 rather than run ansible and configure a running machine.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#79

Post by EdwardDiego » Mon, Jun 08, 2020, 11:27 PM UTC

djsumdog wrote: post #40

EdwardDiego wrote: post #21

> The only way to sanely run Kubernetes is by giving teams/orgs their own clusters Funny, we're running it sanely without doing that. We've separated our clusters based on use-case - delivery vs. back-end, aiming towards the "cell-based" architecture. > Managed Kubernetes (EKS on AWS, GKE on Google) is very much in its infancy and doesn’t solve most of the challenges with owning/operating Kubernetes (if anything it m…

It will catch up with you. I was at one shop with an 11 person platform team dedicated to platforms. The shop moved really fast, and even with 500 employees, they were able to moved from OpenStack to DC/OS in 2~3 months. (We had CoreOS running on open stack, but fully migrated over to DC/OS. Jenkins -> Gitlab also happened very rapidly; really good engineers). At my current shop, we struggle to maintain k8s clusters…

We've used a GitOps model (using Flux[1]) with a reviewer team made of people across our dev teams (and the sysops, natch) to ensure that people aren't just kubectl-ing or helm installing random crap, and we put about 2 weeks of effort into getting RBAC right, so that everyone has read access to cluster resources, but only a subset (generally 1 or 2 per team) have what we call "k8ops" roles - and those are the same people reviewing pull requests in the Flux repo - and the norm is to use the read-only role as default. Only time I've had to recently use my k8ops role was to manually scale an experimental app that was spamming the logs to 0 replicas so the devs responsible could sort it in the morning.

I think the way we've approached it achieves the same goal as just giving each team their own cluster to avoid them messing up other teams.

[1]: https://docs.fluxcd.io/en/1.19.0/

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#80

Post by peterwwillis » Mon, Jun 08, 2020, 11:35 PM UTC

flowerlad wrote: post #42

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

> What is the alternative to k8s that does not need people to have any technical knowledge?

Paying for a managed provider. Heroku, Elastic Beanstalk, GAE, GCP, Fargate, etc. You push some buttons, they manage your cluster/services.

People still think they can get a free lunch by downloading some free software. If that were so, Windows and Mac would be dead and Linux would be the only desktop OS. But good news: I hear 2030 will be the year of the Linux Desktop!