Container technologies at Coinbase: Why Kubernetes is not part of our stack • Hacker News Board

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#191

Post by CraigJPerry » Tue, Jun 09, 2020, 5:08 AM UTC

daze42 wrote: post #133

manigandham wrote: post #99

Earlier quoted context omitted.

You can't make the complexity disappear. Kubernetes just offers a very standardized, stable and relatively polished way to handle it. The other alternatives, like running your own orchestration system, are just as complex if not more so, although you might be more familiar with it since you built it. That being said, many companies don't need any of it in the first place for their scale, and that's probably the bigge…

It's impossible to make essential complexity disappear but it is certainly possible to reduce incidental complexity. Most software is much more complex than it needs to be and Kubernetes is no exception.

Before k8s every serious shop automated the crap out of their infra. Jump/kickstart recipes, rolling cluster patching that split RAID mirrors before applying, blue/green deployment scripts to tickle the loadbalancer, cron jobs to purge old releases ...

That stack is super complex and utterly bespoke to the company.

With k8s it’s standardised and usually better quality.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#192

Post by rudolph9 » Tue, Jun 09, 2020, 5:14 AM UTC

silviogutierrez wrote: post #146

rudolph9 wrote: post #68

One thing people forget is containers don’t necessarily not to be created with Docker. Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we h…

Huge nix fan. Question: how do you deploy these with k8s? Do you just build a NixOS docker image "wrapper" around all the other stuff?

Haven’t actually deployed to k8s, currently using them to isolate local development but there I know (at least at one point) kubernetes supported other images than just docker images but even if they do t NixOS containers supports generating docker images...

I imaging a ci/cd flow where it build the image and installs your app via a local Nix package built from source in your git repo and all that needs to be shared is the git repo and you have everything you need to reproduce the container er running in production.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#193

Post by systematical » Tue, Jun 09, 2020, 5:18 AM UTC

harpratap wrote: post #170

systematical wrote: post #158

Earlier quoted context omitted.

Oh God. This is what is happening at my work. We have an API that has 200 write users and a public front-end that can do reads. None of it is heavy though, with most writes occurring for a month in the Winter and a month in the Fall. In the unlikely event of heavy write loads we could just scale up CPU/RAM for those two months. Any read load be solved with cache or spending time on the worst offenders in SQL. The Lea…

So are you assuming that your service will never grow? Because vertical scaling up is nice and easy as long as you know the limits but once you cross a threshold, not matter how many CPUs you throw at a problem it just won't scale. Your senior lead seems to be anticipating that and preparing in advance.

Who assumes a system will never grow? No one. I'm looking at the 10-year-old legacy system we are replacing. That tells me a lot. Where its been, why its that way, and a lot about where it's likely to go in the next 2-3 years. Its function is actually being reduced in this rewrite.

I'm not against the microservice idea. I just rather focus on solving the problems the legacy system had. None of those had to do with scaling. They were related to a certain federal agency changing their mind every 2-4 years and really poor coding practices. A microservice with kafka and k8 don't really solve those issues.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#194

Post by birdyrooster » Tue, Jun 09, 2020, 5:24 AM UTC

CorpOverreach wrote: post #106

theptip wrote: post #63

Earlier quoted context omitted.

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

> that new team should replace another team that's no longer needed, or more likely, involve a combination of hiring some experts and retraining your existing engineers. The gotcha there is that it rarely goes that way unless you have a very clear direction from senior management, at least at big corps. In most cases, it's just another thing that gets added to the pile, and it's incredibly difficult to migrate entire…

You should always have a next gen and a production version of your infrastructure as a code.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#195

Post by rumanator » Tue, Jun 09, 2020, 5:27 AM UTC

ditonal wrote: post #118

flowerlad wrote: post #100

Earlier quoted context omitted.

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

The difficulty of all that stuff on plain old Linux is overstated and the difficulty of doing that on K8s is understated. And I agree with grandfathers point that if you don’t understand how to do that on plain old Linux you will struggle on K8s. K8s is ok for huge enterprises with tons of engineers but somehow K8s advocates make it seem like learning and operating nginx on ubuntu is this huge challenge when it’s usu…

Your comment makes no sense at all. Kubernetes is not equivalent to running nginx on Ubuntu. This comparison is simply too absurd to the point I really doubt you have any idea of what Kubernetes is designed to do and why people use it.

I mean, who in their right mind would even describe a cluster of VMs and/or COTS hardware running with a VPN and designed to automatically run and manage the lifecycle of containers as well as the computational resources they consume as... Operating nginx on Ubuntu?

If you don't have anything to add to the discussion and your only relationship with the topic is that you are completely unfamiliar and are intimidated by what you don't know then please refrain from increasing the signal-to-noise ratio of the discussion.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#196

Post by punnerud » Tue, Jun 09, 2020, 5:30 AM UTC

K8S is to infrastructure, as SQL is to data?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#197

Post by theptip » Tue, Jun 09, 2020, 5:31 AM UTC

tso wrote: post #117

theptip wrote: post #63

Earlier quoted context omitted.

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

> Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The key is that there is a lot of additional services and interface points to handle. As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that ne…

My personal experience (4 years on GKE in production) has been the opposite; running on k8s has abstracted away a number of things that I’d otherwise have to engineer.

Volumes just get attached (using PersistentVolumeClaims), and automatically migrate to a new node of the original pod dies. Vs. having to do some sort of rsync between nodes to keep disks in sync.

Secrets get encrypted by k8s and mounted where needed. I would agree that RBAC is a bit tricky but I don’t think it’s harder than IAM provisioned with Terraform.

If you are not using a service mesh for your VMs then you don’t need one in k8s. (I don’t use one, and rolled TLS to the pod in less effort than it would take to maintain TLS to the VM). The reason you want a service mesh is to abstract TLS and retry mechanics from the application layer - i.e. make your service authors more productive. If you don’t use a service mesh then you are back to managing TLS per-service, which is where you are with VMs already.

There are definitely more services you _could_ run, but in my experience these are additive, I.e. they are extra work, but give you a productivity boost.

Anyway, YMMV and I haven’t operated a system as large as Coinbase, so I could be missing something. Interested in hearing others’ experiences though.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#198

Post by rudolph9 » Tue, Jun 09, 2020, 5:31 AM UTC

root_axis wrote: post #180

rudolph9 wrote: post #68

One thing people forget is containers don’t necessarily not to be created with Docker. Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we h…

How is a configuration.nix any different than a Dockerfile?

Dockerfiles are imperative and the order of steps changes the container image that ultimately gets built.

Nix packages are declarative and idempotent. Everything is based on a functional package manager and when you configure your container via nix you get the same thing every time and can easily mix and match other packages and dependencies.

A contrived example is you have a program that depends on multiple versions of python, because nix does not globally set dependencies rather links them locally you can have both with versions of python without any conflicts

Containers built from Dockerfiles are difficult to exactly reproduce too. A very common practice is to update the package manager in which depending on what the server returns will be different from one day to the next where as nix hash addresses and builds from source everything (you can use caches to speed this up) and you get the same thing every time.

Note: NixOS isn’t perfect in terms of being purely functional as many package builds use scripts in pearl,bash,etc but from my understanding these are usually source specific reproducibility issues with Building particular packages and generally speaking things works as expected.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#199

Post by rumanator » Tue, Jun 09, 2020, 5:32 AM UTC

brown9-2 wrote: post #70

theptip wrote: post #63

Earlier quoted context omitted.

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups. It’s great if their own solution works well for them at less cost, but that system didn’t built itself and has non-zero maintenance costs.

> Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups.

In that case I would expect Coinbase to write blog posts on how their setup is the absolute best solution to their problem, and not how they refrain from adopting the best solution to their problem because they claim they don't have anyone on the team that is able to pull that off.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#200

Post by rumanator » Tue, Jun 09, 2020, 5:49 AM UTC

tso wrote: post #117

theptip wrote: post #63

Earlier quoted context omitted.

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

> Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The key is that there is a lot of additional services and interface points to handle. As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that ne…

> As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding.

The problem with that assertion is that it does not make any sense at all. For instance, storage and config/secrets is already supported out-of-the-box with Kubernetes. Even so, complaining about storage with Kubernetes is like complaining about EBS or EFs or arguably S3 in AWS. And if you feel strongly about service meshes then you really aren't forced to use them.

> Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.

There really isn't. For example, if they go with managed Kubernetes solutions then the only thing they need to worry about is to actually design their deployments, which would be very strange if they couldn't pull off. That's a ramp-up project for an intern if the solution architecture is already in place.

> You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them

There really aren't. Kubernetes' equivalent to EBS is either setting up a volume or a persistent volume claim on a persistent volume. Just state how much memory you want and you're set.