Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

161–170 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#161
post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

Could you share the resources to get started? I'm in a similar boat, 2 person trying to set up K8S in DO using automation (CI/CD).

Gitlab CI is my weapon of choice here since it’s integrated nicely with Kubernetes. There’s a wealth of tools out there - but the work I do on managing PCGamingWiki is publicly available [1] to give you a starting point. I use Kustomize + kubectl, and when I need to rollback a deployment I can just do it from Gitlab’s environment page.

1: https://gitlab.com/pcgamingwiki/pcgamingwiki

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#162
post #53

Earlier quoted context omitted.

TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so. I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to…

> managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.

Isn't this the same for every software? How is debugging issues with Linux, NGINX, any complex framework any easier?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#163

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith...

> How do you run an application on a cluster of plain old linux machines?

Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers).

> How do you do load balancing?

An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes).

> How do you update your app without downtime?

Blue-green deployment, or phased rollout.

> How do you ensure all your servers are running the same version of dependencies?

Build them from a common image.

> How do you update those dependencies?

Update the Packer template that builds that image.

> How do you replicate your environment if you want to add a new server to your cluster?

Start the server from the same image.

> If your app has microservices how do services discover each other?

Consul, or DNS, depending on your appetite.

> How do you mount volumes from cloud storage?

It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS.

> How do you update configuration?

Either update Consul and have it propagate configuration, or update a configuration package and push it out.

> How do you automatically restart failed applications?

Systemd restart policy.

> How do you monitor if your applications are working?

From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user.

> How do you make sure the right number of MongoDB replicas are running at all times?

Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group.

> How do you view your log files remotely?

Cloudwatch, Syslog, SSH (depending on requirements).

> How do you port-forward from localhost to your Linux server to test your app locally?

SSH.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#164
post #56

Anecdata: Series B startup. I've found GKE to be almost completely painless, and I've been using it in production for more than 4 years now. I don't think the article gave a fair representation on this count; sharing a link to a single GKE incident that (according to the writeup) spanned multiple days and only affected a small segment of users doesn't (for me) substantiate the claim that "it isn’t uncommon for them t…

EKS is a feature parity product. The pricing makes that painfully obvious.

AWS wants k8s to fail because it works against the significant lock-in AWS has tricked teams into building themselves into. They do not want people already in the AWS ecosystem to move to EKS, it is instead there to not lose potential customers.

And that is why pricing and features sit right at “good enough” and not great.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#165
Many of the author's issues with kubernetes are specific comparisons with their current workflow. One I can definitely agree with is the burden of upgrades and keeping the platform current. There are techniques which can make that easier to handle. One statement that surprised me:

>> at Google it isn’t uncommon for them to have multi-hour outages with GKE

For what it's worth we've been running multiple GKE clusters in production for over three years. We're medium size, with some dozen or so in-house services handling perhaps a total of 20k rps. We are rarely affected by any GCP issue, and as far as I can recall we have never been down due to a GKE-specific problem. In addition to the basic orchestration features we make significant use of ingress and storage primitives. It all quite literally just works.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#166
post #87

imo running k8s in the cloud makes no sense. If you've already setup on prem servers, load balancers, config management, patching, access control, etc to allow developers to run applications on VMs k8s can provide an integrated experience with significantly less work. If you're running k8s in the cloud then just use a hosted service and leverage enterprise support. In the on prem case, you already have a dedicated in…

> In the on prem case, you already have a dedicated infra team that probably has the tools to effectively deploy and manage a cluster.

Tools? Yes. Care and attention? Hell no.

The ops team at my work still has important services running on CentOS 6, and they are spending all their time trying to get Kubernetes configured and working.

You would be surprised at how much infrastructure out there is just glued together adhoc. Several senior engineers at my work are successfully blocking adoption of gitops and CD.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#167
post #32
post #29

Earlier quoted context omitted.

I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful). How often/how many cron jobs are you running that spawning a new pod per job is a problem ?

The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.

50k/day is less than 1 qps. This is nothing. This is either not the full story or your cluster was setup completely wrong

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#168

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture.

I'd have probably gotten used to it had I spent more time using it, but it'd have taken some rewiring of thinking process.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#169
post #156

1) Those guys can't imagine their life outside of amazon. this is bad. And yes, amazon k8s was bad, it's getting better, still bad. 2) They said we need a separate team and instead they wrote and maintain their own, sort of lightweight solution. Nothing more about that article. One opinionated team decided to roll out something homemade to add it to CV later on.

Something that won't even help their resumes in the long run.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#170
post #92

I partially blame keyword driven recruitment in tech for these kinds of responses to a platform/tooling. Kubernetes isn't a magic bullet - it is a platform which solves a very specific set of problems with scaling. And of course it doesn't come for free. You can't just throw in k8s into your existing infrastructure and expect your devs to manage it, in addition to their regular work. And yet, we keep reading about te…

Oh God. This is what is happening at my work. We have an API that has 200 write users and a public front-end that can do reads. None of it is heavy though, with most writes occurring for a month in the Winter and a month in the Fall. In the unlikely event of heavy write loads we could just scale up CPU/RAM for those two months. Any read load be solved with cache or spending time on the worst offenders in SQL. The Lea…

So are you assuming that your service will never grow? Because vertical scaling up is nice and easy as long as you know the limits but once you cross a threshold, not matter how many CPUs you throw at a problem it just won't scale. Your senior lead seems to be anticipating that and preparing in advance.
Post reply on HN