Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

351–360 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#351

Post by thinkharderdev »

DevKoala wrote: post #154

I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics. Introducing Kubernetes into our o…

Exactly. A lot of the comments here are along the lines of "we setup a k8s cluster and now managing it is a huge burden" which is not surprising. The power of k8s is to allow separation of concerns in your technology organization. You can have a dedicated team to build and maintain the underlying cluster and then app development teams are consumers who deploy their application on the common infrastructure. Kubernetes provides a nice abstraction layer so those two teams/orgs can interact through a well-defined API. As a dev team, we can manage our own infrastructure and pipeline through declarative configurations and let someone else manage the underlying compute and network infrastructure. As long as you don't fall into the anti-pattern of "every team builds it's own k8s cluster" then you should be able to derive some nice economies of scale.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#352

Post by sciurus »

jrockway wrote: post #308
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

My experience is the same. I really like automating tedious and error prone parts of deployments, and Kubernetes is the best tool I've found for that. It is a lot to learn about, and there are a lot of missing features that people go to great lengths to build for themselves (see "service mesh" for example), but the core is very solid. I like loosely coupling things, and Kubernetes is the first ecosystem where that ha…

> What people really want is not Managed Kubernetes but "namespace as a service". They just want to kubectl apply something and let a background task provision their machines. They don't want to screw around with RBAC, service meshes, managing the Linux distribution on their worker nodes, managing the master nodes, etc. That service unfortunately doesn't exist.

I think you just described AWS Fargate and Google Cloud Run.

https://aws.amazon.com/fargate/

https://cloud.google.com/run

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#353

Post by kryptk »

pg-gadfly wrote: post #302
jayd16 wrote: post #232

Earlier quoted context omitted.

What other solution is cloud agnostic as well as has an easy local dev story?

_any_ non-proprietary tool is "cloud agnostic". Kubernetes is bundled software, and achieves the things those pieces achieve. There is nothing holy about k8s specifically, the tools you train with are easy, and it's very easy to get skewed opinions on that. For example a lot of people would find writing scripts cumbersome, but not a person who's written a lot of them. They're not any more fragile than other logic err…

You didnt avtually answer the question: What free, cloud agnostic tools let me specify "keep 60% cpu load average" and work the rest out?

With k8s, a horizontal autoscaler is a few lines in a yaml file and the result works in any cluster run by any vendor.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#354

Post by sciurus »

jen20 wrote: post #163
flowerlad wrote: post #100

Earlier quoted context omitted.

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do yo…

Great answers.

To be fair though, that's not "Plain old linux" like zaptheimpaler suggested was somehow possible. That's linux plus AWS managed services plus software from Hashicorp. Which is a great stack to be on, but has its own complexities and tradeoffs.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#355

Post by jen20 »

hardwaresofton wrote: post #324
15155 wrote: post #294

Earlier quoted context omitted.

Devops person here: I already know all of these answers and would still choose Kubernetes over hand-rolling all of this again and again. "Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!" No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.

> CF Templates Yeah, this is where it kicked in for me. Never mind the fact that all of that is AWS specific and absolutely doesn't help you if you ever move clouds. Great to know all the stuff below, but Kubernetes is a wonderful abstraction layer above that stuff, and it gets better every day. CF could have become Kubernetes -- it was supposed to be, but it just never got the mixture right (and of course is AWS exc…

I find the ridiculous false dichotomy between Terraform for Kubernetes and Cloudformation for more basic infrastructure even more ironic given that I am still the eighth most prolific contributor to Terraform _over three years after leaving HashiCorp_.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#356

Post by jen20 »

Longwelwind wrote: post #303
jen20 wrote: post #163

Earlier quoted context omitted.

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do yo…

I feel like your post describes exactly what Kubernetes and container images would bring to your infra. If you were to deploy a solution like you described, you would get something more complex than simply running Kubernetes, except worse. I suspect that you believe your solution would be simpler only because you are more comfortable with those technologies than with k8s. The more I read criticism of k8s, the more I'…

> The advantage of Docker images over AMI is that you have a file that describes how the image was built.

https://packer.io - how is this even a discussion?

> the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with".

My infrastructure runs in Nomad on bare metal. I am by no means opposed to “progress”, I just don’t think Kubernetes is the be-all-and-end-all of infrastructure and would like to have a less hysterical debate about it than the parent to my original post presented.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#357

Post by jrockway »

sciurus wrote: post #352
jrockway wrote: post #308

Earlier quoted context omitted.

My experience is the same. I really like automating tedious and error prone parts of deployments, and Kubernetes is the best tool I've found for that. It is a lot to learn about, and there are a lot of missing features that people go to great lengths to build for themselves (see "service mesh" for example), but the core is very solid. I like loosely coupling things, and Kubernetes is the first ecosystem where that ha…

> What people really want is not Managed Kubernetes but "namespace as a service". They just want to kubectl apply something and let a background task provision their machines. They don't want to screw around with RBAC, service meshes, managing the Linux distribution on their worker nodes, managing the master nodes, etc. That service unfortunately doesn't exist. I think you just described AWS Fargate and Google Cloud…

AWS's version is pretty half-baked. You can't provision or use persistent volumes (so no stateful apps), and you have to use their load balancer which terminates TLS (preventing your software from being able to do ALPN, using Let's Encrypt, supporting HTTP/3, etc.).

Cloud Run just seems like standard "serverless" stuff, nothing to do with Kubernetes. (The downsides involve not being able to run applications that are designed to run on a generic Linux box; everything has to be specially developed. That is fine, and they have open-sourced all the tools necessary to move off of them so you aren't locked in, but it's a bigger paradigm shift.)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#358

Post by cdelsolar »

Juliate wrote: post #244
merb wrote: post #239

Earlier quoted context omitted.

most people probably want the following: - no downtime deployments - distributed jobs - as managed infra as possible without k8s some things would be hard.

no downtime deployments? happened before k8s; used to do that several times with some haproxy. distributed jobs? same; nothing prevents from spawning runners with adhoc libraries and queues. managed infra? not specific to k8s

"some haproxy" - what is the haproxy configuration? I tried doing this myself and then realized setting up a cheap k8s cluster on DO was way easier. Now, yes, there have been occasional problems, but not since I just let DO handle the whole thing. I can do zero-downtime deploys, cronjobs (why is spawning a pod so wasteful? it spawns it, runs the job, and then kills it?), all with a single "kubectl apply" command that takes a split second to run.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#359

Post by Juliate »

manigandham wrote: post #248
Juliate wrote: post #244

Earlier quoted context omitted.

no downtime deployments? happened before k8s; used to do that several times with some haproxy. distributed jobs? same; nothing prevents from spawning runners with adhoc libraries and queues. managed infra? not specific to k8s

The "adhoc" part is the problem. K8S is standardized and offers high-availability, failover, logging, monitoring, load balancing, networking, service discovery, deployments, stateful services, storage volumes, batch jobs, etc. And it can run self-contained on a single machine or scale out to 1000 nodes. Why piece all of that functionality together yourself into some fragile framework instead of using a industry stand…

k8s is nice, yes. But, by its age, no more standardized now, than MySQL was 15 years ago.

k8s also adresses a very small portion of the market. If you have to scale, yes, you might need k8s. Chances are, you don't. Really.

I stopped counting the (supposedly big) customers that burnt themselves into k8s when they logistically need not to, and only brought organisational issues on them.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#360

Post by mauflows »

jrockway wrote: post #357
sciurus wrote: post #352

Earlier quoted context omitted.

> What people really want is not Managed Kubernetes but "namespace as a service". They just want to kubectl apply something and let a background task provision their machines. They don't want to screw around with RBAC, service meshes, managing the Linux distribution on their worker nodes, managing the master nodes, etc. That service unfortunately doesn't exist. I think you just described AWS Fargate and Google Cloud…

AWS's version is pretty half-baked. You can't provision or use persistent volumes (so no stateful apps), and you have to use their load balancer which terminates TLS (preventing your software from being able to do ALPN, using Let's Encrypt, supporting HTTP/3, etc.). Cloud Run just seems like standard "serverless" stuff, nothing to do with Kubernetes. (The downsides involve not being able to run applications that are…

Cloud Run uses KNative which is a stateless extension of k8s
Post reply on HN