Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

261–270 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#261

Post by patricjansson »

silvester23 wrote: post #229
santamarias wrote: post #218

Earlier quoted context omitted.

> It's a pity that docker swarm did not make it. Sorry I do not understand that statement, in my naive opinion Docker Swarm seems to be a thing. Care to elaborate, please?

It is, but at this point it is unclear for how long Docker Swarm will be supported, see e.g. https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kuber... We are actually currently in the process of migrating from Docker Swarm to k8s and I am not 100% sure that's a good idea. We will see.

"conversations have led us to the conclusion that our customers want continued support of Swarm without an implied end date."

https://www.mirantis.com/blog/mirantis-will-continue-to-supp...

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#262

Post by mvanbaak »

jen20 wrote: post #163
flowerlad wrote: post #100

Earlier quoted context omitted.

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do yo…

> Somewhat flippant answer here: the right number of MongoDB servers is zero.

AMEN to this one.

Very nice answer. It's the way we run infra.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#263

Post by jmchuster »

felipelemos wrote: post #7

What I don't like in these kind of posts is the fact that he seems to know much more than he could. "For example, most folks that run large-scale container orchestration platforms cannot utilize their built-in secret or configuration management." So, with thousands (maybe more?) of "folks" out there, each with a different environment, requirement, background and so on. How can someone knows what most of them can or c…

Probably by "most" he just means he's talked to 20 friends who head up devops for such large-scale platforms, and 19 out of 20 explained how it didn't work for them, and he then assumed that's generally applicable for the other couple thousand out there.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#264

Post by anang »

rumanator wrote: post #195
ditonal wrote: post #118

Earlier quoted context omitted.

The difficulty of all that stuff on plain old Linux is overstated and the difficulty of doing that on K8s is understated. And I agree with grandfathers point that if you don’t understand how to do that on plain old Linux you will struggle on K8s. K8s is ok for huge enterprises with tons of engineers but somehow K8s advocates make it seem like learning and operating nginx on ubuntu is this huge challenge when it’s usu…

Your comment makes no sense at all. Kubernetes is not equivalent to running nginx on Ubuntu. This comparison is simply too absurd to the point I really doubt you have any idea of what Kubernetes is designed to do and why people use it. I mean, who in their right mind would even describe a cluster of VMs and/or COTS hardware running with a VPN and designed to automatically run and manage the lifecycle of containers as…

I think the point is more that the complexity of a cluster of VMs that manage the lifecycle of containers is often overkill for a service that would work with nginx installed on Ubuntu, and that often times the former is sold as reducing complexity and the latter as increasing it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#265

Post by p_l »

yjftsjthsd-h wrote: post #123
flowerlad wrote: post #100

Earlier quoted context omitted.

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

DNS round-robin sends inbound to 2 haproxy servers, which proxy traffic on to the rest of the cluster. Scaling means "add another box, install your app, add haproxy entry". Service discovery is just an internal-facing haproxy front. If you must have "volumes from cloud storage", you use NFS (but if you can help it, you don't do that). Updates, restarts (including zero-downtime courtesy of draining the node out of hap…

Having done exactly that in the past, both by hand and with configuration management (including custom scripting that synchronized HAproxy configs etc.), I would say that I can do all of that much, much simpler in k8s.

Installing application and routing it with load balancing, TLS etc and support for draining is really simplified, something that used to require annoying amount of time (it got better as Lua scripting was extended in HAproxy, but by that time I also had k8s).

Resource allocation, including persistent storage (including NFS) is so much easier it's not funny, it makes my old years painful to think about. Syslog is not novel but getting all applications to log the same way was always a PITA, and at least with containers it's a bit easier (I still have to sometimes ship physical logs from containers...).

As for monitoring, that's one of the newer and more advanced topics, but it's possible to integrate more complex server monitoring with k8s - it already provides a lot of OS/application side monitoring that really makes it easier to setup observability - but now there's quite simple way to integrate custom node states into kubernetes allowing a quick glance way to check why a node is not accepting jobs, integrated with the system so you can actually trigger from a health-check that the node is in trouble and should not take jobs.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#266

Post by brianwawok »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

Counterpoint:

Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#267

Post by temptemptemp111 »

mmgutz wrote: post #64
flowerlad wrote: post #42

Earlier quoted context omitted.

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well. An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.

Agreed. Good support for compose currently, and swarm is usually overkill. With a bit more elegant HA functionality, docker-compose could be the go-to for many more. The comment below claiming he doesn't understand your comment reminds me of all of those who will say "well its not FOR production" - seems more like superstition than science.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#268

Post by eddieoz »

The discussion and pressure always start on the engineers' side, but normally they don't consider it is different to build a system from scratch using the technology or migrating.

The article is clear: Coinbase has a very strong, tested and validated infrastructure and moving to Kubernetes 'just to be part of the hype' does not bring any benefits at this point. And a statement like that is a nightmare to some DevOps.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#269

Post by tomerbd »

jen20 wrote: post #163
flowerlad wrote: post #100

Earlier quoted context omitted.

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do yo…

I wish I knew to do all this, is there any guide best practice that I can read like a book that includes all above?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#270

Post by p_l »

treeman79 wrote: post #149
joana035 wrote: post #82

Earlier quoted context omitted.

For me k8s kinda extends what you can do with ansible. For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere. I mean, you terminate one computer, k8s will figure out that the pr…

I go in and out of development and ops and devops. 20 years. To me as a developer heroku is the gold standard. Docket-compose makes sense. Kubernes and helm is a giant soul crushing wtf On getting anything done. Now I’ll get it figure it out. Angular + exhaust+ typescript was a similar experience after I took a break from front end for a few years. A few months of pain, then it starts clicking. but this just seems in…

Many tutorials are unfortunately bad, they actually got worse over time.

Oooold presentations (think 2016 and older) tended to talk more about basic building blocks and how they interacted, especially the design involving resources and their controllers working in a loop of "check requested state, check real state, do changes to implement requested state", and how those loops went up from single pods, through ReplicationControllers (now ReplicaSets), then Deployments, etc.

Post reply on HN