Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

201–210 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#201

Post by Lukas_Skywalker »

arcticfox wrote: post #141
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

I'm also on a 2-person project on DigitalOcean k8s, also very happy. K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said. Also, we run a VPC-only K3s node for some simple internal tools that works great as well.

For those missing Heroku: there exists Dokku [1], a small Heroku-like implementation for container management. It uses the same underlying buildpacks and you get the same comfort as with Heroku. And it's free to use. You can't deploy to multiple host machines though. But for small projects that fit on a single host, it's very nice to use.

[1] http://dokku.viewdocs.io/dokku/

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#202

Post by rodgerd »

manigandham wrote: post #99
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

You can't make the complexity disappear. Kubernetes just offers a very standardized, stable and relatively polished way to handle it. The other alternatives, like running your own orchestration system, are just as complex if not more so, although you might be more familiar with it since you built it. That being said, many companies don't need any of it in the first place for their scale, and that's probably the bigge…

> That being said, many companies don't need any of it in the first place for their scale, and that's probably the biggest issue with K8S today.

Yes. I helped push out a k8s platform where I work, and it's been running well in production for ~3 years.

We most definitely did not start with "we want to deploy k8s". We started with "we're being asked to meet certain business requirements", which lead to "we will need to change how we do some of our development and deployment", which lead to "our platform will require these characteristics."

The easiest way to get those turned out to be buying a packaged k8s from a vendor (OpenShift, not that it matters; there's plenty of options).

Most of the pain with k8s seems to come down to people wanting to polish their CVs (by "having done k8s"), or people who sneer at packaged/hosted solutions because they want to build a cottage industry of building k8s services from scratch because it's their idea of a good time.

Unsurprisingly both take orgs down the route of pain, money, and regrets. Oftentimes the people driving this decision then prefer to say the problems is k8s, or containers. Or microservices or whatever fad it was they were chasing without understanding whether it would be a good idea.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#203

Post by redis_mlc »

rumanator wrote: post #195
ditonal wrote: post #118

Earlier quoted context omitted.

The difficulty of all that stuff on plain old Linux is overstated and the difficulty of doing that on K8s is understated. And I agree with grandfathers point that if you don’t understand how to do that on plain old Linux you will struggle on K8s. K8s is ok for huge enterprises with tons of engineers but somehow K8s advocates make it seem like learning and operating nginx on ubuntu is this huge challenge when it’s usu…

Your comment makes no sense at all. Kubernetes is not equivalent to running nginx on Ubuntu. This comparison is simply too absurd to the point I really doubt you have any idea of what Kubernetes is designed to do and why people use it. I mean, who in their right mind would even describe a cluster of VMs and/or COTS hardware running with a VPN and designed to automatically run and manage the lifecycle of containers as…

rumanator: the mistake is yours. Please read the parent post again, since you missed any nuance.

I agree with said parent. You can go a long way with ASGs, iptables and yum install - literally the entire first decade of your startup.

In larger companies, just the politics introduced with managing k8s by another dept. is mind-numbing.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#204

Post by lenkite »

jen20 wrote: post #163
flowerlad wrote: post #100

Earlier quoted context omitted.

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do yo…

I loved this list of succinct answers.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#205

Post by rumanator »

hwbehrens wrote: post #76
joana035 wrote: post #62

Earlier quoted context omitted.

K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc. For people who already know Linux, kubernetes comes naturally because it is pretty obvious. But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.

Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.

> Couldn't you get nearly the same behavior using some basic Ansible playbooks?

No. For starters, Ansible playbooks don't allow you to dynamically deploy and redeploy a set of containers in the node that at a given moment happens to have more available computational resources such as bandwidth, nor do teu respawn a container that just happened to crash.

Arguably Ansible only implementos a single feature of Kubernetes: deploy an application. Even so, while with Ansible you need to know and check all details about the underlying platform, with Kubernetes in essence you only need to say "please run x instantes of these containers".

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#206

Post by vegai_ »

Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails.

Kubernetes supports mostly Linux, although has recently gained Windows node support.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#207

Post by GOTERTHe »

Lukas_Skywalker wrote: post #201
arcticfox wrote: post #141

Earlier quoted context omitted.

I'm also on a 2-person project on DigitalOcean k8s, also very happy. K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said. Also, we run a VPC-only K3s node for some simple internal tools that works great as well.

For those missing Heroku: there exists Dokku [1], a small Heroku-like implementation for container management. It uses the same underlying buildpacks and you get the same comfort as with Heroku. And it's free to use. You can't deploy to multiple host machines though. But for small projects that fit on a single host, it's very nice to use. [1] http://dokku.viewdocs.io/dokku/

Tried Dokku, but found CapRover [1] to be a much better / easier option

[1] https://caprover.com/

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#208

Post by lenkite »

harpratap wrote: post #162
solidasparagus wrote: post #108

Earlier quoted context omitted.

> managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.

Isn't this the same for every software? How is debugging issues with Linux, NGINX, any complex framework any easier?

Debugging k8s issues is like debugging a vast distributed monolith with a vague guesses on where the problem occurred.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#209

Post by speedgoose »

Shicholas wrote: post #34

As someone whose gone the opposite way (moving from ECS to Kubernetes), I think the author is understating how good managed Kubernetes solutions are. At my current job, I use Azure's managed Kubernetes service, which does a great job at providing a consistent environment that's very easily managed, no unexpected updates, great dataviz, and if you choose, simple integrations to their data storage solutions (run statel…

You don't see random network errors, often visible with DNS, on your Azure managed kubernetes clusters?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#210

Post by jillesvangurp »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

It's a pity that docker swarm did not make it. It wasn't perfect but it was a lot simpler to setup and manage than kubernetes.

If you can get away with it, vanilla docker hosts running docker compose provide most of the same benefits with a fraction of the cost. For most startups, that's a great way to avoid getting sucked into a black hole of non value adding devops activity. You lose some flexibility but vanilla ubuntu hosts with vanilla docker installs are easy to setup and manage. We used packer and ansible to build amis for this a few years ago with some very minimal scripts for container deploys.

Post reply on HN