Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

141–150 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#141

Post by arcticfox »

dvcrn wrote: post #131
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

I'm also on a 2-person project on DigitalOcean k8s, also very happy.

K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said.

Also, we run a VPC-only K3s node for some simple internal tools that works great as well.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#142

Post by Lx1oG-AWb6h_ZG0 »

harpratap wrote: post #138
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

I have said this multiple times in the past ( https://news.ycombinator.com/item?id=23361176 https://news.ycombinator.com/item?id=23243626 ) and will say it again - You (business logic developers) are not meant to use K8s directly, you are supposed to use a PaaS built on top of K8s (like Cloud Functions, Lambda etc)

But this makes no sense. Why would a cloud operator even bother with k8s if the customer is only interacting with functions? It’s much more efficient to bypass Kubernetes and run directly on the cloud’s native system (like borg).

I think you’re right though - Kubernetes is a massive red herring, we should ideally be running containers/functions on as close to bare metal as possible. Fundamentally, VMs are the wrong abstraction if all your code is containerized.

Joyent’s triton is the closest thing we have to this... I really don’t know why AWS/Azure/GCP haven’t cottened onto this, it would massively reduce their COGS and improve our developer experience.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#143

Post by EdwardDiego »

modeless wrote: post #39
ur-whale wrote: post #16

Given Coinbase's reliability track record [1][2][3], I'm not entirely sure why I'd listen to devops "wisdom" coming from their corner. [1] https://www.reddit.com/r/CoinBase/comments/gh3b5t/coinbase_c... [2] https://www.reddit.com/r/Bitcoin/comments/6gtwyi/every_singl... [3] https://www.reddit.com/r/CryptoCurrency/comments/ggr80l/i_do...

While this may not be stated in the most diplomatic way, it's important and relevant for this discussion and I don't think it should be downvoted. It's very well established that Coinbase's infrastructure is unable to handle traffic spikes, despite the fact that traffic spikes are a fact of life in their industry and downtime can easily cost their customers millions of dollars.

Huh, have they considered Kubernet... nevermind.

In seriousness, we find k8s makes handling our traffic spikes a doddle. As in, we don't have to do anything.

Of course, we didn't necessarily need K8s to accomplish that, but it's the tool we used, and the complexity cost is paying off for us.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#144

Post by dvcrn »

arcticfox wrote: post #141
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

I'm also on a 2-person project on DigitalOcean k8s, also very happy. K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said. Also, we run a VPC-only K3s node for some simple internal tools that works great as well.

> Also, we run a VPC-only K3s node for some simple internal tools that works great as well.

We do exactly the same thing! We have a one-node k8s for all these dev things that just works. Everything is containerized for local dev anyway so moving it to k8s was just writing the deployment manifest.

On heroku, all of these would be separate dynos (or one glued-together dyno that does everything). On a self-hosted VM we'd have to deal with managing that.

I liked this approach so much that I now have a small 1-node personal cluster that hosts all of my private hobby projects that aren't ready for prime time yet, that were on heroku previously. Costs me only $10 + (persistent storage + IP address if needed)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#145

Post by ashtonkem »

arcticfox wrote: post #141
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

I'm also on a 2-person project on DigitalOcean k8s, also very happy. K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said. Also, we run a VPC-only K3s node for some simple internal tools that works great as well.

Heroku is also very cheap on the low end. EKS costs at least $72/mo before adding compute & storage. That would get you a lot of dynos.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#146

Post by silviogutierrez »

rudolph9 wrote: post #68

One thing people forget is containers don’t necessarily not to be created with Docker. Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we h…

Huge nix fan. Question: how do you deploy these with k8s? Do you just build a NixOS docker image "wrapper" around all the other stuff?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#147

Post by rhizome »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

Your comment and the post and threads in general are making me chuckle, having just spent the last half hour reading this post, for no particular reason, from a few months ago about the transformation of racknut-and-perl roles like neteng and sysadmin into text editor-based devops:

https://news.ycombinator.com/item?id=22508968

I have to say, certain comments -- which I'm sure are just as real-world as yours -- lie in...let's call it tension...with the threads and comments here in this post. That everything in the old post appears to be subsumed into a "sysadmin II: internet boogaloo" of orchestrator expertise is humorously ironic.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#148

Post by harpratap »

Lx1oG-AWb6h_ZG0 wrote: post #142
harpratap wrote: post #138

Earlier quoted context omitted.

I have said this multiple times in the past ( https://news.ycombinator.com/item?id=23361176 https://news.ycombinator.com/item?id=23243626 ) and will say it again - You (business logic developers) are not meant to use K8s directly, you are supposed to use a PaaS built on top of K8s (like Cloud Functions, Lambda etc)

But this makes no sense. Why would a cloud operator even bother with k8s if the customer is only interacting with functions? It’s much more efficient to bypass Kubernetes and run directly on the cloud’s native system (like borg). I think you’re right though - Kubernetes is a massive red herring, we should ideally be running containers/functions on as close to bare metal as possible. Fundamentally, VMs are the wrong a…

> Why would a cloud operator even bother with k8s if the customer is only interacting with functions?

Because there are various "tiers" of users, some companies (like coinbase) could actually leverage K8s in their Codeflow/Odin project and prevent a lock-in. But a regular developer looking to just "get things shipped" isn't meant to waste his/her time with pure K8s.

> Kubernetes is a massive red herring

We agree but on a different note. The biggest selling point of K8s is it's API design. The entire industry needs to converge on one "defacto" standard of packing and deployment. Google's Cloud function is a perfect example of this. The API is based on K8s and Knative but under the hood it actually runs directly on GCE rather than GKE. What happens underneath is hidden from business developer, you only care about the data in your yaml and your docker image.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#149

Post by treeman79 »

joana035 wrote: post #82
hwbehrens wrote: post #76

Earlier quoted context omitted.

Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.

For me k8s kinda extends what you can do with ansible. For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere. I mean, you terminate one computer, k8s will figure out that the pr…

I go in and out of development and ops and devops. 20 years.

To me as a developer heroku is the gold standard.

Docket-compose makes sense.

Kubernes and helm is a giant soul crushing wtf On getting anything done.

Now I’ll get it figure it out. Angular + exhaust+ typescript was a similar experience after I took a break from front end for a few years. A few months of pain, then it starts clicking.

but this just seems insane for a basic web app. So many different tools needed to get it going. Tutorials all have many steps that don’t apply, or use other odd pieces swapped in.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#150

Post by snuxoll »

httpz wrote: post #121
chillfox wrote: post #115

Earlier quoted context omitted.

Heroku is missing the ability to get a dedicated IP for a project.

You just have to pay more to get private spaces on Heroku.

Pay more is the Salesforce motto.
Post reply on HN