Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

321–330 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#321

Post by cameronh90 »

rawoke083600 wrote: post #221
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

if you can run everything on a $10 vm...do you really need k8 ?

I use k3s to manage my Plex/NAS/home server machine. Just simply as a deployment tool, it's much easier for me than managing a collection of services deployed by Ansible/Puppet held together by custom scripts and systemd units.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#322

Post by bvm »

brianwawok wrote: post #266
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

Counterpoint: Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.

I agree with you completely, we use Kops and see a similar workload. The real boon for us is not in production, where the HA/error-tolerant/easy horizontal scaling certainly helps, but in development, where we can easily bring up ephemeral feature branch deploys as part of the CI/CD pipeline (which itself runs on k8s using Gitlab CI)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#323

Post by harpratap »

StreamBright wrote: post #310
Lx1oG-AWb6h_ZG0 wrote: post #142

Earlier quoted context omitted.

But this makes no sense. Why would a cloud operator even bother with k8s if the customer is only interacting with functions? It’s much more efficient to bypass Kubernetes and run directly on the cloud’s native system (like borg). I think you’re right though - Kubernetes is a massive red herring, we should ideally be running containers/functions on as close to bare metal as possible. Fundamentally, VMs are the wrong a…

>> I really don’t know why AWS/Azure/GCP haven’t cottened onto this Conflict of interest. If k8s yields to the most revenue why would they try to decrease that? If some customers are so delusional that they go for an inefficient abstraction so be it. Btw. this is my experience with k8s too, people use it because it is a trend. Not a single company / developer could justify using it to me over leaner resources like EC…

How about companies who don't want to rely on cloud and want on-prem as much as possible? Or those who don't want to be tied to a single vendor? Apple is already getting rid of their Mesos based PaaS and moving to Kubernetes.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#324

Post by hardwaresofton »

15155 wrote: post #294
hardwaresofton wrote: post #275

Earlier quoted context omitted.

So if you'll indulge me -- this list is exactly why a system like Kubernetes is valuable and why I think personally that it contains a lot of essential complexity. Kubernetes attempts to do all of the above, which is why it's so massive, and I'd argue it's actually less complex than knowing all the tools above -- but it's an equal degree less universally applicable. In this way, it's perfect for the dev who never wan…

Devops person here: I already know all of these answers and would still choose Kubernetes over hand-rolling all of this again and again. "Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!" No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.

> CF Templates

Yeah, this is where it kicked in for me. Never mind the fact that all of that is AWS specific and absolutely doesn't help you if you ever move clouds. Great to know all the stuff below, but Kubernetes is a wonderful abstraction layer above that stuff, and it gets better every day.

CF could have become Kubernetes -- it was supposed to be, but it just never got the mixture right (and of course is AWS exclusive).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#325

Post by donedealomg »

brianwawok wrote: post #266
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

Counterpoint: Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.

You could do the same with AWS ECS

there is nothing special about K8s in your situation.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#326

Post by namelosw »

p_l wrote: post #279
namelosw wrote: post #230

Kubernetes is great but most projects don't really need it. Better go with Heroku equivalents than feeding a whole team, before you really need a whole team to feed Kubernetes.

Great if the pricing works out for you. Unfortunately most cases if I used Heroku I'd drive the cost way too high, as Heroku (or cloud) pricing doesn't change with local developer costs.

That's why I said 'equivalents'.

There are also application engines in major cloud providers, base on your technology. Some tech stack can run serverless. For self-host scenarios, things like Dokku could also work.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#327

Post by manigandham »

spottybanana wrote: post #284
manigandham wrote: post #248

Earlier quoted context omitted.

The "adhoc" part is the problem. K8S is standardized and offers high-availability, failover, logging, monitoring, load balancing, networking, service discovery, deployments, stateful services, storage volumes, batch jobs, etc. And it can run self-contained on a single machine or scale out to 1000 nodes. Why piece all of that functionality together yourself into some fragile framework instead of using a industry stand…

"Why piece all of that functionality together yourself into some fragile framework instead of using a industry standard?" Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. I personally would use the word "industry standard" a little bit differently.

You still have to put them all together into some custom solution just for your setup which adds overhead and fragility. New employees will have to learn that instead of using K8S APIs. Deploying new components can’t take advantage of the wide and fast growing ecosystem.

There’s really nothing like the full suite that kubernetes provides.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#328

Post by p_l »

namelosw wrote: post #326
p_l wrote: post #279

Earlier quoted context omitted.

Great if the pricing works out for you. Unfortunately most cases if I used Heroku I'd drive the cost way too high, as Heroku (or cloud) pricing doesn't change with local developer costs.

That's why I said 'equivalents'. There are also application engines in major cloud providers, base on your technology. Some tech stack can run serverless. For self-host scenarios, things like Dokku could also work.

It's not a matter of which product you use, more that "setup on unmanaged" often infers huge enough savings that you can pay for more engineers.

Cause in many places the cost of fully loaded experienced engineer might be cheaper than managed services.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#329

Post by p_l »

dijit wrote: post #299
p_l wrote: post #273

Earlier quoted context omitted.

Spawning a bare bones VM is easy and fast, the problem is getting the applications there, which is where k8s shines.

Maybe I'm spoiled then, it takes less than 2s to spin up my application on a new instance. :\ There are also many options for creating Debian based image files with all you need baked in. I guess it depends very much on the application you have.

A lot depends on whether "bake application into AMI, use ASG to scale and zero-downtime deploy" is applicable or cost effective.

I think I've been in total of one company where it was somewhat true, and even there we didn't do it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#330

Post by p_l »

bschwindHN wrote: post #313
p_l wrote: post #258

Earlier quoted context omitted.

Building it yourself is also a case of "resume-driven development". Not that there aren't good reasons or good outcomes from having done a deep dive, but just putting it there.

I think using something like k8s prematurely is more an example of "resume driven development" than the other way around. Building it yourself doesn't mean building all of it. For example, it's quite easy to get zero downtime deployments with a tiny bit of systemd configuration and the SO_REUSEPORT socket option. That seems easier for a team to understand than "here is kubernetes and everything that comes with it"

That is, however, a very tiny slice of what might be needed.

You need to deliver the application to the machines somehow. You might need to configure the network to reach the application, etc. All quite common tasks.

(And honestly, systemd can bite you just as much if not worse than k8s definitions, because the API is much less cohesive and the defaults like to cut off your hands)

Post reply on HN