Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

371–380 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#371

Post by nednar »

dvcrn wrote: post #131
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

> I used normal blue/green deployments on self-managed VMs in the past, then worked with beanstalk, heroku, appengine and I much prefer k8s. Yes it's easier on heroku, but try to run 2-3 different containers on the same dyno for dev to keep cost down. On k8s I can run my entire stack on one single small digitalocean $10 VM if I wanted to.

So you already spend about a decade learning all the skills. What the other guy is talking about coming from dev not from ops. If you come from dev you don't necessarily know what an ingress or egress is, and might never have done a blue/green deployment etc. This is all stuff that needs to be learned first. I worked with many many teams who had zero skills in data center tech before they were moved to k8s full time.

I personally like it to learn all that stuff. And I love that my job requires it now. But it's more like vim than like node-red, and that was a shock for many people, from engineer to EVP.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#372

Post by rudolph9 »

jmchuster wrote: post #260
rudolph9 wrote: post #198

Earlier quoted context omitted.

Dockerfiles are imperative and the order of steps changes the container image that ultimately gets built. Nix packages are declarative and idempotent. Everything is based on a functional package manager and when you configure your container via nix you get the same thing every time and can easily mix and match other packages and dependencies. A contrived example is you have a program that depends on multiple versions…

Then have your Dockerfile build from source. Many of the public layers that people build their containers upon do that for that exact reason. If NixOS is only "generally speaking" correct because of the order it runs its scripts, then put it in a Docker container, where the whole point is to perfectly reproduce the order you run your scripts in order to minimize the errors that arise from doing otherwise. Still worri…

Ha! I did this but the containers end up being somewhat large since you have all the NixOS package stuff in the container too.

However, building the container from a configuration.nix file gives you exactly what you need in the container and nothing more. It's like basing your container from scratch in a Dockerfile and copying only the needed compiled dependencies and nothing more into the container.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#373

Post by bobbyi_settv »

cdelsolar wrote: post #358
Juliate wrote: post #244

Earlier quoted context omitted.

no downtime deployments? happened before k8s; used to do that several times with some haproxy. distributed jobs? same; nothing prevents from spawning runners with adhoc libraries and queues. managed infra? not specific to k8s

"some haproxy" - what is the haproxy configuration? I tried doing this myself and then realized setting up a cheap k8s cluster on DO was way easier. Now, yes, there have been occasional problems, but not since I just let DO handle the whole thing. I can do zero-downtime deploys, cronjobs (why is spawning a pod so wasteful? it spawns it, runs the job, and then kills it?), all with a single "kubectl apply" command that…

If you're in AWS, you can use their ELB (Elastic Load Balancer) service instead of setting up your own haproxy. I worked on a team that used it for years without any issues orchestrating zero-downtime deployments. It was extremely easy and didn't require any real configuration.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#374

Post by ex_amazon_sde »

flowerlad wrote: post #100
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

This is how Amazon works internally, and by choice, because unnecessary complexity is the enemy.

> How do you do load balancing?

With dedicated load balancers. They are even cheaper than servers because they are optimized for doing one thing.

> How do you scale up and down?

With pretty trivial automation than trigger deployments when load is too high.

> How do you update your app without downtime?

(app?!) A package manager pulls updated packages, deploys them and restarts the services. The LB does the flipping.

> How do you roll back easily if something goes wrong?

The package manager does that quite easily

> How do you ensure all your servers are running the same version of dependencies? > How do you update those dependencies?

Package managers support versioning on dependencies.

> How do you replicate your environment if you want to add a new server to your cluster?

You install MyStuff version 1.2.3 and everything goes in.

> If your app has microservices how do services discover each other?

Thankfully Amazon uses services that are not micro. DNS does the trick.

> How do you update configuration?

It's in the packages.

> How do you automatically restart failed applications?

OSes do that since decades.

> How do you monitor if your applications are working?

LBs check the endpoints, monitoring tools do the rest.

Essentially you can build all of this with the technology that existed 15 years ago. It works reliably and it takes much less learning.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#375

Post by ex_amazon_sde »

15155 wrote: post #294
hardwaresofton wrote: post #275

Earlier quoted context omitted.

So if you'll indulge me -- this list is exactly why a system like Kubernetes is valuable and why I think personally that it contains a lot of essential complexity. Kubernetes attempts to do all of the above, which is why it's so massive, and I'd argue it's actually less complex than knowing all the tools above -- but it's an equal degree less universally applicable. In this way, it's perfect for the dev who never wan…

Devops person here: I already know all of these answers and would still choose Kubernetes over hand-rolling all of this again and again. "Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!" No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.

Both options are questionable.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#376

Post by cameronbrown »

vertex-four wrote: post #316
cameronbrown wrote: post #289

Earlier quoted context omitted.

> Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. k8s is based off of Borg, which has existed for far longer. https://research.google/pubs/pub43438/

"What Google does" is not an industry standard.

That's not what I'm saying. Your point was about the maturity of k8s which makes sense if it sprung from nowhere, but k8s encapsulates a lot of "lessons learned" of a very stable and mature product (even if proprietary).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#377

Post by root_axis »

rudolph9 wrote: post #198
root_axis wrote: post #180

Earlier quoted context omitted.

How is a configuration.nix any different than a Dockerfile?

Dockerfiles are imperative and the order of steps changes the container image that ultimately gets built. Nix packages are declarative and idempotent. Everything is based on a functional package manager and when you configure your container via nix you get the same thing every time and can easily mix and match other packages and dependencies. A contrived example is you have a program that depends on multiple versions…

The declarative and idempotent container specification seems like a nice developer QOL improvement over docker but...

> Containers built from Dockerfiles are difficult to exactly reproduce

This isn't true in my experience, at least for a level of exactness that could have an impact on the behavior of the software running in the container. I have been working with docker in various contexts for 5 years and have never experienced any issues related to reproducibility (which is pretty much the primary selling point of docker). To your point though, I can see how reproducibility issues could happen in theory, but since its not a problem in practice it doesn't make for a very compelling reason to try nix instead. Just to be clear, I'm not trying to disparage nix, it honestly sounds pretty interesting, especially as a way to automatically provision a local workstation (docker is not good at this), but I am still curious as to the practical tradeoffs in terms of nix vs docker as a tool for building containers.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#378

Post by brianwawok »

donedealomg wrote: post #325
brianwawok wrote: post #266

Earlier quoted context omitted.

Counterpoint: Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.

You could do the same with AWS ECS there is nothing special about K8s in your situation.

I do not use AWS

ECS does not bin pack, does it?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#379

Post by notaharvardmba »

brianwawok wrote: post #378
donedealomg wrote: post #325

Earlier quoted context omitted.

You could do the same with AWS ECS there is nothing special about K8s in your situation.

I do not use AWS ECS does not bin pack, does it?

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#380

Post by yCombLinks »

DevKoala wrote: post #154

I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics. Introducing Kubernetes into our o…

You're comparing 18 months of migrating to K8s to doing nothing, rather than 18 months of other possible solutions.
Post reply on HN