Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

361–370 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#361

Post by Shicholas »

speedgoose wrote: post #209
Shicholas wrote: post #34

As someone whose gone the opposite way (moving from ECS to Kubernetes), I think the author is understating how good managed Kubernetes solutions are. At my current job, I use Azure's managed Kubernetes service, which does a great job at providing a consistent environment that's very easily managed, no unexpected updates, great dataviz, and if you choose, simple integrations to their data storage solutions (run statel…

You don't see random network errors, often visible with DNS, on your Azure managed kubernetes clusters?

I haven't yet our ingresses have passed routine ping checks (we use New Relic synthetics for this) for a while now. Fingers crossed.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#362

Post by treeman79 »

p_l wrote: post #270
treeman79 wrote: post #149

Earlier quoted context omitted.

I go in and out of development and ops and devops. 20 years. To me as a developer heroku is the gold standard. Docket-compose makes sense. Kubernes and helm is a giant soul crushing wtf On getting anything done. Now I’ll get it figure it out. Angular + exhaust+ typescript was a similar experience after I took a break from front end for a few years. A few months of pain, then it starts clicking. but this just seems in…

Many tutorials are unfortunately bad, they actually got worse over time. Oooold presentations (think 2016 and older) tended to talk more about basic building blocks and how they interacted, especially the design involving resources and their controllers working in a loop of "check requested state, check real state, do changes to implement requested state", and how those loops went up from single pods, through Replica…

This is the exact problem I ran into on rxjs. Every tutorial was badly out of date. Even ones written six months earlier.

You had to become a master of figuring out the direction of where community was going and stay on bleeding edge.

Some people found this fun. I have many times enjoyed this. When I want to get a boring feature built and hand off to the jr develops so I can spend some time with my kids. It’s frustrating

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#363

Post by kelseyhightower »

Coinbase built and maintains their own platform that's working for them.

Coinbase provided an analysis worth studying. The major takeaway for me: asking people to manage their own Kubernetes cluster is like asking people to manage their own hypervisors when they just want VMs.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#364

Post by smilliken »

jmchuster wrote: post #260
rudolph9 wrote: post #198

Earlier quoted context omitted.

Dockerfiles are imperative and the order of steps changes the container image that ultimately gets built. Nix packages are declarative and idempotent. Everything is based on a functional package manager and when you configure your container via nix you get the same thing every time and can easily mix and match other packages and dependencies. A contrived example is you have a program that depends on multiple versions…

Then have your Dockerfile build from source. Many of the public layers that people build their containers upon do that for that exact reason. If NixOS is only "generally speaking" correct because of the order it runs its scripts, then put it in a Docker container, where the whole point is to perfectly reproduce the order you run your scripts in order to minimize the errors that arise from doing otherwise. Still worri…

In practice, building Dockerfiles is unreliable and can produce errors because they inherit all of the deficiencies of aptitude, python packaging, javascript packaging, et al.

In practice, building nix dependencies is reliable and reproducible because of the militant isolation of build environments, hash integrity checks on _all_ inputs, explicit dependencies, and the long tail of bug fixes related to non-determinism in unix tooling.

This is my personal experience, not abstract.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#365

Post by StreamBright »

harpratap wrote: post #323
StreamBright wrote: post #310

Earlier quoted context omitted.

>> I really don’t know why AWS/Azure/GCP haven’t cottened onto this Conflict of interest. If k8s yields to the most revenue why would they try to decrease that? If some customers are so delusional that they go for an inefficient abstraction so be it. Btw. this is my experience with k8s too, people use it because it is a trend. Not a single company / developer could justify using it to me over leaner resources like EC…

How about companies who don't want to rely on cloud and want on-prem as much as possible? Or those who don't want to be tied to a single vendor? Apple is already getting rid of their Mesos based PaaS and moving to Kubernetes.

>> How about companies who don't want to rely on cloud and want on-prem as much as possible?

I don't know. Are you saying that staying on-prem means you have to use k8s? Google is not a single vendor?

>> Apple is already getting rid of their Mesos based PaaS and moving to Kubernetes.

And? Should every other company follow Apple?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#366

Post by vegai_ »

lsofzz wrote: post #317
vegai_ wrote: post #206

Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails. Kubernetes supports mostly Linux, although has recently gained Windows node support.

> Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails. I think `rkt` and `lxc` are missing from your list.

LXC yes, but RKT is marked as deprecated on Nomad's documentation.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#367

Post by egberts1 »

This is why I ditched systemd, et. al. and went bareboned embedded Linux on QEMU. Then build up using Makefile. Probably could have used Yast, but didn’t want the huge pullin of libraries.

BUT, my prototyping turnaround is way much faster and more stable than K8s. Way less staffing too.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#368

Post by GordonS »

jillesvangurp wrote: post #210
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

It's a pity that docker swarm did not make it. It wasn't perfect but it was a lot simpler to setup and manage than kubernetes. If you can get away with it, vanilla docker hosts running docker compose provide most of the same benefits with a fraction of the cost. For most startups, that's a great way to avoid getting sucked into a black hole of non value adding devops activity. You lose some flexibility but vanilla ub…

I'm also a fan of swarm, and still use it in production.

It's just so damn simple in comparison to k8s - basically, if you know Docker Compose, you know Docker Swarm.

I appreciate it doesn't have the full power of k8s, but it has what most apps need: simple deployments, zero-downtime updates, distributed configs and secrets.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#369

Post by brianwawok »

bvm wrote: post #322
brianwawok wrote: post #266

Earlier quoted context omitted.

Counterpoint: Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.

I agree with you completely, we use Kops and see a similar workload. The real boon for us is not in production, where the HA/error-tolerant/easy horizontal scaling certainly helps, but in development, where we can easily bring up ephemeral feature branch deploys as part of the CI/CD pipeline (which itself runs on k8s using Gitlab CI)

Yah.

I mean, k8s has 1000s of features. I am not using every one of them. I carved out the subset I need, and it is FANTASTIC. I get

* Great rolling deploys

* Self healing clusters

* Resource sharing / bin packing (rather than have 20 half used servers, I can have 6 much more heavily utilized servers)

I could maybe frankenstein these features on top of something else.. but frankly it seems absurd. If these are the only 3 features I use to run docker pods, k8s is a HUGE win to me.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#370

Post by thomasrognon »

jrockway wrote: post #357
sciurus wrote: post #352

Earlier quoted context omitted.

> What people really want is not Managed Kubernetes but "namespace as a service". They just want to kubectl apply something and let a background task provision their machines. They don't want to screw around with RBAC, service meshes, managing the Linux distribution on their worker nodes, managing the master nodes, etc. That service unfortunately doesn't exist. I think you just described AWS Fargate and Google Cloud…

AWS's version is pretty half-baked. You can't provision or use persistent volumes (so no stateful apps), and you have to use their load balancer which terminates TLS (preventing your software from being able to do ALPN, using Let's Encrypt, supporting HTTP/3, etc.). Cloud Run just seems like standard "serverless" stuff, nothing to do with Kubernetes. (The downsides involve not being able to run applications that are…

I use Cloud Run and nothing is specially developed for it, other than just being stateless. I can take my container and run it on a generic Linux box with zero changes (in fact, I run it in WSL2 on my Windows machine all the time). And I just install the normal Node.js, imagemagick, etc in my Dockerfile, no special builds or flags.
Post reply on HN