Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

91–100 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#91

Post by threeseed »

whatsmyusername wrote: post #58
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

The 'needing a team' aspect of Kubernetes sounds remarkably similar to conversations I had like 8 years ago when Openstack was the new hotness. We went with ECS and have been happy with it. It plays well with all of AWS's other products and features. For the few things we have to run On-Prem we use Docker Swarm in single node mode and it works well (albeit missing a few features like crons from Kubernetes).

AWS Kubernetes (EKS) with Fargate is just as simple as ECS.

Far simpler if you can include the fact that with Kubernetes you can install new applications as simply as "helm install prometheus".

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#92

Post by Karupan »

I partially blame keyword driven recruitment in tech for these kinds of responses to a platform/tooling. Kubernetes isn't a magic bullet - it is a platform which solves a very specific set of problems with scaling. And of course it doesn't come for free. You can't just throw in k8s into your existing infrastructure and expect your devs to manage it, in addition to their regular work.

And yet, we keep reading about teams falling into the trap because their lead engineer wanted to put "production kubernetes" on his resume. I hope the k8s team adds a huge "Who is this NOT for" disclaimer to their docs (if it doesn't already exist).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#93

Post by random3 »

I dropped midway through the post. It reads as a classic engineering justification for X vs A, B or C (the same justification can be made for anything vs anything).

I found it interesting that for someone that doesn't use Kubernetes they spend a lot of time describing it.

There are benefits in using mainstream solutions like Kubernetes. I've spent over a decade building distributed systems from Hadoop to Mesos and Kubernetes and have seen the pains of datacenter and AWS, Azure and GCP and all I can say is GKE works great so I don't buy the simpler in-house argument. Once you start properly integrating concerns that need to be end-2-end integrated, the complexity explodes and you end up with a partially working system.

I do believe that serverless will likely make Kubernetes and similar-level tech irrelevant for most users, however only part of a rich ecosystem that provides all other concerns.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#94

Post by gen220 »

ridv wrote: post #84
gen220 wrote: post #41

Earlier quoted context omitted.

We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of…

I'm a former Apache Aurora maintainer. Aurora has been (and continues to be) awesome for us and I'm so happy to hear other folks are still using it and it's working out for them. Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as t…

Oh it’s super cool to see you in the wild! To clarify, I had a lot of qualifying thoughts running through my head when I said “kinda sucks” (hence the “kinda”!). :)

I actually think managing aurora configs is way easier than managing yml files, and I agree that I think aurora configs were ahead of the game: having access to python in your config feels like a super power. I feel like we’ll converge on something that compiles aurora configs into yml files, prior to runtime.

That being said, we’ve never been able to get good editor support for things like “go to definition”, with the whole “include” syntax. We have maybe 2-3k aurora config files, of which maybe 100 are shared boilerplate. Do you have any advice on this? I tell vim to treat them like python files, but pylint hates them :)

We were bummed by the PMC decision too. I think some people at my company have considered becoming maintainers over the years, but, for the most part, everything “just works”, so we haven’t felt a selfish need to, so to speak. I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers. That being said, I’ll set aside some time this weekend to take a look at some issues. :)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#95

Post by threeseed »

snuxoll wrote: post #74
rtempaccount1 wrote: post #9

Earlier quoted context omitted.

I'd agree that k8s has a lot of functionality built-in, another important thing to realise is what k8s doesn't do. In addition to the well-known integration points (Container Runtime/Network/Storage Interfaces), there's things like the lack of a good built-in user authentication mechanism with Kubernetes, which means you pretty much always need some external authentication service for your clusters. That's not too ba…

> That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem. Go spin up Keycloak, join it to your user-directory of choice (or not and just use the internal directory), configure it as your authentication provider, done.

Or use Dex: https://github.com/dexidp/dex

Which has the advantage of not needing any external databases.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#96

Post by threeseed »

yahyaheee wrote: post #48
darkwater wrote: post #19

It puzzles me a bit that they don't want to invest on.k8s expertise (at their scale, especially security-wise can be though, I guess) but at the same time they develop their own deploy system which sounds a lot like Spinnaker and they have their own secrets/config management system (Hashicorp's stack is pretty neat and battle tested).

Works great! ... till it doesn’t

More likely that it works great until the lead developer leaves.

And then you're stuck with a proprietary solution that lags further and further behind established, open source projects.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#97

Post by random3 »

nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

Mesos was great. It was a much cleaner, flexible separation of concerns. Too bad Mesosphere killed it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#98

Post by antonvs »

hwbehrens wrote: post #76
joana035 wrote: post #62

Earlier quoted context omitted.

K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc. For people who already know Linux, kubernetes comes naturally because it is pretty obvious. But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.

Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.

Think of kubernetes as a cluster operating system. Instead of dealing with vms, you deal with your applications directly, without worrying about where they're running. It gives you a unified view and ability to manage a distributed system at the level of the application components.

Ansible can't give you anything like that. Even if you use Ansible to automate something like network setup, the commands and modules you use will be different between e.g. cloud providers. Kubernetes give you a consistent abstraction layer that's almost unchanged across providers, with the exception of annotations that are needed in some cases for integration with certain provider services.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#99

Post by manigandham »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

You can't make the complexity disappear. Kubernetes just offers a very standardized, stable and relatively polished way to handle it.

The other alternatives, like running your own orchestration system, are just as complex if not more so, although you might be more familiar with it since you built it.

That being said, many companies don't need any of it in the first place for their scale, and that's probably the biggest issue with K8S today.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#100

Post by flowerlad »

zaptheimpaler wrote: post #75
flowerlad wrote: post #42

Earlier quoted context omitted.

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> Plain old linux is the alternative

How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environment if you want to add a new server to your cluster? If your app has microservices how do services discover each other? How do you mount volumes from cloud storage? How do you update configuration? How do you automatically restart failed applications? How do you monitor if your applications are working? How do you make sure the right number of MongoDB replicas are running at all times? How do you view your log files remotely? How do you port-forward from localhost to your Linux server to test your app locally?

Post reply on HN