Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

21–30 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#21

Post by EdwardDiego »

> The only way to sanely run Kubernetes is by giving teams/orgs their own clusters

Funny, we're running it sanely without doing that. We've separated our clusters based on use-case - delivery vs. back-end, aiming towards the "cell-based" architecture.

> Managed Kubernetes (EKS on AWS, GKE on Google) is very much in its infancy and doesn’t solve most of the challenges with owning/operating Kubernetes (if anything it makes them more difficult at this time)

...some details on the challenges they don't solve, or indeed make more difficult would be good.

But yep, K8s is complex. So, to paraphrase `import this`, you only want to use it when you have sufficiently complicated systems that the complexity is worth it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#22

Post by thrwn_frthr_awy »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase.

I've experienced this as well. At the last large company I worked at we had a Heroku-like system to run our apps on. That was deprecated for a Docker-based solution. And then _that_ has now been deprecated for a Kubernetes offering. We just ran some Python web apps–we didn't want to have to learn and support an entire system. And here's the thing, most big tech companies I've worked with are all made up of these small, "internal" services, that just want a simple place to run their services.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#23

Post by outworlder »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> so we kept running into dev-ops walls

This reads like an oxymoron. 'DevOPS' was supposed to break down walls between teams. You probably have a traditional "Ops" organization, regardless of team naming conventions.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#24

Post by dpenguin »

RcrdBrt wrote: post #18
dpenguin wrote: post #15

Can fully relate to the author. We have been struggling with effective k8s at our company for 2 years now. Too much to learn to get your first service in production. You will end up writing wrappers after wrappers that make you think “we can’t be the first guys to solve this” and after googling, you find that every issue you find is described as “just” one of the downsides of using k8s and here’s how we overcame it.…

What about docker swarm? It seems to answer your request about an orchestrator as easy to use as Docker

Sorry “typo”. I meant, as easy to use as Heroku is.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#26

Post by [deleted] »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

[deleted]

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#27

Post by champtar »

vbernat wrote: post #6

The history is a bit fuzzy. The interesting feature introduced in 2.6.24 was PID and network namespaces. Containers were "complete" by Linux 3.8 with user namespaces. Cgroups are not that important to be able to build containers (isolation first). There were other out-of-tree technologies before that, notably VServer and OpenVZ.

Not sure user namespaces are the default yet sadly

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#28

Post by corford »

nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

FWIW, where I work we're going with Nomad over K8S. It gives us everything we want and nothing more (plus it's from a source we trust and love: HashiCorp).

The nice thing with Nomad is, thanks to it's straightforward design/approach, it should be easy(ish) in the future to swap it out and go with something else if we outgrow it (or HashiCorp abandon it for some reason).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#29

Post by avita1 »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

I agree with most of this, but was surprised by your comment:

> Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful).

How often/how many cron jobs are you running that spawning a new pod per job is a problem ?

Post reply on HN