Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

51–60 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#51

Post by ponyous »

flowerlad wrote: post #42
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

I feel like what 98% companies need is really just barebones linux with some good documentation how to spawn new nodes.

To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often.

What I really wonder is where did you learn k8s, which parts did you learn the most? It seems huge and I would love to be in your position

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#52

Post by davidpaulyoung »

vbernat wrote: post #6

The history is a bit fuzzy. The interesting feature introduced in 2.6.24 was PID and network namespaces. Containers were "complete" by Linux 3.8 with user namespaces. Cgroups are not that important to be able to build containers (isolation first). There were other out-of-tree technologies before that, notably VServer and OpenVZ.

Where are Solaris containers in the "history"?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#53

Post by dashwav »

lifeisstillgood wrote: post #46
flowerlad wrote: post #42

Earlier quoted context omitted.

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(

TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so.

I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to have a lot of problems. The big thing to note is that starting a project on Erlang very well might solve the problems that a Java EE project ran into, but that is because they were able to solve them at the ground floor, and just popping a Java EE project with all of it's architecture into an Erlang project will probably end up in a worse spot.

I think that this is what often happens with k8s as well - if you or your company have a currently working implementation that isn't on k8s, of course you won't be able to just easily plop it into a k8s cluster and everything be all well and good, but I think the problem is that people are equating that issue with k8s itself, which is a completely different paradigm.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#54

Post by harrisonjackson »

gen220 wrote: post #41
nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of…

5 dedicated folks = a lot of dollars. $500k-1M? Do those 5 folks keep things up reliably without burning out?

Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure.

edit: meant to say mesos but applies to both

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#55

Post by [deleted] »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

[deleted]

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#56

Post by theptip »

Anecdata: Series B startup. I've found GKE to be almost completely painless, and I've been using it in production for more than 4 years now. I don't think the article gave a fair representation on this count; sharing a link to a single GKE incident that (according to the writeup) spanned multiple days and only affected a small segment of users doesn't (for me) substantiate the claim that "it isn’t uncommon for them to have multi-hour outages".

In my experience, multi-hour control-plane outages are very rare, and I've only had a single data-plane outage (in this case IIRC it was a GKE networking issue). Worst case I see is once or twice a year I'll get a node-level issue, of the level of severity where a single node will become unhealthy and the healthchecks don't catch it; most common side-effect is this can block pods from starting. These issues never translate into actual downtime, because your workloads are (if you're doing it right) spread across multiple nodes.

I wouldn't be surprised if EKS is bad, they are three years behind GKE (GA released in 2018 vs 2015). EKS is a young service in the grand scheme of things, GKE is approaching "boring technology" at 5 yrs old.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#58

Post by whatsmyusername »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

The 'needing a team' aspect of Kubernetes sounds remarkably similar to conversations I had like 8 years ago when Openstack was the new hotness.

We went with ECS and have been happy with it. It plays well with all of AWS's other products and features. For the few things we have to run On-Prem we use Docker Swarm in single node mode and it works well (albeit missing a few features like crons from Kubernetes).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#59

Post by whatsmyusername »

theptip wrote: post #56

Anecdata: Series B startup. I've found GKE to be almost completely painless, and I've been using it in production for more than 4 years now. I don't think the article gave a fair representation on this count; sharing a link to a single GKE incident that (according to the writeup) spanned multiple days and only affected a small segment of users doesn't (for me) substantiate the claim that "it isn’t uncommon for them t…

EKS is a feature parity product. The pricing makes that painfully obvious.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#60

Post by whatsmyusername »

nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

Swarm being dead is news to me, we use it for our on-prem bits without issue (though we only have a few of them).

We use ECS for the majority of our container orchestration and love it.

Post reply on HN