Container technologies at Coinbase: Why Kubernetes is not part of our stack • Hacker News Board

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#411

Post by dlidstrom » Mon, Jun 15, 2020, 10:34 AM UTC

mvanbaak wrote: post #262

jen20 wrote: post #163

Earlier quoted context omitted.

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do yo…

> Somewhat flippant answer here: the right number of MongoDB servers is zero. AMEN to this one. Very nice answer. It's the way we run infra.

What does this mean?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#412

Post by vandalatyou » Thu, Jun 18, 2020, 2:38 PM UTC

folmar wrote: post #391

vandalatyou wrote: post #385

Earlier quoted context omitted.

I'm a fan of cloud native kubernetes. But: I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. But: I've also been put into the terrible position of fighting vendors armed with agile, k8s and microservices selling that combined mayhem as a replacement for an openmp,openacc based massively parallel, on prem, bare metal HPC sy…

> I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. Would you share your experience?

Sorry, can't share details as I enjoy being employed.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#413

Post by whatsmyusername » Sat, Jun 20, 2020, 5:12 PM UTC

amerine wrote: post #213

whatsmyusername wrote: post #185

Earlier quoted context omitted.

I'm sure it works fine if you can get it running (documentation didn't work when I played with it). I'm referring more to the $200/month it was per control plane. That to me is a product they offer because someone else is offering it, but they don't want you to actually use it.

I take it the isolated $200/control plane is too expensive. What do you think the right price point is for that isolation?

ECS control planes cost nothing.

If you're actually looking to build isolation in AWS then you're going to need EC2 dedicated for your EKS member hosts. So you're not getting isolation for $200/month (I'd have to spec it out but dedicated hosts are pricey to the point that it'd be competing with physical hardware in a colo).

That $200/month for not-really-isolated is also per plane. So if you want a separate staging environment it's another $200/month. Client API sandbox? Same thing. It's wack.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#414

Post by whatsmyusername » Sat, Jun 20, 2020, 5:19 PM UTC

StreamBright wrote: post #311

whatsmyusername wrote: post #58

Earlier quoted context omitted.

The 'needing a team' aspect of Kubernetes sounds remarkably similar to conversations I had like 8 years ago when Openstack was the new hotness. We went with ECS and have been happy with it. It plays well with all of AWS's other products and features. For the few things we have to run On-Prem we use Docker Swarm in single node mode and it works well (albeit missing a few features like crons from Kubernetes).

Yep. I remember the Openstack consultants swarming to office trying to sell it as a solution and not a giant overhead / problem. Luckily they failed the POC so we did not need to waste our life on a non-issue trying to solve it with a non-solution. Now it is k8s' time to do the same. We will see how far this buzzword train gonna go.

TBF I like k8s as a service or platform. Once metallb came out to solve the "bare metal load balancer" problem it could deliver the entire product I was looking for.

My big issue with it is the underlying complexity and house of cards nature of running your control plane as sidecar containers on your runtime infrastructure.

You CAN set it up and run the control plane out of band, but last I looked there wasn't step by step documentation for doing so. I also couldn't find anyone doing it in prod which to me is a nonstarter. If I can't figure it out myself and can't hire for it I'm not doing it.

I'm sure it's fine on google cloud, but ECS solves 90% of our problems AND integrates with everything else we're using already.