Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

171–180 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#171

Post by techntoke »

darkwater wrote: post #19

It puzzles me a bit that they don't want to invest on.k8s expertise (at their scale, especially security-wise can be though, I guess) but at the same time they develop their own deploy system which sounds a lot like Spinnaker and they have their own secrets/config management system (Hashicorp's stack is pretty neat and battle tested).

Coinbase hasn't scaled very well IMO, probably because of people like this. Once you try out Kraken it is like night and day difference in terms of features and supported currencies.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#172

Post by yjftsjthsd-h »

manigandham wrote: post #135
yjftsjthsd-h wrote: post #124

Earlier quoted context omitted.

I'd say it the other way; k8s has to do all the things that another management system would have to do, but sufficiently tied together that you can't ease into it as needed.

I guess that depends primarily on whether you're installing and operating K8S yourself. Use something like GKE and it's a very seamless experienced, with AKS getting pretty good and the rest being rather crummy. Once you have a managed cluster, the deploying apps is fairly easy. A single container/pod is a 1-liner and you can work your way up from there.

That's fair; all of this is heavily influenced by your operating environment. If you can run GKE or if you have an ops team to deal with that stuff, then yeah k8s is great. Unfortunately, I'm part of the ops team, and our company is too small to have a dedicated k8s team and too low-budget to (likely) do well with a managed service (we get absurd value per money out of bare metal servers, which is very much a tradeoff that is sometimes painful). So to me, k8s looks like a very iffy tradeoff. Bigger company, bigger budget, different constraints? Yeah, k8s would be great.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#173

Post by harpratap »

erulabs wrote: post #139
dpenguin wrote: post #15

Can fully relate to the author. We have been struggling with effective k8s at our company for 2 years now. Too much to learn to get your first service in production. You will end up writing wrappers after wrappers that make you think “we can’t be the first guys to solve this” and after googling, you find that every issue you find is described as “just” one of the downsides of using k8s and here’s how we overcame it.…

> I wish someone can make a deployment orchestrator for your private DC that is as simple to use as Heroku is. This is what I'm working on, although it is built on top of Kubernetes! We feel Kube suffers from a disconnect between the insane complexity requirements of enterprise deployments and what most coders and businesses actually need - which is exactly what you said: a consistent Heroku-ish experience on their o…

I don't think there is a disconnect, the K8s team knows about it but this is not a problem meant to be solved by K8s alone. Project like Knative, OpenDeis, Fargate, Cloud Functions are supposed to provide Heroku-like PaaS on top of Kubernetes.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#174

Post by michaeldwan »

schmichael wrote: post #129
michaeldwan wrote: post #45

Earlier quoted context omitted.

We're using Nomad to manage a large fleet of firecracker vms at fly.io. It's not as robust as k8s, but I think that's a feature. It's well documented, extensible, and predictable. Not a big community, but hashicorp folks are responsive on GitHub.

We're working on community! There's an office hours tomorrow: https://mobile.twitter.com/HashiCorp/status/1270126346103132... There's also https://discuss.hashicorp.com/ and we're discussing (harhar) how else to improve our community relations.

That's great to hear!

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#175

Post by calt »

Their system seems very sane. I'm jealous, and I hope it stays that way (sane, not static). It's also entirely dependent of features of AWS. I used to work for a company that had a very insane hybrid datacenter/aws deployment environment, and containers provided some sanity.

Service discovery, cluster management, secret storage, these are all problems that we _already_ had. Containers (for us on mesos) just solved some parts of that picture.

I lived and breathed containers, distributed systems, config management, deployment pipelines etc. for years, and I forget that to many K8s is just seen as one magic bullet solution. You will have to pick it apart and interact with pieces of it if you really want to use it at a medium sized company. That takes a lot of research and understanding.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#176

Post by harpratap »

erulabs wrote: post #139
dpenguin wrote: post #15

Can fully relate to the author. We have been struggling with effective k8s at our company for 2 years now. Too much to learn to get your first service in production. You will end up writing wrappers after wrappers that make you think “we can’t be the first guys to solve this” and after googling, you find that every issue you find is described as “just” one of the downsides of using k8s and here’s how we overcame it.…

> I wish someone can make a deployment orchestrator for your private DC that is as simple to use as Heroku is. This is what I'm working on, although it is built on top of Kubernetes! We feel Kube suffers from a disconnect between the insane complexity requirements of enterprise deployments and what most coders and businesses actually need - which is exactly what you said: a consistent Heroku-ish experience on their o…

And I looked at your service, looks a lot like kubeapps i.e a marketplace for Helm charts.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#177

Post by calt »

adrianscott wrote: post #31
modeless wrote: post #8

tl;dr We are all in on AWS proprietary stuff instead.

Yes, except more like 'in addition to' than instead... unbelievable... #lockin

To me it looks like they saved quite a lot of engineering effort, and the price was lockin. Seems like it was probably a fair trade for them.

At least if it's sane and well thought out it can be taken back apart and repurposed for something different. There will be cost. Consider it paying back the loan. That's why it's called technical debt. The terms of the debt looks good to me.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#178

Post by mwcampbell »

yjftsjthsd-h wrote: post #172
manigandham wrote: post #135

Earlier quoted context omitted.

I guess that depends primarily on whether you're installing and operating K8S yourself. Use something like GKE and it's a very seamless experienced, with AKS getting pretty good and the rest being rather crummy. Once you have a managed cluster, the deploying apps is fairly easy. A single container/pod is a 1-liner and you can work your way up from there.

That's fair; all of this is heavily influenced by your operating environment. If you can run GKE or if you have an ops team to deal with that stuff, then yeah k8s is great. Unfortunately, I'm part of the ops team, and our company is too small to have a dedicated k8s team and too low-budget to (likely) do well with a managed service (we get absurd value per money out of bare metal servers, which is very much a tradeof…

> we get absurd value per money out of bare metal servers, which is very much a tradeoff that is sometimes painful

What do you find most painful about your use of bare-metal servers? The thing that I like most about a hyperscale cloud provider is the level of redundancy, including even multiple data centers per region, and their built-in health checks and recovery (e.g. through auto-scaling groups) based on that redundancy. With bare-metal servers, I'd have to cobble together my own failover system for the occasional time when one of those servers goes down or becomes unreachable due to a network issue. And of course, I'd probably find out that my home-made failover system doesn't actually work at the worst possible time.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#179

Post by Aeolun »

threeseed wrote: post #88
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> I don't see how k8s avoids the need to learn about cloud vendor specific tech e.g searching "aws RDS k8s" Not sure what you mean here. Kubernetes is designed to be cloud and vendor agnostic. And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.

> Kubernetes is designed to be cloud and vendor agnostic.

But it’s not. Connecting to EKS is completely different from connecting to GCP. Setting up worker nodes is completely different too, oh and load balancers.

It’s only the last mile that’s similar.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#180

Post by root_axis »

rudolph9 wrote: post #68

One thing people forget is containers don’t necessarily not to be created with Docker. Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we h…

How is a configuration.nix any different than a Dockerfile?
Post reply on HN