Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

121–130 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#121

Post by httpz »

chillfox wrote: post #115
d0m wrote: post #50

Quick shout-out toAptible (Heroku for HIPAA) which have been amazing for our healthcare startup. Still, I honestly feel that we've taken a step back in the industry where we went from ec2 --> heroku --> k8s. But I know there are many people working hard to create the next infrastructure so we don't have to deal with containers and all that re-inventing dev-ops non-sense for every project.

Heroku is missing the ability to get a dedicated IP for a project.

You just have to pay more to get private spaces on Heroku.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#122

Post by j4ah4n »

threeseed wrote: post #90
mmgutz wrote: post #64

Earlier quoted context omitted.

I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well. An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.

I don't understand this comment at all. Kubernetes is designed for a cluster of nodes not a single one. Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.

I presume they are using docker swarm in this instance.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#123

Post by yjftsjthsd-h »

flowerlad wrote: post #100
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

DNS round-robin sends inbound to 2 haproxy servers, which proxy traffic on to the rest of the cluster. Scaling means "add another box, install your app, add haproxy entry". Service discovery is just an internal-facing haproxy front. If you must have "volumes from cloud storage", you use NFS (but if you can help it, you don't do that). Updates, restarts (including zero-downtime courtesy of draining the node out of haproxy), etc. are all quite doable with SSH and shell scripts. You run an actual monitoring system, because it's not like k8s could monitor server hardware anyways. Likewise, syslog is not exactly novel. I... don't understand why you're port forwarding? Either run on the dev cluster or your laptop.

So yes, you'd need a handful of different systems, but "k8s" is hardly a single system once you have to actually manage it, and most of the parts are simpler.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#124

Post by yjftsjthsd-h »

manigandham wrote: post #99
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

You can't make the complexity disappear. Kubernetes just offers a very standardized, stable and relatively polished way to handle it. The other alternatives, like running your own orchestration system, are just as complex if not more so, although you might be more familiar with it since you built it. That being said, many companies don't need any of it in the first place for their scale, and that's probably the bigge…

I'd say it the other way; k8s has to do all the things that another management system would have to do, but sufficiently tied together that you can't ease into it as needed.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#125

Post by schmichael »

mpfundstein wrote: post #10
nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

Nomad, Consul + Swarm is still a lovely solution and I prefer it a lot to K8s. K8s is a big monolith and often way too complex for my personal use cases. I hope Hashicorp sooner or later builds a proper replacement for Swarm so that we can have overlay networks without hassle. I know there is Weave, but never tried it.

Broader CNI and multi-interface networking support coming in 0.12!

Would love to hear what else you miss from swarm: schmichael@hashicorp.com

Update: Honestly if you haven't tried out our Consul Connect integration, please do. mTLS with just "connect { sidecar_proxy {} }" stanza and Consul running: https://www.nomadproject.io/docs/integrations/consul-connect...

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#126

Post by snuxoll »

ramraj07 wrote: post #109
snuxoll wrote: post #72

Earlier quoted context omitted.

Not the GP, but I honestly couldn't tell you. A lot probably comes down to tooling, the applications you are deploying, security requirements, etc., as well as how familiar you are with k8s itself. I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applicatio…

Seems like you are a thoughtful engineer who can also make sure you don't make any fundamental design flaws while building these systems well. Whenever I have seen kubernetes fail it's often also because the engineer(s) who built it were not thoughtful at all and often didn't fully understand what they were doing. Perhaps k8s's failing is it makes people who dont know enough think that they do.

I mean, I’m far from perfect - the trick has always been to KISS. I don’t use istio, it’s absolute overkill for my needs. I use nginx-ingress because it fits the bill, I know nginx as do enough other people that they could exec into a pod to debug it. I don’t run stateful applications that aren’t prepared to have servers randomly vanishing because it take a LOT of work to get these running in-cluster. I don’t use public helm charts because they often suck and making your own container is something you can do quickly if you were able to deploy the software on a traditional server. Every choice I make is done with the day 2 operations in mind, not what is hot, what gets initially deployed fastest - but what makes it so I can touch the thing as frequently as possible.

PCGW is a great example - installing a new Mediawiki extension, changing a config file, upgrading to a newer MW release is just updating a file or a git submodule and committing. I don’t get paid thousands a month to manage the site for the owner, so I make my time spent as efficient as can be done.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#128

Post by andrenth »

ramraj07 wrote: post #109
snuxoll wrote: post #72

Earlier quoted context omitted.

Not the GP, but I honestly couldn't tell you. A lot probably comes down to tooling, the applications you are deploying, security requirements, etc., as well as how familiar you are with k8s itself. I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applicatio…

Seems like you are a thoughtful engineer who can also make sure you don't make any fundamental design flaws while building these systems well. Whenever I have seen kubernetes fail it's often also because the engineer(s) who built it were not thoughtful at all and often didn't fully understand what they were doing. Perhaps k8s's failing is it makes people who dont know enough think that they do.

I think this is a big failing of the DevOps movement as a whole (at least what DevOps became in practice — devs doing ops) which results in things like passwordless mongodb exposed to the internet...

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#129

Post by schmichael »

michaeldwan wrote: post #45
nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

We're using Nomad to manage a large fleet of firecracker vms at fly.io. It's not as robust as k8s, but I think that's a feature. It's well documented, extensible, and predictable. Not a big community, but hashicorp folks are responsive on GitHub.

We're working on community! There's an office hours tomorrow: https://mobile.twitter.com/HashiCorp/status/1270126346103132...

There's also https://discuss.hashicorp.com/ and we're discussing (harhar) how else to improve our community relations.

Post reply on HN