tl;dr We are all in on AWS proprietary stuff instead.
Yes, except more like 'in addition to' than instead... unbelievable... #lockin
Container technologies at Coinbase: Why Kubernetes is not part of our stack
31–40 of 414 posts
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#32> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…
I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful). How often/how many cron jobs are you running that spawning a new pod per job is a problem ?
The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#33> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…
> We've since migrated to cloud functions (on GCP; but AWS lambdas could also work) and it's just been a breeze. Did you run into cold start delays with GCP cloud functions? AFAIK it is one of the reasons many prefer AWS Lambda over GCP
We actually had a cold start tolerance of ±30 seconds and so far, very few jobs are out of that range. This was one of our main sticking points, and the Google guys gave us some great tips on how to reduce cold starts.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#34As someone whose gone the opposite way (moving from ECS to Kubernetes), I think the author is understating how good managed Kubernetes solutions are.
At my current job, I use Azure's managed Kubernetes service, which does a great job at providing a consistent environment that's very easily managed, no unexpected updates, great dataviz, and if you choose, simple integrations to their data storage solutions (run stateless K8 clusters if you can) and key vault. We don't do much outside of our kubectl YAML files, which as commented below has a de-facto understanding by a large number of people.
CVEs will always exist, which is why network security is important. I think we can agree that the only ingress into your cloud environments should be through API servers your team builds, and everything else should be locked down to be as strict as possible (e.g. VPNs and SSO). With a system like K8, so many eyes on the code mean so many more CVEs will exist, so I don't find this argument compelling.
My team, and so many other teams worldwide are betting that the K8 community will accelerate much faster than roll-your-own solutions, and K8 gives us the best opportunity to create cloud-agnostic architecture. Additionally, helm charts are easy to install, and afaict more software vendors are providing "official" versions - which means for a team like mine, which is happy to pay for services to manage state, in the same vain a company chooses AWS RDS over managing their own Postgres server, we can get the same benefits as the author with a cloud-agnostic solution.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#35One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…
I know quite a few startups going the Nomad route. It might not be as mainstream, but I don't think it's going to be phased out any time soon. Rancher was still maintaining their own scheduler for a while (not sure if they still do) because there were a lot of legacy customers still on it. Racher and RancherOS has pretty much moved to a full k8s management shop though.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#36One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…
Nomad, Consul + Swarm is still a lovely solution and I prefer it a lot to K8s. K8s is a big monolith and often way too complex for my personal use cases. I hope Hashicorp sooner or later builds a proper replacement for Swarm so that we can have overlay networks without hassle. I know there is Weave, but never tried it.
Which part of Swarm are you looking to replace?
Nomad already does the container part, and Consul Connect is the networking overlay / service mesh. Work is being done to get Nomad better integrated with Connect.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#37Earlier quoted context omitted.
I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful). How often/how many cron jobs are you running that spawning a new pod per job is a problem ?
The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.
[deleted]
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#38One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…
Nomad, Consul + Swarm is still a lovely solution and I prefer it a lot to K8s. K8s is a big monolith and often way too complex for my personal use cases. I hope Hashicorp sooner or later builds a proper replacement for Swarm so that we can have overlay networks without hassle. I know there is Weave, but never tried it.
K8s is a lot of things, but a monolith it is not. Quite the opposite - the complexity comes from the large number of relatively simple components interacting in various ways.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#39Given Coinbase's reliability track record [1][2][3], I'm not entirely sure why I'd listen to devops "wisdom" coming from their corner. [1] https://www.reddit.com/r/CoinBase/comments/gh3b5t/coinbase_c... [2] https://www.reddit.com/r/Bitcoin/comments/6gtwyi/every_singl... [3] https://www.reddit.com/r/CryptoCurrency/comments/ggr80l/i_do...
While this may not be stated in the most diplomatic way, it's important and relevant for this discussion and I don't think it should be downvoted. It's very well established that Coinbase's infrastructure is unable to handle traffic spikes, despite the fact that traffic spikes are a fact of life in their industry and downtime can easily cost their customers millions of dollars.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#40> The only way to sanely run Kubernetes is by giving teams/orgs their own clusters Funny, we're running it sanely without doing that. We've separated our clusters based on use-case - delivery vs. back-end, aiming towards the "cell-based" architecture. > Managed Kubernetes (EKS on AWS, GKE on Google) is very much in its infancy and doesn’t solve most of the challenges with owning/operating Kubernetes (if anything it m…
It will catch up with you. I was at one shop with an 11 person platform team dedicated to platforms. The shop moved really fast, and even with 500 employees, they were able to moved from OpenStack to DC/OS in 2~3 months. (We had CoreOS running on open stack, but fully migrated over to DC/OS. Jenkins -> Gitlab also happened very rapidly; really good engineers).
At my current shop, we struggle to maintain k8s clusters with an 8 person team. We inherited the debt of a previous team that had deployed k8s and their old legacy stuff was full of dependency rot. We have new clusters, and we update them regularity, but it's taken nearly half a year so far and we don't have everything moved over.
You do need good teams to move fast; and good leaders to prioritize minimizing tech debt.