Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

61–70 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#61

Post by splix »

ur-whale wrote: post #16

Given Coinbase's reliability track record [1][2][3], I'm not entirely sure why I'd listen to devops "wisdom" coming from their corner. [1] https://www.reddit.com/r/CoinBase/comments/gh3b5t/coinbase_c... [2] https://www.reddit.com/r/Bitcoin/comments/6gtwyi/every_singl... [3] https://www.reddit.com/r/CryptoCurrency/comments/ggr80l/i_do...

Wanted to mention the same. Though I kind of disagree with the conclusion. Though the have a lot of problems with the stability of their service, it also means they know what they want to solve with their current architecture. So we'll see

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#62

Post by joana035 »

ponyous wrote: post #51
flowerlad wrote: post #42

Earlier quoted context omitted.

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

I feel like what 98% companies need is really just barebones linux with some good documentation how to spawn new nodes. To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often. What I really wonder is where did you learn k8s, which parts did you learn the most? It seems hug…

K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc.

For people who already know Linux, kubernetes comes naturally because it is pretty obvious.

But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#63

Post by theptip »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> > We would need to build/staff a full-time Compute team

I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so programmable. The idea that you'd need a completely new full-time team doesn't grok for me; that new team should replace another team that's no longer needed, or more likely, involve a combination of hiring some experts and retraining your existing engineers.

I do take seriously the other issues raised RE: security (though one GKE cluster per security boundary is a perfectly reasonable approach and gets you further than you might think).

> Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy

I think this is a different issue than the OP was raising; in any case, in order for a technology to succeed, you need to have subject matter experts embedded in your dev teams, or a separate function that provides the service to the teams that use it.

In the context of the OP, I think your case would be more like saying "hey data team, you need to build your data jobs into AMIs, go figure it out". Regardless of the technology chosen, it's not going to succeed if the teams doing the work don't know how to use the tools.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#64

Post by mmgutz »

flowerlad wrote: post #42
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well.

An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#65

Post by rudolph9 »

Having used Kubernetes extensively at my last position, adoption of k8s strikes me as similar to the adoption of Linux as your desktop OS at the Turn of the century. Some people are doing amazing things with it! Other never figure out how to get their WiFi to consistently work and are bitter people keep talking about it.

Eventually something akin to Ubuntu will grow up in the k8 ecosystem and people will stop complaining that WiFi doesn’t work.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#66

Post by theptip »

whatsmyusername wrote: post #59
theptip wrote: post #56

Anecdata: Series B startup. I've found GKE to be almost completely painless, and I've been using it in production for more than 4 years now. I don't think the article gave a fair representation on this count; sharing a link to a single GKE incident that (according to the writeup) spanned multiple days and only affected a small segment of users doesn't (for me) substantiate the claim that "it isn’t uncommon for them t…

EKS is a feature parity product. The pricing makes that painfully obvious.

Perhaps, in the box-ticking sense.

I'd be surprised if they have the same operational maturity, which translates to uptime, which was what I was talking about.

A quick search turns up these SLAs:

vs

https://cloud.google.com/blog/products/containers-kubernetes...

EKS: 3 nines

GKE: 3.5 nines

So Google is committing to 1/2 the downtime.

(Note the GKE SLA is for regional clusters, which is what you should be doing if you care about uptime. The zonal cluster SLA is 2.5 nines. I couldn't find a difference in EKS, maybe there's an equivalent better SLA for regional clusters I couldn't find.)

Edit - that EKS SLA link formatted weirdly, and so I did a little more digging and found a more recent SLA which matches GKE: https://aws.amazon.com/about-aws/whats-new/2020/03/amazon-ek...

So, per my original comment, I am surprised. (Having never used EKS directly I have no idea what their actual uptime is; in my experience GKE has been way higher than 3.5 nines, but obviously I don't have enough data to make statistically significant observations on this.)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#67

Post by jupp0r »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls

This seems like an organizational problem to me. "DevOps walls" sounds like there is a "DevOps team" (a famous DevOps anti-pattern) and there are knowledge silos between development and ops, which ironically is the exact opposite of what DevOps is about. What this also means is that developers need to be aware of the environment in which their services run and should be very familiar with how k8s works, why not take that as an opportunity to learn?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#68

Post by rudolph9 »

One thing people forget is containers don’t necessarily not to be created with Docker.

Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers

The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we have separate repo for sharing massive container images sometimes multiple GB rather that a simple file that can be tracked in git and will produce the same image every-time has been very eye opening.

Note: NixOS isn’t perfect but it has a growing community and hands down the best thing out there except for maybe guix which has container support too https://guix.gnu.org/blog/2018/tarballs-the-ultimate-contain...

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#69

Post by freedomben »

rb808 wrote: post #57

We haven't even got to ops stage yet. Small team has spent 6 months trying to repackage our app into openshift and still barely works. I think everyone regrets even looking at it.

What sort of challenges have you run into? What are some of the things the "small team" have done for the last 6 months?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#70

Post by brown9-2 »

theptip wrote: post #63
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups.

It’s great if their own solution works well for them at less cost, but that system didn’t built itself and has non-zero maintenance costs.

Post reply on HN