Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

101–110 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#101

Post by wwright »

zaptheimpaler wrote: post #75
flowerlad wrote: post #42

Earlier quoted context omitted.

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

“Plain old Linux” really isn’t an alternative to K8S though. You would need a load balancer, service discovery, a distributed scheduler, a configuration management system (K8S is a very strong alternative to building things around Ansible IMO). You can do all of those things without K8S, of course, but not with “plain old Linux” (what would that be anyway? GNU coreutils and the kernel? Vanilla Debian stable?)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#102

Post by ridv »

gen220 wrote: post #94
ridv wrote: post #84

Earlier quoted context omitted.

I'm a former Apache Aurora maintainer. Aurora has been (and continues to be) awesome for us and I'm so happy to hear other folks are still using it and it's working out for them. Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as t…

Oh it’s super cool to see you in the wild! To clarify, I had a lot of qualifying thoughts running through my head when I said “kinda sucks” (hence the “kinda”!). :) I actually think managing aurora configs is way easier than managing yml files, and I agree that I think aurora configs were ahead of the game: having access to python in your config feels like a super power. I feel like we’ll converge on something that c…

Oh no worries, no offence taken at all with the comment, configurations files tend to suck in general :).

Pystachio was indeed very forward looking and the folks who worked on this at Twitter at the time deserve all the credit there.

I think what you mention is a general problem I've encountered with IDEs when it comes to dealing with Python (esp. the "go to" issue you mention). Even when I've had to touch the Aurora client code, which is full on Python code, I've come out pulling my hair thanks to PyCharm acting wonky.

> We have maybe 2-3k aurora config files

Those are some big files! The boiler plate stuff is definitely something I've heard before from users but, unfortunately, there doesn't seem to be a better answer.

When it comes to managing job configs, I'm pretty low on the pecking order in terms of knowledge since we ended up creating our own thrift client using Go to use with Aurora. (As a consequence, all our job definitions exist as Go code.)

Stephan Erb (https://twitter.com/erbstephan/) may have better advice in this case. Some of the Twitter folks may have good info too, but they've been radio silent for months.

> I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers.

That's definitely a great point and a great compliment to the project. There's a lot of love that went into this project and I'd be ecstatic to get some new contributions, even if it something simple like fixing documentation or bumping up dependencies :D.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#103

Post by sk0g »

danenania wrote: post #86

"For example, most folks that run large-scale container orchestration platforms cannot utilize their built-in secret or configuration management. These primitives are generally not meant, designed, or built for hundreds of engineers on tens of teams and generally do not include the necessary controls to be able to sanely manage, own, and operate their applications. It is extremely common for folks to separate their s…

Unfortunately I live in not-US (Aus), but I have to say, envkey looks very good! I'm actually floating this and HashiCorp Vault by management at the moment, since even with 4 devs, keeping environment variables in sync is a PAIN!

If you don't mind me asking, how does the Go integration work? I initially thought it was actually some sort of alternate `os.Getenv()` that you imported, but that doesn't seem to be the case. And what would the latency be, for changes in environment variables being synced to running deployments?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#104

Post by stock_toaster »

Here[1] is a better link to the "jails > FreeBSD" paper (aka, "Jails: Confining the omnipotent root").

I believe it initially shipped as part of FreeBSD 4.0 in March 2000.

Wikipedia says[2] it was in use in the late 90s in a hosting provider before inclusion in FreeBSD though. Interesting! TIL.

[1]: https://papers.freebsd.org/2000/phk-jails.files/sane2000-jai...

[2]: https://en.wikipedia.org/wiki/FreeBSD_jail

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#105

Post by [deleted] »

wwright wrote: post #101
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

“Plain old Linux” really isn’t an alternative to K8S though. You would need a load balancer, service discovery, a distributed scheduler, a configuration management system (K8S is a very strong alternative to building things around Ansible IMO). You can do all of those things without K8S, of course, but not with “plain old Linux” (what would that be anyway? GNU coreutils and the kernel? Vanilla Debian stable?)

[deleted]

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#106

Post by CorpOverreach »

theptip wrote: post #63
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

> that new team should replace another team that's no longer needed, or more likely, involve a combination of hiring some experts and retraining your existing engineers.

The gotcha there is that it rarely goes that way unless you have a very clear direction from senior management, at least at big corps. In most cases, it's just another thing that gets added to the pile, and it's incredibly difficult to migrate entirely out of whatever the old solution was, so now you end up supporting both.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#107

Post by theptip »

One note on the security side of things -- if you're interested in seeing what a truly hardened k8s/GKE configuration looks like, check out the Vault examples:

https://learn.hashicorp.com/vault/kubernetes/k8s-reference-a...

https://github.com/sethvargo/vault-on-gke

In summary, for your security-critical workloads you're going to want to put them in their own cluster; treat k8s in this case as an API for updating the code that's running on your VMs. (Except your VMs can run a stripped-down read-only OS like Container-OS or CoreOS).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#108

Post by solidasparagus »

dashwav wrote: post #53
lifeisstillgood wrote: post #46

Earlier quoted context omitted.

Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(

TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so. I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to…

> managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly

And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#109

Post by ramraj07 »

snuxoll wrote: post #72
lifeisstillgood wrote: post #46

Earlier quoted context omitted.

Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(

Not the GP, but I honestly couldn't tell you. A lot probably comes down to tooling, the applications you are deploying, security requirements, etc., as well as how familiar you are with k8s itself. I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applicatio…

Seems like you are a thoughtful engineer who can also make sure you don't make any fundamental design flaws while building these systems well. Whenever I have seen kubernetes fail it's often also because the engineer(s) who built it were not thoughtful at all and often didn't fully understand what they were doing. Perhaps k8s's failing is it makes people who dont know enough think that they do.
Post reply on HN