My company is about 10 years behind so I'm not using K8s yet. Hopefully all the issues will be sorted by the time I get to it.
Container technologies at Coinbase: Why Kubernetes is not part of our stack
81–90 of 414 posts
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#82Earlier quoted context omitted.
K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc. For people who already know Linux, kubernetes comes naturally because it is pretty obvious. But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.
Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.
For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere.
I mean, you terminate one computer, k8s will figure out that the program should be running and it will be started in another computer. (Including moving detaching/attaching the block device from/into computers, this kind of things)
There is no much secret.
For me it really feels like an (well designed/integrated) API on top of standard Linux technology (ipvs, netfilter, mount, process mgmt, virtual ip, etc) via yaml running your stuff in 1..N computers but you manage it as one big computer :)
EDIT: added more context within "()"
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#83Earlier quoted context omitted.
The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.
Why use cronjobs at all for this? This is a classic work queue problem.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#84One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…
We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of…
Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as the remedy so I think you're right on the money with your statement.
When the Project Management Committee (PMC) voted to put Aurora in the attic we were all super bummed but we just ran out of interested developers :(.
The PMC agreed to kick off an "official" fork but so far it's just me maintaining it: https://github.com/aurora-scheduler/aurora
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#85Earlier quoted context omitted.
We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of…
5 dedicated folks = a lot of dollars. $500k-1M? Do those 5 folks keep things up reliably without burning out? Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure. edit: meant to say mesos but applies to both
But, to be fair, they’re not dedicated to only keeping the lights on, they probably spend 20% (1 eng year per year) of their collective hours fighting fires, so I don’t think burnout is too much of an issue. The remainder of their time is spent maintaining libraries and web interfaces. They’re a pretty standard “platform engineering” team.
In my perspective, it doesn’t matter what technology you go with (K8s or otherwise): a responsibly-managed your platform team requires at least one team with a full on-call rotation (i.e. at least 4 engineers), depending on how wide your golden path is.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#86Fyi to anyone who is reaching (or close to reaching) this point: this is exactly the problem we're trying to solve with EnvKey[1] in a secure, robust, and scalable way.
Our current product runs on the cloud (using zero-trust end-to-end encryption) and solves the problem quite well imho for companies with up to 50 or so engineers and moderately complex infrastructure. It fits easily into either containerized or non-containerized stacks.
But our v2 is almost done after years of work, and I think it will now be able to handle almost any scale and workload. The launch target is August 1st. Some cool features it will offer:
- Source available self-hosting with auto-scaling, HA, and strong consistency that just works.
- "Config blocks" that can be used in multiple projects, allowing you to de-duplicate configuration and secrets.
- Version control with simple and advanced rollback capability.
- Comprehensive access logs with simple and advanced filtering and auditing capabilities.
- Ability to manage local environments.
- Ability to react to updates and e.g. restart servers when config values change.
- A CLI that will have full parity with the UI, either for automation or those who prefer a CLI-based workflow.
- An option to use our UI via an incognito web browser pointed at localhost instead of Electron (for those who hate Electron)
- Faster, lighter, modernized end-to-end encryption built on NaCl (v1 uses OpenPGP, which is great, but it's time to move on).
- Device based auth with optional passphrases (think SSH) and an easy, secure workflow for granting access to new devices.
- Ability to authenticate and invite users via Github, Gitlab, Google, Okta, or SAML (including inviting from multiple sources within a single org).
- Teams/groups and advanced group management: grant teams of users access to groups of apps, connect groups of blocks to groups of apps, etc.
- Customizable everything: environments, sub-environments, and access roles can all be molded to fit your workflow.
Our goal is to fully solve this crucial piece of the stack so that it "just works" with minimal time spent on integration (for new projects it can be installed and integrated in minutes). If you're interested (and/or want early access), submit your email to the form at the bottom of our site: https://www.envkey.com -- and of course if the v1 can solve your configuration and secrets management needs as it already does for hundreds of our customers, give that a shot! Upgrading from v1 to v2 will be very quick so you won't be duplicating any work.
Also, we're hiring--remote anywhere in the US. We'll put up a jobs page with more detail soon, but our stack is TypeScript (node + react), Go, and polyglot (since we need to write lots of integrations). Shoot me an email if this sounds like a system you'd be excited to work on: dane@envkey.com
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#87In the on prem case, you already have a dedicated infra team that probably has the tools to effectively deploy and manage a cluster.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#88Earlier quoted context omitted.
> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…
Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…
Not sure what you mean here.
Kubernetes is designed to be cloud and vendor agnostic.
And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#89> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…
> This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. I've experienced this as well. At the last large company I worked at we had a…
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#90Earlier quoted context omitted.
> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…
I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well. An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.
Kubernetes is designed for a cluster of nodes not a single one.
Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.