Earlier quoted context omitted.
I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…
Could you share the resources to get started? I'm in a similar boat, 2 person trying to set up K8S in DO using automation (CI/CD).
Container technologies at Coinbase: Why Kubernetes is not part of our stack
161–170 of 414 posts
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#162Earlier quoted context omitted.
TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so. I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to…
> managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#163Earlier quoted context omitted.
Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…
> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…
> How do you run an application on a cluster of plain old linux machines?
Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers).
> How do you do load balancing?
An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes).
> How do you update your app without downtime?
Blue-green deployment, or phased rollout.
> How do you ensure all your servers are running the same version of dependencies?
Build them from a common image.
> How do you update those dependencies?
Update the Packer template that builds that image.
> How do you replicate your environment if you want to add a new server to your cluster?
Start the server from the same image.
> If your app has microservices how do services discover each other?
Consul, or DNS, depending on your appetite.
> How do you mount volumes from cloud storage?
It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS.
> How do you update configuration?
Either update Consul and have it propagate configuration, or update a configuration package and push it out.
> How do you automatically restart failed applications?
Systemd restart policy.
> How do you monitor if your applications are working?
From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user.
> How do you make sure the right number of MongoDB replicas are running at all times?
Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group.
> How do you view your log files remotely?
Cloudwatch, Syslog, SSH (depending on requirements).
> How do you port-forward from localhost to your Linux server to test your app locally?
SSH.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#164Anecdata: Series B startup. I've found GKE to be almost completely painless, and I've been using it in production for more than 4 years now. I don't think the article gave a fair representation on this count; sharing a link to a single GKE incident that (according to the writeup) spanned multiple days and only affected a small segment of users doesn't (for me) substantiate the claim that "it isn’t uncommon for them t…
EKS is a feature parity product. The pricing makes that painfully obvious.
And that is why pricing and features sit right at “good enough” and not great.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#165>> at Google it isn’t uncommon for them to have multi-hour outages with GKE
For what it's worth we've been running multiple GKE clusters in production for over three years. We're medium size, with some dozen or so in-house services handling perhaps a total of 20k rps. We are rarely affected by any GCP issue, and as far as I can recall we have never been down due to a GKE-specific problem. In addition to the basic orchestration features we make significant use of ingress and storage primitives. It all quite literally just works.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#166imo running k8s in the cloud makes no sense. If you've already setup on prem servers, load balancers, config management, patching, access control, etc to allow developers to run applications on VMs k8s can provide an integrated experience with significantly less work. If you're running k8s in the cloud then just use a hosted service and leverage enterprise support. In the on prem case, you already have a dedicated in…
Tools? Yes. Care and attention? Hell no.
The ops team at my work still has important services running on CentOS 6, and they are spending all their time trying to get Kubernetes configured and working.
You would be surprised at how much infrastructure out there is just glued together adhoc. Several senior engineers at my work are successfully blocking adoption of gitops and CD.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#167Earlier quoted context omitted.
I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful). How often/how many cron jobs are you running that spawning a new pod per job is a problem ?
The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#168One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…
I'd have probably gotten used to it had I spent more time using it, but it'd have taken some rewiring of thinking process.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#1691) Those guys can't imagine their life outside of amazon. this is bad. And yes, amazon k8s was bad, it's getting better, still bad. 2) They said we need a separate team and instead they wrote and maintain their own, sort of lightweight solution. Nothing more about that article. One opinionated team decided to roll out something homemade to add it to CV later on.
Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack
#170I partially blame keyword driven recruitment in tech for these kinds of responses to a platform/tooling. Kubernetes isn't a magic bullet - it is a platform which solves a very specific set of problems with scaling. And of course it doesn't come for free. You can't just throw in k8s into your existing infrastructure and expect your devs to manage it, in addition to their regular work. And yet, we keep reading about te…
Oh God. This is what is happening at my work. We have an API that has 200 write users and a public front-end that can do reads. None of it is heavy though, with most writes occurring for a month in the Winter and a month in the Fall. In the unlikely event of heavy write loads we could just scale up CPU/RAM for those two months. Any read load be solved with cache or spending time on the worst offenders in SQL. The Lea…