A story about a Kubernetes migration • Hacker News Board

Re: A story about a Kubernetes migration

#41

Post by barrkel » Tue, Oct 02, 2018, 4:47 PM UTC

shawabawa3 wrote: post #16

TeeWEE wrote: post #4

For me kubernetes is also a breeze. There is some learning curve because we started with Helm, Tiller, Grafana, Prometheus right from the start. But the kubectl command is easy to work with, and the k8s yaml files are really a breeze of fresh air compared to Ansible playbooks. We're not on production yet, but moving soon.

> k8s yaml files are really a breeze of fresh air compared to Ansible playbooks Hah. I'm a huge kubernetes fan but not sure I can agree here. k8s yaml files are the most verbose and spammy things imaginable. granted, ansible playbooks can be horrific, but i'd say that's more down to the authors of the playbook than ansible itself.

Ansible is a glorified templating language for composing, distributing and executing shell scripts.

K8s is designed around a desired state of the world with control loops.

The two are very different conceptually, and lead you in different directions organizationally.

Ansible encourages you to code the derivative and hopefully approach the integral, whereas K8s encourages you to code the integral and infer the derivative in your controller, if that makes sense.

Re: A story about a Kubernetes migration

#42

Post by colek42 » Tue, Oct 02, 2018, 4:51 PM UTC

Fishkins wrote: post #39

Since they mention it a couple times in the article, how do other folks handle auth for their k8s dashboards? I'm trying to figure out the best approach that right now.

You could build an authorization proxy that creates a token with the Kube API server and sets the Authorization header. This probably exists, but a project I worked on: https://github.com/boxboat/okta-nginx might be a good starting point.

Re: A story about a Kubernetes migration

#43

Post by falcolas » Tue, Oct 02, 2018, 4:54 PM UTC

halbritt wrote: post #30

zedpm wrote: post #24

It sounds like they really wanted to switch to K8s and rationalized it. The cons of their existing solution are minor and easily addressed with correct use of Ansible, and the massive complexity of K8s is understated. As an example, they suggest that there's a heavy cognitive load associated with having devs run some Ansible playbooks, and then argue that to avoid that, they just have to introduce an entirely new too…

Regardless of your skepticism, the benefits are real. Scaling applications in k8s, updating, and keeping configs consistent are a great deal easier for me than using Ansible or any other config management tool. In the end, it's a singular platform that one can build tooling against that allows an organization to abstract away the infrastructure. My team has done that (on top of k8s). As such, a developer can spin up…

With ECS running on Fargate, idle instances don't exist. Throw in service autoscaling, and you have a simple scaling solution with no K8s cluster management required.

Re: A story about a Kubernetes migration

#44

Post by falcolas » Tue, Oct 02, 2018, 5:04 PM UTC

halbritt wrote: post #31

tekkk wrote: post #7

I hope that the original title of the story was intended sarcasm: "Unbabel migrated to Kubernetes and you won’t believe what happened next!" But so they managed to consolidate their infrastructure around Kubernetes and Google Cloud which made the management of their servers easier and faster? I wonder how much actual money they saved but I guess it will pay off for them in the long run. I've been dabbling with Kubern…

My org made a similar transition. It's hard to articulate exactly precisely how much we saved as our "production" or revenue generating environments are roughly the same. We transitioned out of AWS where we had relatively well managed instances of our stack managed with chef and terraform to GKE in Google Cloud where we migrated to a helm chart and custom orchestration tooling on top of that. Prior to the migration I…

Idle instances are what autoscaling groups were made for. Frankly, if there are a lot of idle instances, AWS features are not being taken advantage of - that's not on them.

Re: A story about a Kubernetes migration

#45

Post by adwf » Tue, Oct 02, 2018, 5:19 PM UTC

falcolas wrote: post #44

halbritt wrote: post #31

Earlier quoted context omitted.

My org made a similar transition. It's hard to articulate exactly precisely how much we saved as our "production" or revenue generating environments are roughly the same. We transitioned out of AWS where we had relatively well managed instances of our stack managed with chef and terraform to GKE in Google Cloud where we migrated to a helm chart and custom orchestration tooling on top of that. Prior to the migration I…

Idle instances are what autoscaling groups were made for. Frankly, if there are a lot of idle instances, AWS features are not being taken advantage of - that's not on them.

Only if you've got a homogeneous workload that can scale out.

More typical is Team A working on Project A spin up a test DB + test app server. Team B also spin up a test app server + web server + DB for project B

Then Project A gets productionized and you have SIT/UAT/Stage copies of all of that sitting mostly idle.

Then project C comes along and the devs need to test on a 3 node C* cluster with 3 kafka brokers...

Suddenly you have a whole bunch of dev environments sitting mostly idle. No-one would ever fork out on reserving a t2.medium, but they all add up to $$$$$ every month. With k8s you can reclaim all that idling power, reserve some beefy instances, whilst also gaining easily deployable artefacts, CI/CD, production scaling, etc.

Re: A story about a Kubernetes migration

#46

Post by falcolas » Tue, Oct 02, 2018, 5:30 PM UTC

adwf wrote: post #45

falcolas wrote: post #44

Earlier quoted context omitted.

Idle instances are what autoscaling groups were made for. Frankly, if there are a lot of idle instances, AWS features are not being taken advantage of - that's not on them.

Only if you've got a homogeneous workload that can scale out. More typical is Team A working on Project A spin up a test DB + test app server. Team B also spin up a test app server + web server + DB for project B Then Project A gets productionized and you have SIT/UAT/Stage copies of all of that sitting mostly idle. Then project C comes along and the devs need to test on a 3 node C* cluster with 3 kafka brokers... Su…

But those use cases don't change between AWS and GCE. They only change if you start using containers in a container environment; something possible in AWS and GCE.

Re: A story about a Kubernetes migration

#47

Post by wgjordan » Tue, Oct 02, 2018, 5:37 PM UTC

falcolas wrote: post #44

halbritt wrote: post #31

Earlier quoted context omitted.

My org made a similar transition. It's hard to articulate exactly precisely how much we saved as our "production" or revenue generating environments are roughly the same. We transitioned out of AWS where we had relatively well managed instances of our stack managed with chef and terraform to GKE in Google Cloud where we migrated to a helm chart and custom orchestration tooling on top of that. Prior to the migration I…

Idle instances are what autoscaling groups were made for. Frankly, if there are a lot of idle instances, AWS features are not being taken advantage of - that's not on them.

Seconded. If the primary driver of this transition was 'getting a lot more utility for a little less money', then simply adopting Auto Scaling Groups within the existing AWS setup could have improved instance-utilization with much less migration effort.

Re: A story about a Kubernetes migration

#48

Post by oppositelock » Tue, Oct 02, 2018, 5:48 PM UTC

falcolas wrote: post #43

halbritt wrote: post #30

Earlier quoted context omitted.

Regardless of your skepticism, the benefits are real. Scaling applications in k8s, updating, and keeping configs consistent are a great deal easier for me than using Ansible or any other config management tool. In the end, it's a singular platform that one can build tooling against that allows an organization to abstract away the infrastructure. My team has done that (on top of k8s). As such, a developer can spin up…

With ECS running on Fargate, idle instances don't exist. Throw in service autoscaling, and you have a simple scaling solution with no K8s cluster management required.

Or, you use EKS, no k8s cluster management required either.

I'm running a production service on EKS, also tried it on GKE. Both take away most of the cluster management pain.

Re: A story about a Kubernetes migration

#49

Post by cdelsolar » Tue, Oct 02, 2018, 6:00 PM UTC

eeZah7Ux wrote: post #20

falcolas wrote: post #12

Earlier quoted context omitted.

Saying that Kubernetes is a bit complicated seems like saying that water can be a bit wet. Even their documentation can't keep up. And with a release cycle of 3 months, and a deprecation cycle of 6 months, you need a team dedicated to keeping up with K8s state-of-the-art; so much of that knowledge you picked up a year ago is at best stale, and at worst wrong. Sure, it makes setting up and keeping a set of containers…

"Now we have a distributed monolith that requires 2x less developers to build and 5x more system engineers to deploy"

deploying: `kubectl apply -f file_with_changed_docker_image.yaml`

Re: A story about a Kubernetes migration

#50

Post by user5994461 » Tue, Oct 02, 2018, 6:03 PM UTC

Facepalm. This entire blog post reads like they didn't figure out how to deploy to more than one server with ansible, while ansible is made just for that.