What we learned after a year on Kubernetes • Hacker News Board

Re: What we learned after a year on Kubernetes

#141

Post by cbushko » Thu, Sep 17, 2020, 2:25 PM UTC

dpc_pw wrote: post #4

ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

This is why I use Terraform for my kubernetes objects. Yes, it is a DSL and not perfect but it has many advantages over helm such as:

- State. I cannot express how useful having the desired and actual state of a system is. It makes it so much easier to determined what has changed in between terraform applies which in turn makes tracking down problems easier.

- Providers. Being able to chain resources together from other systems is very powerful. As an example, I can:

1) create a database with a random name 2) create a user/password for that database 3) save the data above as a kubernetes secret 4) pass that secret into my kubernetes container for the service to use.

And you can easily change database to be anything such as a storage bucket, dns entry, twilio account, stripe account, etc.

Now try and do the exact same thing with helm!

- Providers Part 2: You can write your own providers to talk to your OWN services. I have done this and written up some go code to talk to our internal APIs when setting up new customers. It is very easy to create your own terraform resources to do this.

There are more reasons but I have work to do :)

Re: What we learned after a year on Kubernetes

#142

Post by selfhoster11 » Thu, Sep 17, 2020, 2:49 PM UTC

Keycap wrote: post #72

pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

Technologies don't last long. Another new tech comes out that does the job better or more cheaply, and the old tech either dies or competes.

I think that K8s is much too complicated for medium-scale workloads, so as soon as something makes it easier than K8s to scale and manage workloads at that scale, K8s will be out of that space.

Re: What we learned after a year on Kubernetes

#143

Post by t3rabytes » Thu, Sep 17, 2020, 3:45 PM UTC

GauntletWizard wrote: post #8

t3rabytes wrote: post #2

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct…

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection…

> It's so worth it, though.

It was until it caused a production outage, then it became very not worth it.

Re: What we learned after a year on Kubernetes

#144

Post by AaronFriel » Thu, Sep 17, 2020, 4:20 PM UTC

gscho wrote: post #17

AaronFriel wrote: post #9

Earlier quoted context omitted.

Are you familiar with Pulumi? It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch. It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as sa…

...and you need to use their SaaS to make it work, no?

No, just like Terraform works with different storage backends, theirs does too.

Re: What we learned after a year on Kubernetes

#145

Post by rsanders » Thu, Sep 17, 2020, 5:51 PM UTC

GauntletWizard wrote: post #8

t3rabytes wrote: post #2

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct…

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection…

Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves".

Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection.

We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.

Re: What we learned after a year on Kubernetes

#146

Post by smt88 » Thu, Sep 17, 2020, 6:03 PM UTC

Keycap wrote: post #103

smt88 wrote: post #76

Earlier quoted context omitted.

"out of the box" is a huge stretch. So is "simple". > Is it young? Yes. Do we need more people with more expierence? yes. Of course this is a problem! It's costly (in time, money, and security) to pay your team to ramp up on Kubernetes. The question is, what is the actual benefit? 99% of companies don't need it at all.

The features i mentioned, are solved 'out of the box'. I'm not seeing a huge stretch here at all. Its costly and risky to run 100 VMs, maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed. It is cost ineffective to have security audits on 100 VMs, maintaining access to them, auditing whats happening on them. It is a ton easier to allow someone only access to one nam…

> Its costly and risky to run 100 VMs...

A tiny fraction of companies need to run 100+ VMs. It takes a lot of traffic to require that scale, unless you're just throwing money at infra to avoid optimizing your code (which can be cost effective).

A large chunk of companies should just be on something like SquareSpace or Shopify, another large chunk should be using off-the-shelf services like Azure's App Containers or Amazon's Elastic Beanstalk/Lightsail/whatever else they have now, and some internal services can easily run on a single VM with no orchestration at all.

All that stuff is really expensive at scale, but unless you're a large company, your engineering time is going to cost way more than just using whatever managed services your cloud offers.

> ...maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed.

This is not the only alternative to using Kubernetes. You can automate all of this without Kubernetes, or you can use a cloud provider's managed Kubernetes service. A lot of companies get by just fine with Heroku.

If you assume people are using practices from 2005 (or that cloud providers don't already provide a layer of abstraction on top of Kubernetes), of course Kubernetes looks better.

Re: What we learned after a year on Kubernetes

#147

Post by GauntletWizard » Thu, Sep 17, 2020, 6:43 PM UTC

rsanders wrote: post #145

GauntletWizard wrote: post #8

Earlier quoted context omitted.

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection…

Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the n…

You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown.

Here's the annotation that I used to fix that:

    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30

Re: What we learned after a year on Kubernetes

#148

Post by wronglebowski » Thu, Sep 17, 2020, 7:44 PM UTC

cbushko wrote: post #139

pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

It must not fit your use case and that is ok. Kubernetes is made for the people that have to run your systems and keep them operational (ops/devops). It makes our lives a billion times easier on two fronts. Disclaimer, I am referring to a managed solution like GKE, I can imagine running k8s on your own hardware would be harder. Uptime & Stability: autoscaling, health checks, replacing nodes, restarting containers. Th…

I'd like to hear more on troubleshooting with a managed k8s platform and how you diagnose such issues.

My organization is playing with EKS and while perhaps it is different from GKE I feel like I have minimal visibility into if the control plane is healthy. I've had several instances where all of my node groups have become unhealthy for various reasons and EKS failed to spin up new nodes to resolve the issue.

Re: What we learned after a year on Kubernetes

#149

Post by pjmlp » Thu, Sep 17, 2020, 7:54 PM UTC

kkapelon wrote: post #115

pjmlp wrote: post #63

Earlier quoted context omitted.

VMs as usual.

Have you ever worked at a company that has more than 300 VMs that run more than 10 different apps? Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is a fad.

A small company called Nokia Networks, on a tiny product for telecommunication cluster monitoring called NetAct.

Then there was that tiny physics research center close to Geneva creating what was one of the first grid computing platforms.

But what do I know, that was almost 15 years ago.

Re: What we learned after a year on Kubernetes

#150

Post by ChicagoDave » Thu, Sep 17, 2020, 8:38 PM UTC

auspex wrote: post #127

ChicagoDave wrote: post #95

My question. Why not convert the rails code/APIs to blob/FaaS and skip all the DevOps complexity? Are containers providing enough long-term cost-effectiveness to a complete cloud-native application architecture? Or is it just mapping VMs to something “like” VMs so your topology remains mostly the same?

Serverless has its own complexity. Deploying a smaller, lightweight purpose built image, with no orchestration lock-in is the main benefit.

Those are fair infrastructure reasons, but not necessarily cost-effective. There are always competing perspectives. What does the business want. What do the developers want. What does the infrastructure team want. When power is centralized in one area, decisions can be made that overlook those competing purposes.

I see containers being adopted without openly discussing the options and long-term consequences.