Kubernetes Failure Stories • Hacker News Board

Re: Kubernetes Failure Stories

#11

Post by An0mammall » Thu, Feb 11, 2021, 8:48 PM UTC

I first thought this was kind of a smear attempt but I realise now that this is a great learning ressource, thanks!

Re: Kubernetes Failure Stories

#12

Post by kureikain » Thu, Feb 11, 2021, 8:49 PM UTC

The first link in this article is about switching from fluentbit to fluentd.

Me too. I have issue with fluenbit. Especially around their JSON processing. As in this issue: https://github.com/fluent/fluent-bit/issues/1588

If you want a simple log forwarding, fluentbit is really good. But if you found yourself started to tweak fluentbit config, or write custom plugin in C and (recompile yourself) then it's time to move to fluentd.

Re: Kubernetes Failure Stories

#13

Post by bo0tzz » Thu, Feb 11, 2021, 8:57 PM UTC

superbcarrot wrote: post #10

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

https://github.com/kubernetes-sigs/cluster-api

Re: Kubernetes Failure Stories

#14

Post by bcrosby95 » Thu, Feb 11, 2021, 8:59 PM UTC

superbcarrot wrote: post #10

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

A Kubernetes for Kubernetes! But what about when we want to upgrade that?

Re: Kubernetes Failure Stories

#15

Post by Benlights » Thu, Feb 11, 2021, 9:06 PM UTC

I see you read the article about litestream, I love following the rabbit hole. Good work!

Re: Kubernetes Failure Stories

#16

Post by thirdlamp » Thu, Feb 11, 2021, 9:07 PM UTC

superbcarrot wrote: post #10

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

it's kubernetes all the way down

Re: Kubernetes Failure Stories

#17

Post by codeulike » Thu, Feb 11, 2021, 9:07 PM UTC

superbcarrot wrote: post #10

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

Kubermeta

Re: Kubernetes Failure Stories

#18

Post by mullingitover » Thu, Feb 11, 2021, 9:10 PM UTC

superbcarrot wrote: post #10

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

Rancher does this nicely.

Re: Kubernetes Failure Stories

#19

Post by opsunit » Thu, Feb 11, 2021, 9:12 PM UTC

kureikain wrote: post #12

The first link in this article is about switching from fluentbit to fluentd. Me too. I have issue with fluenbit. Especially around their JSON processing. As in this issue: https://github.com/fluent/fluent-bit/issues/1588 If you want a simple log forwarding, fluentbit is really good. But if you found yourself started to tweak fluentbit config, or write custom plugin in C and (recompile yourself) then it's time to move…

I did a lot of the work to get sub-second timestamps working within the fluentd+ElasticSearch ecosystem and the thing is a tire fire.

Re: Kubernetes Failure Stories

#20

Post by YawningAngel » Thu, Feb 11, 2021, 9:18 PM UTC

superbcarrot wrote: post #10

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region