Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
1–10 of 32 posts
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#2This can be one reason to run the control plane not on k8s itself. When the control plane runs on k8s you can get these weird states where the control plane is borked and the system cannot recover.
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#3I appreciate the transparency and detail in publishing this. With that said, the narrative style and wordy,casual language makes it harder to get to the meat (the five whys) than a typical postmortem.
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#4I appreciate the transparency and detail in publishing this. With that said, the narrative style and wordy,casual language makes it harder to get to the meat (the five whys) than a typical postmortem.
The intended audience is probably a mix of engineers and regular Reddit users, hence the more casual tone.
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#5> In the 1.20 series, Kubernetes changed its terminology from “master” to “control-plane.” And in 1.24, they removed references to “master,” even from running clusters. This is the cause of our outage. Kubernetes node labels.
Wow, so the word police brought down Reddit. Why on earth did someone think it a good idea to screw with existing names in running clusters in a cluster management tool?
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#6> In the 1.20 series, Kubernetes changed its terminology from “master” to “control-plane.” And in 1.24, they removed references to “master,” even from running clusters. This is the cause of our outage. Kubernetes node labels. Wow, so the word police brought down Reddit. Why on earth did someone think it a good idea to screw with existing names in running clusters in a cluster management tool?
[deleted]
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#7This can be one reason to run the control plane not on k8s itself. When the control plane runs on k8s you can get these weird states where the control plane is borked and the system cannot recover.
I always find this sort of dogfooding to be academically clever, but operationally risky.
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#8This can be one reason to run the control plane not on k8s itself. When the control plane runs on k8s you can get these weird states where the control plane is borked and the system cannot recover.
Back when we built our own Kubernetes distribution around the Kube 1.6 era I had to fight really hard with our architect to let me run the control plane with systemd instead of within Kube. The extra nodes were considered to be “a waste of resources”.
But in the five or so years we ran that distro the control plane didn’t fail once. Posts like this make me glad I pushed for it.
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#9This can be one reason to run the control plane not on k8s itself. When the control plane runs on k8s you can get these weird states where the control plane is borked and the system cannot recover.
Back when we built our own Kubernetes distribution around the Kube 1.6 era I had to fight really hard with our architect to let me run the control plane with systemd instead of within Kube. The extra nodes were considered to be “a waste of resources”. But in the five or so years we ran that distro the control plane didn’t fail once. Posts like this make me glad I pushed for it.
Technically it already runs kinda “outside of the loop” using static/mirrored pods so it doesn’t go through scheduler assignment/kcm reconciliation loop. If they ran their reflectors that way it probably wouldn’t happen
Re: Reddit Releases Post Mortem for Its 3 Hour Outage Last Week
#10> In the 1.20 series, Kubernetes changed its terminology from “master” to “control-plane.” And in 1.24, they removed references to “master,” even from running clusters. This is the cause of our outage. Kubernetes node labels. Wow, so the word police brought down Reddit. Why on earth did someone think it a good idea to screw with existing names in running clusters in a cluster management tool?
The word police seem to have gone kinda quiet lately.
I wonder if they realised that banning a few words wasn't really helping their cause.