Earlier quoted context omitted.
> I have never seen an Ops team being rewarded for avoiding incidents That's why their salaries are so high.
Ops salaries are high??? Where?!?!
Definitely commands better salaries than us pitty DEs.
Cloudflare outage on December 5, 2025
141–150 of 601 posts
Re: Cloudflare outage on December 5, 2025
#142What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…
The question is perhaps what the shape and status of their tech stack is. Obviously, they are running at massive scale, and they have grown extremely aggressively over the years. What's more, especially over the last few years, they have been adding new product after new product. How much tech debt have they accumulated with that "move fast" approach that is now starting to rear its head?
I think this is probably a bigger root cause and is going to show up in different ways in future. The mere act of adding new products to an existing architecture/system is bound to create knowledge silos around operations and tech debt. There is a good reason why big companies keep smart people on their payroll to just change couple of lines after a week of debate.
Re: Cloudflare outage on December 5, 2025
#143Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…
> I truly believe they're really going to make resilience their #1 priority now I hope that was their #1 priority from the very start given the services they sell... Anyway, people always tend to overthink about those black-swan events. Yes, 2 happened in a quick succession, but what is the average frequency overall? Insignificant.
[deleted]
Re: Cloudflare outage on December 5, 2025
#144[deleted]
Re: Cloudflare outage on December 5, 2025
#145Earlier quoted context omitted.
It's not about outages. It's about the why. Hardware can fail. Bugs can happen. But to continue a roll out despite warning sings and without understanding the cause and impact is on another level. Especially if it is related to the same problem as last time.
[dead]
It is healthy for tech companies to have outages, as they will build experience in resolving them. Success breeds complacency.
Re: Cloudflare outage on December 5, 2025
#146I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…
it can't even spy on us silently, damn
Re: Cloudflare outage on December 5, 2025
#147This is total amateur shit. Completely unacceptable for something as critical as Cloudflare.
Re: Cloudflare outage on December 5, 2025
#148Is there some underlying factors that resulted in the recent outages (e.g., new processes, layoffs, etc.) or just a series of pure coincidences?
Sounds like their "FL1 -> FL2" transition is involved in both.
It was involved in the previous one, but not in this latest one. All FL2 did was prevent the outage being even wider spread than it was. None of this had anything to do with migration.
Re: Cloudflare outage on December 5, 2025
#149> This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur. Cloudflare deployed code that was literally never tested, not even once, neither manually nor by unit test, otherwise the straightforward error wo…
Large scale infrastructure changes are often by nature completely untestable. The system is too large, there are too many moving parts to replicate with any kind of sane testing, so often, you do find out in prod, which is why robust and fast rollback procedures are usually desirable and implemented.
> Large scale infrastructure changes are often by nature completely untestable.
You're changing the subject here and shifting focus from the specific to the vague. The two postmortems after the recent major Cloudflare outages both listed straightforward errors in source code that could have been tested and detected.
Theoretical outages could theoretically have other causes, but these two specific outages had specific causes that we know.
> which is why robust and fast rollback procedures are usually desirable and implemented.
Yes, nobody is arguing against that. It's a red herring with regard to my point about source code testing.
Re: Cloudflare outage on December 5, 2025
#150I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…
You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?