521–530 of 560 posts
From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…
Earlier quoted context omitted.
They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…
Yea agree.. This is the same discussion point that came up last time they had an incident. I really don’t buy this requirement to always deploy state changes 100% globally immediately. Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions. Then if they really think something is…
Earlier quoted context omitted.
Nobody is being forced to use Cloudflare
Since everything is absolutely correct, no one forced it; they just provided a good, excellent solution for free, and consequently, the whole internet has gotten hooked on it. As they say, free cocaine causes harm. So, what are the alternatives? What options are there to protect against DDoS attacks and to make a website quickly accessible from different parts of the world? And at the same time, without paying a sky-…
That sums up my gripe with the vocal cloudflare haters. They will tell you all day long to move but every solution they push costs more time and money.
Earlier quoted context omitted.
They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…
For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout? Blue/green and temporarily ossify capacity? Regional?
- Push a version that enables new logic for 1% of traffic
- Continue rollout until 100%
Earlier quoted context omitted.
Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do m…
Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem the…
Sometimes the solution is to not let certain people do certain things which are risky.
Earlier quoted context omitted.
>rest assured there will be no available hands to keep the status page updated That's not how status pages if implemented correctly work. The real reason status pages aren't updated is SLAs. If you agree on a contract to have 99.99% uptime your status page better reflect that or it invalidates many contracts. This is why AWS also lies about it's uptime and status page. These services rarely experience outages accordi…
This is weird - at this level contracts are supposed to be rock solid so why wouldn't they require accurate status reporting? That's trivial to implement, and you can even require to have it on a neutral third-party like UptimeRobot and be done with it. I'm sure there are gray areas in such contracts but something being down or not is pretty black and white.
it really isn't. We often have degraded performance for a portion of customers, or just down for customers of a small part of the service. It has basically never happened that our service is 100% down.
Earlier quoted context omitted.
That's exactly my point. There should be no such thing as choosing to crash if you want reliable software. Choosing to crash is idiomatic in Rust but not in managed languages in which exceptions are the standard way to handle errors.
I am not a C# guy, but I wrote a lot of Java back in the day, and I can authoritatively tell you that it has so-called "checked exceptions" that the compiler forces you to handle. However, it also has "runtime exceptions" that you are not forced to handle, and they can happen any where and any time. Conceptually, it is the same as error versus panic in Rust. One such runtime exception is the notorious `java.lang.Null…
For trapping a bad data load it's as simple as:
try {
data = loadDataFile();
} catch (Exception e) {
LOG.error("Failed to load new data file; continuing with old data", e);
}
Earlier quoted context omitted.
So. I don't understand the 5 nines they promote. One bad day those nines are gone. So they next year you are pushing 2 nines.
Its just fabricated bullshit. It's how all the companies do it. 99.999% over a year is literally 5 minutes. Or under an hour in a decade, that's wildly unrealistic. Reddit was once down for a full day and that month they reported 99.5% uptime instead of 99.99% as they normally claimed for most months. There is this amazing combination of nonsense going on to achieve these kinds of numbers: 1. Straight up fraudulent i…
Earlier quoted context omitted.
> The culture clearly is not fit for their level of criticality I don't think anyone's is.
How often do you hear of Akamai going down and they host a LOT more enterprise/high value sites than Cloudflare. There's a reason Cloudflare has been really struggling to get into the traditional enterprise space and it isn't price.
Akamai was historically only serving enterprise customers. Cloudflare opened up tons of free plans, new services, and basically swallowed much of that market during that time period.