51–60 of 953 posts
28M 500 errors/sec for several hours from a single provider. Must be a new record. No other time in history has one single company been responsible for so much commerce and traffic. I wonder what some outage analogs to the pre-internet ages would be.
Earlier quoted context omitted.
Probably because this case was something more akin to an assert than an error check.
You are saying this would not have happened in a C release build where asserts define to nothing? Wonder why these old grey beards chose to go with that.
Afaik, Go and Java are the only languages that make you pause and explicitly deal with these exceptions.
I don't use Rust, but a lot of Rust people say if it compiles it runs.
Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out.
end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually wanna try it for industrial robot stuff or small controllers etc.
there's no bad language - just occassional hiccups from us users who use those tools.
"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…
Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
Probably because this case was something more akin to an assert than an error check.
Surely a unwrap_or_default() would have been a much better fit--if fetching features fails, continue processing with an empty set of rules vs stop world.
As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.
Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
> a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system ... to keep [that] system up to date with ever changing threats > The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail A configuration error can cause internet-scale outages. What an era…
This wild `unwrap()` kinda took me aback as well. Someone really believed in themselves writing this. :)
> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
Earlier quoted context omitted.
You are saying this would not have happened in a C release build where asserts define to nothing? Wonder why these old grey beards chose to go with that.
> You are saying this would not have happened in a C release build where asserts define to nothing? Afaik, Go and Java are the only languages that make you pause and explicitly deal with these exceptions.