851–860 of 953 posts
Earlier quoted context omitted.
They only recently rewrote their core in Rust ( https://blog.cloudflare.com/20-percent-internet-upgrade/ ) -- given the newness of the system and things like "Over 100 engineers have worked on FL2, and we have over 130 modules" I won't be surprised for further similar incidents.
The irony of a rust rewrite taking down the internet is not lost on me.
A fucking unhandled exception brought down a majority of the internet? Why do we continue to let these clowns run a large portion of the internet? Big tech is a fucking joke.
this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about
How about 1. The permissions change project is paused or rolled back until 2. All impacted database interactions (SQL queries) are evaluated for improper assumptions or better 3. Their design that depends on database metainfo and schema is replaced with ones that use specific tables and rows in tables instead of using the meta info as part of their application. 4. All hard coded limits are centralized in a single global module and referenced from their users and then back propagated to any separate generator processes that validate against the limit before pushing generated changes
Earlier quoted context omitted.
So what happens if it ends up being nil? How does your app react? In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace. Silent failure is ultimately terrible for users. Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rath…
> Silent failure is ultimately terrible for users. Agreed. Unfortunately, crashes in iOS are “silent failures,” and are a loss of control. What this practice does, is give me the option to handle the failure “noisily,” and in a controlled manner; even if just emitting a log entry, before calling a system failure. That can be quite helpful, in threading. Also, it gives me the option to have a valid value applied, if t…
Earlier quoted context omitted.
Fair points; there's certainly a balance to be struck between raising awareness and amplifying, and I admittedly have no idea where that line is.
I'm not the person you were replying to, but there is a rule I often see about not directly replying/quote tweeting because "engagement" appears to boost support for the ideas expressed. The recommendation then, would be to screenshot it (often with the username removed) and link to that.
Posts like that deserve to be flagged if the sum of their substance is jingoist musing & ogling dumb people on Twitter.
> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…
> much better than their completely false “checking the security of your connection” message The exact wording (which I can easily find, because a good chunk of the internet gives it to me, because I’m on Indian broadband): > example.com needs to review the security of your connection before proceeding. It bothers me how this bald-faced lie of a wording has persisted. (The “Verify you are human by completing the acti…
Earlier quoted context omitted.
> Silent failure is ultimately terrible for users. Agreed. Unfortunately, crashes in iOS are “silent failures,” and are a loss of control. What this practice does, is give me the option to handle the failure “noisily,” and in a controlled manner; even if just emitting a log entry, before calling a system failure. That can be quite helpful, in threading. Also, it gives me the option to have a valid value applied, if t…
Crashes are silent failures but as I mentioned: you can get a lot of your crashes reported via the App Store. This is why I prefer crashes in this situation: it gives me something actionable over silent failures on the client.
Also, I have found App Store crash reports to be next to useless. TestFlight ones are a bit better.
But if I spend a lot of time, doing it right, the first time, we can avoid all kinds of heartbreak.
Earlier quoted context omitted.
Thank you. I am sympathetic to CF’s need to deploy these configs globally fast and don’t think slowing down their DDoS mitigation is necessarily a good trade off. What I am saying is this presents a bigger reliability risk and needs correspondingly fine crafted observability around such config changes and a rollback runbook. Greater risk -> greater attention.
But the rapid deployment mechanism for bot features wasn’t where the bug was introduced. In fact, the root bug (faulty assumption?) was in one or more SQL catalog queries that were presumably written some time ago. (Interestingly the analysis doesn’t go into how these erroneous queries made it into production OR whether the assumption was “to spec” and it’s the security principal change work that was faulty. Seems mo…
Generally I would say we as an industry are more nonchalant about config changes vs binary changes. Where an org might have great processes and systems in place for binary rollouts, the whole fleet could be reading config from a database in a much more lax fashion. Those systems are quite risky actually.
Earlier quoted context omitted.
I don't understand why they didn't validate and sanitize the new config file revision. If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.
> I don't understand why they didn't validate and sanitize the new config file revision. The new config file was not (AIUI) invalid (syntax-wise) but rather too big : > […] That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. > The software running on these machines to route traffic across our network reads this feature fi…