41–50 of 159 posts
Note: Golang's stdlib regexp ( https://golang.org/pkg/regexp/ ) is guaranteed to run in time linear to the size of input. Russ Cox has a detailed article https://swtch.com/~rsc/regexp/regexp1.html
Golang's regexp is derived from RE2, referenced in footnote four of the post.
New idea: scan github for regexes that exhibit unbounded backtracking and generate worst case examples.
Always appreciate the transparency from you and Cloudflare. :) My main fright during this outage wasn't really the outage itself, but the fact that I couldn't log into the dashboard and simply click the orange cloud to bypass Cloudflare in the meantime. I'm assuming that this is now covered by this mitigation: >> 6. Putting in place an emergency ability to take the Cloudflare Dashboard and API off Cloudflare's edge.…
I've always been a proponent of separating the monitoring from the infra. Otherwise your insight is binary: the service is either up or down. You don't have an context as to why. Edit: Additionally, from a competitive standpoint, I don't see a problem with using a third-party platform for a monitoring service.
Incredible write up. Really appreciate the detail, and am really impressed by how mature their response coordination seems to be.
So in response to a catastrophic failure due to testing in prod, they're going to push out a brand new regex engine with an ETA of 2 weeks. Can anyone say testing in prod? The constant use of 'I' and 'me' (19 occurrences in total) deeply tarnishes this report, and repeatedly singling out a responsible engineer, nameless or not, is a failure in its own right. This was a collective failure, any individual identity is t…
This report is written by me, the CTO of Cloudflare. I say "I" throughout because organizational failings are my responsibilty. If I'd said "we" I imagine you'd be criticizing me for NOT taking responsibility. If you read the report you'd see I do not blame the engineer responsible at all. Not once. I made that perfectly clear.
I don't mean this as some sort of lame 'lol shoulda known better' dunk - stories about technical organizations' decision-making and tradeoff-handling are just more interesting than the details of how regexes typed in a control panel grow up to become Jira tickets.
Suggestion for future, learned from bitter experience: separate your control plane from your data plane. In this case, make sure that the tools you use to manage your infrastructure don't depend on that infrastructure being functional.
That way you won't have to remember how to use a bypass procedure -- it will just be your normal procedure.
Always appreciate the transparency from you and Cloudflare. :) My main fright during this outage wasn't really the outage itself, but the fact that I couldn't log into the dashboard and simply click the orange cloud to bypass Cloudflare in the meantime. I'm assuming that this is now covered by this mitigation: >> 6. Putting in place an emergency ability to take the Cloudflare Dashboard and API off Cloudflare's edge.…
> If so, and if this would have prevented the dashboard outage even during the WAF fiasco
It wouldn't prevent the initial dashboard outage. However, in a similar situation where the main issue can't be resolved quickly, it would allow them to restore dashboard access.