131–140 of 159 posts
Earlier quoted context omitted.
Yeah, that is true in most cases. However, here is Cloudflare was using Cloudflare on dash.cloudflare.com as well. This calls for not using Cloudflare for their web dashboard.
Then comes the inevitable tweets every few weeks. "cloudflare doesn't event trust cloudflare to run their own control panel"
One thing set of alarm bells in my head from an operational perspective: > Switching to either the re2 or Rust regex engine which both have run-time guarantees. (ETA: July 31) That's short timescales for quite a significant change. I know it's just replacing a piece of automation with one that does the same task, but the guts are all changing and all automation introduces some level of instability, and a bunch of unk…
Earlier quoted context omitted.
The idea is that the fix itself is being tested. If you knew your 'rollback' will work for certain, then you'd just deploy it to everyone asap. But since you don't, you test it and as a potential outcome of your test is no fix or making things worse, you don't test it on your highest-value customers. Imagine what your postmortem would read like if your fix made an even bigger mess.
Oh, I misunderstood you originally. I thought you said rollout from highest to lowest. You're actually saying lowest to highest.
What about: WAF cpu usage wasn't isolated from the ability to serve requests? This would allow requests that don't go throwugh WAF to be able to proceed as usual.
A firewall that fails open sounds like a terrible plan. As far as problems go, an outage is preferable to a breach.
That was a fantastic demonstration of what backtracking meant. Thank you John for your in depth description of what went wrong. As a follow up, would something like `[^=] =. ` be a better capture group regex?
Yes. I think HN stole your asterisks. You meant: /.*=.*/ becomes /[^=]*=.*/ That is, zero or more 'not-equals-sign-characters', followed by an equals sign. Where the first regex is 57 steps for x=xxxxxxxxxxxxxxxxxxxxxxxx, the second is just 7. Avoid using greedy .* for backtracking regex engines! Give your greedy regex engine the hints it needs to do what it does best.
I'm a relatively novice regex user. Could anyone explain to me why someone might use an expression like `. (?:. =.*)` ? What is the meaning of the group if it's boundary could be placed in any number of places? Hope that makes sense.
It can be useful depending on how the engine handles the match. The non-capturing group is the important part, the .* is just there so that the 'full match' isn't just an empty string, but contains the whole line the rule is being run against
Earlier quoted context omitted.
The main problem is that their Regex library doesn't have a recrusion limit. I'm honestly amazed they've been able to scale Lua scripts to the point they can use it as a global WAF. Knowing this, it may be easy to create attacks against their filters. My takeaway is that it's time to move to a custom solution using a more flexible language. A simple async watchdog on total rule execution time would have prevented thi…
I am wondering why you are being downvoted. This outage could have been prevented with better deployment procedures too. For example my company (nowhere near the scale of Cloudflare) does progressive deployments. New code is deployed only to a handful machines first, and then as the hours pass and checks remain green it propagates to the rest of the server fleet. Full deployment takes 24 hours. We never had code brea…
Recursion attacks against Regex are extremely well known. The only reason I can fathom for not having an execution time watchdog is that Nginx Lua runtime doesn't allow it. I assume the scripts run during a single request cycle on one thread due to Nginx async IO (one thread per core only).
That's still no excuse. They admit to running THOUSANDS of Regex rules in custom Lua scripts embedded in Nginx. This sounds like a bad idea to anyone that knows anything about software because it is.
My previous employer embedded way too much Lua script inside Nginx plugins for the same reasons (it's easy). Even at our "scale" (50 requests/second) we had constant issues. To think they run ~10% of internet traffic on such a rube Goldberg machine is proof you can use just about anything in prod (until it inevitably explodes at least)
Well, if I still worked on Hyperscan, this would be my "what am I, a potted plant?" moment. I think Cloudflare is pretty determined to avoid x86-only implementations of anything, though. It's entertaining to see people making the same mistakes that have been widely known about in network security well before there was Hyperscan, RE2, etc.
Hmmm...
That was a fantastic demonstration of what backtracking meant. Thank you John for your in depth description of what went wrong. As a follow up, would something like `[^=] =. ` be a better capture group regex?
Yes. I think HN stole your asterisks. You meant: /.*=.*/ becomes /[^=]*=.*/ That is, zero or more 'not-equals-sign-characters', followed by an equals sign. Where the first regex is 57 steps for x=xxxxxxxxxxxxxxxxxxxxxxxx, the second is just 7. Avoid using greedy .* for backtracking regex engines! Give your greedy regex engine the hints it needs to do what it does best.