101–110 of 137 posts
I really want to know the regexp and corresponding input(s) which killed the internet now :) Was it just aaaaaaaaaaaah? https://swtch.com/~rsc/regexp/regexp1.html
That's the one that took down Stack Overflow a few years ago https://stackstatus.net/post/147710624694/outage-postmortem-...
Earlier quoted context omitted.
If other bidders are irrational, then bid-sniping can work. It doesn't give others the opportunity to contemplate, "I've been out-bid, do I actually want this item more than I originally thought?"
And it's well-known since early eBay days that many bidders are irrational, including but not limited to competitive impulse to "win". Plus you sometimes have shill bidders. Sniping approximates sealed bids, with the highest-bidder the second-highest sealed bid amount or a small increment above it. (Unfortunately for eBay, that would tend to decrease their cuts, unless the appeal of the sealed bid format brings in su…
Earlier quoted context omitted.
> None of the major cloud vendors actually hit 99.99% uptime. None of them even promise that -- last time I checked, it was 99.95% for most of them.
AWS services have their own individual SLAs. Route53, in particular, has a 100% SLA: https://aws.amazon.com/route53/sla/ (To my knowledge, it's the only AWS service to promise 100%.)
Earlier quoted context omitted.
Good SLAs are also for engineers.
That's what SLOs are for
Nothing wrong with just using SLOs, but if you are a technical lead or senior engineer, you should have the big picture.
Good.
> Unfortunately, these WAF rules were deployed globally in one go and caused today’s outage.
Wow. This seems like a very immature operational stance.
Any deployment of any kind should be subject to minimum deployment safety, that they claim they have.
> At 1402 UTC we understood what was happening and decided to issue a ‘global kill’ on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic. That occurred at 1409 UTC.
Many large companies would have had automatic roll-back of this kind of change in less time than it took CloudFlare to (apparently) have humans decide to roll-back, and possibly before a single (usually not global) deployment had actually completed on all hosts/instances.
However, what is more concerning is that it seems you shouldn't rely on CloudFlare's "WAF Managed Rulesets" at all, since they seem to be willing to turn it off instead of correctly rolling back a bad deployment, which they only did > 43 minutes later:
> We then went on to review the offending pull request, roll back the specific rules, test the change to ensure that we were 100% certain that we had the correct fix, and re-enabled the WAF Managed Rulesets at 1452 UTC.
How were they not able to trivially roll back to the previous deployment?
>"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place."[1] [1] https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...
Earlier quoted context omitted.
Cloudflare was responsive and reasonable. Verizon was unreachable and deflected responsibility when they finally made a statement. And public shaming does often motivate companies to be more responsive to their customers.
AFAIK Cloudflare isn't in any way a "customer" of Verizon. Verizon doesn't owe Cloudflare any kind of response or devotion of resources. Verizon owes it's actual customers a resolution to their problem, which they gave . I'm not saying Verizon is perfect nor absolved of fault, but Cloudflare was/is not owed any kind of explanation or assistance by VZ, and it's absurd of CF to still be whining about that fact (as they…
Kind of similar to a homes association saying “hey that trash on your lawn affects your neighbor, clean it up!”
It’s true that they are not a customer but at that level what they do affects each other, and it’s better to resolve things civilly and privately instead of publicly on twitter.
If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…
Since the work involved in doing a regular expression match can depend largely on the input for non-trivial expressions, one fun case (probably not the one here, though) is that a user of your system could start using a pathological case input that no amount of standard testing (synthetic or replayed traffic, staging environments, production canaries) would have caught. Didn't take anything down, but did cause an ino…