My blog post[1] on how to test for catastrophic backtracking using RegEx buddy.
[1] https://blog.mozilla.org/webdev/2010/11/15/avoiding-catastro...
Stack Overflow Outage Postmortem
131–140 of 335 posts
Re: Stack Overflow Outage Postmortem
#132They implemented trim with a regex? Neither Java nor .NET do that. The postmortem here should probably be "why are you reimplementing trim".
Most likely answer: The built-in trim isn't Unicode aware or has buggy behavior when dealing with Unicode.
Assuming they're still using ASP.net 4+, it is very unicode aware/safe. I don't know why a developer would reinvent Trim() but I do know it isn't a .Net limitation.
Re: Stack Overflow Outage Postmortem
#133In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?
This is the right way. It's too easy to bork everything otherwise; yesterday I borked a new build server because I set security up to disallow anonymous access, which lead to the main page returning a 302 redirect to the login page and thereby failed the load balancer health check.
You live and learn.
Re: Stack Overflow Outage Postmortem
#134My rephrasing of their follow-up actions: * "Audit our regular expressions and post validation workflow for any similar issues" * ==> "Not even people who've worked for years on the guts of regex engines can easily predict the runtime of a given regex, but somehow our engineers will be expected to do that". * "Add controls to our load balancer to disable the healthcheck – as we believe everything but the home page wo…
Regex engines usually have an option to set upper limit on the time they take to match so they don't end up in expensive loops. A simple solution could be to setup such time limits.
Re: Stack Overflow Outage Postmortem
#135Google cache saved me during these 34 minutes.
Mac app Dash allows downloading & browsing Stack Overflow offline by language. 834 MB for Python.
Re: Stack Overflow Outage Postmortem
#136[deleted]
Re: Stack Overflow Outage Postmortem
#137Could this has been a deliberate/malicious act? Why else would someone post 20,000 consecutive characters of whitespace on a comment line? Also, the "homepage" of StackOverflow does not show any 'comments' - it is just the top questions? Why was the page loading any comments in the first place?
We don't think it was intentional. Maybe copy and paste, or something an editor did. To clarify the "comment" was not a Stack Overflow comment, but rather a comment in a code block inside a question.
May we see this code comment? Sounds interesting.
Re: Stack Overflow Outage Postmortem
#138Earlier quoted context omitted.
Well in this case a post contained 20K whitespaces, so I wouldn't jump to the conclusion that it was a mistake rather than intentional.
Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.
Bottle of tequila on the spacebar
Re: Stack Overflow Outage Postmortem
#139Yesterday I couldn't use hipchat for a couple hours because it would lock up a cpu and fail to load. After doing some free debugging for them I realized they were locking up trying to extract urls out of some text with a regex. Simplified code: https://gist.github.com/shanemhansen/c4e5580f7d4c6265769b0df...
Pasting that content into hipchat will probably lock up your browser and webview based clients. Beware.
Lesson learned: don't parse user input with a regex.
Re: Stack Overflow Outage Postmortem
#140For more bugs caused by quadratic complexity: