If it was in a comment, why was the home page loading it?
preemptive caching?
Stack Overflow Outage Postmortem
171–180 of 335 posts
Re: Stack Overflow Outage Postmortem
#172Earlier quoted context omitted.
Trim would not have worked, the post started with '-- play happy sound for player to enjoy', had 20000 characters of whitespace, and then some other character.
Which also wouldn't have been caught by the regex, because it was designed to do the same as Trim. Given the input, all that whitespace was not actually supposed to be removed and the regex worked, it was just a degenerate case that slowed it down to a crawl. Trim would not slow down on this input.
Trim won't work, otherwise SO would be using trim. The regex did work in all cases, but it was slow under certain conditions. The slowness caused a separate system to shutdown the site.
The regex worked fine, 100% within expectations.
Re: Stack Overflow Outage Postmortem
#173Earlier quoted context omitted.
Any idea if it was a malicious attempt? It kind of sounds like it was.
Nah, https://en.wikipedia.org/wiki/Hanlons_razor If it was malicious they would have made a bunch of them, not just one. I personally have seen many files with unreal amounts of whitespace at the end.
Yeah, that's kinda what we're thinking. If it were malicious, the question itself would be unlikely to actually look like a real honest question, too. It'd just be a bunch of garbage input.
Re: Stack Overflow Outage Postmortem
#174Any more proof needed that caching should become a system-provided service over the next 10-20 years, the same way memory management did in the past 10-20 years?
Re: Stack Overflow Outage Postmortem
#175A few months ago, a Stack Overflow representative asked me if their presence at a dev conference was justified. My positive answer more or less revolved around the importance SO took in the daily life of programmers everywhere. If only she was there to witness the effect of a 34 minute downtime on an open space full of mobile/back/front developers.
Google's cached versions of SO answers are generally up to date, no? Nice try, lazy developers!
Re: Stack Overflow Outage Postmortem
#176My rephrasing of their follow-up actions: * "Audit our regular expressions and post validation workflow for any similar issues" * ==> "Not even people who've worked for years on the guts of regex engines can easily predict the runtime of a given regex, but somehow our engineers will be expected to do that". * "Add controls to our load balancer to disable the healthcheck – as we believe everything but the home page wo…
I don't think they're disabling health checks in general, just adding a way to temporarily disable checks when they are misbehaving.
Re: Stack Overflow Outage Postmortem
#177Earlier quoted context omitted.
Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.
Bottle of tequila on the spacebar
Eh, just type a single space.
Then copy the entire post and paste it a couple dozen times.
Re: Stack Overflow Outage Postmortem
#178Easy to reproduce [1]. Just remove the a in the end and your timeout disappears. Anybody knows which regex engine they used? [1] http://regexr.com/3drn3
I could not reproduce the slowness in Perl, but I could in JavaScript (specifically node.js).
There are many, many modules that use /\s+$/, including at least one form validator.
Re: Stack Overflow Outage Postmortem
#179Earlier quoted context omitted.
I think you've got the right approach - a vertical slice through the app that checks every layer. You want to know if a user can get useful info from your site, and it tracks (separately!) the common path their query would follow. The danger is that the endpoint becomes public knowledge and comes under a DDOS attack. Putting an IP address filter on that endpoint is usually enough to stop that.
> Putting an IP address filter on that endpoint is usually enough to stop that. Why is there even direct external IP connectivity to the realserver, sidestepping the loadbalancer?
Not all load balancers function at the same OSI layer. If you are using an IP based load balancer with transparent NAT, you don't really have a choice. Requests will be balanced across systems to that health check, but it's still a DOS if it's intensive enough to serve and someone hammers it hard enough.
Re: Stack Overflow Outage Postmortem
#180Earlier quoted context omitted.
The key quote here is: "Regular expressions are one of computer science's shining examples of how using good theory leads to good programs ..." "Today, regular expressions have also become a shining example of how ignoring good theory leads to bad programs. The regular expression implementations used by today's popular tools are significantly slower than the ones used in many of those thirty-year-old Unix tools." The…
FWIW, the conversion from NFA (non-deterministic, i.e. backtracking) to DFA (deterministic, linear time) can take exponential space. So there's another avenue for DDOS; it's a lot harder to exploit, though, because it requires the attacker to control the input (i.e. regular expression) to the NFA->DFA transformation, rather than merely provide a string that takes a long time for an NFA to recognize.
Only if you want to model it with every combination of states from the nfa getting one from the dfa. If you don't mind describing the current state as a list of potential states from the nfa, the conversion is linear space.