Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

161–170 of 335 posts

Re: Stack Overflow Outage Postmortem

#161

Post by ghusbands »

rhizome wrote: post #127
smegel wrote: post #70

> If the string to be matched against contains 20,000 space characters in a row, but not at the end, then the Regex engine will start at the first space, check that it belongs to the \s character class, move to the second space, make the same check, etc. After the 20,000th space, there is a different character, but the Regex engine expected a space or the end of the string. Realizing it cannot match like this it back…

Is there a difference between greedy and non-greedy atoms?

In the order of searching and hence the match you can get, yes. In performance in the case of a non-match, no.

Re: Stack Overflow Outage Postmortem

#162

Post by davidron »

The whole postmortem focuses on a regular expression bug and how that bug was fixes and completely ignores the fact that if the home page becomes unavailable, the load balancer logic will shut down the entire site.

Re: Stack Overflow Outage Postmortem

#163

Post by mioelnir »

chiph wrote: post #28
redbeard0x0a wrote: post #13

In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?

I think you've got the right approach - a vertical slice through the app that checks every layer. You want to know if a user can get useful info from your site, and it tracks (separately!) the common path their query would follow. The danger is that the endpoint becomes public knowledge and comes under a DDOS attack. Putting an IP address filter on that endpoint is usually enough to stop that.

> Putting an IP address filter on that endpoint is usually enough to stop that.

Why is there even direct external IP connectivity to the realserver, sidestepping the loadbalancer?

Re: Stack Overflow Outage Postmortem

#164

Post by davidron »

The whole postmortem focuses on a regular expression bug and how that bug was fixes and completely ignores the fact that if the home page becomes unavailable, the load balancer logic will shut down the entire site.

Re: Stack Overflow Outage Postmortem

#165

Post by OJFord »

zymhan wrote: post #45
Someone1234 wrote: post #36

Earlier quoted context omitted.

Well in this case a post contained 20K whitespaces, so I wouldn't jump to the conclusion that it was a mistake rather than intentional.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

It was in a multiline code block, so it just had a tonne of horizontal scroll.

See the edit: http://stackoverflow.com/revisions/38484433/2

Re: Stack Overflow Outage Postmortem

#166

Post by lambda »

Hmm. I wonder why one of the followup mitigations is not to move to a non-backtracking regex engine by default.

Most of what you want to do with a regex can be done with an NFA or DFA based engine. That which can't be done with an NFA or DFA based engine is generally better handled with a parser than a regex.

There are plenty of good DFA based regex matchers out there; RE2, the Rust regex crate, GNU grep, etc. At a glance, it even looks like glibc uses a DFA, though it supports POSIX REs which support backreferences so it must use backtracking at least for REs that contain backreferences.

Predictable hash collisions were a big sources of DOS attacks in web scripting languages which use tables a lot, until they started rolling out randomized hashing algorithms to prevent easily predictable hash collisions. It seems like it would be best for languages and libraries to move to DFA based regexps, at least for anything that doesn't contain backreferences, to mitigate these kinds of issues from being easy to exploit.

Re: Stack Overflow Outage Postmortem

#167

Post by burntsushi »

mwpmaybe wrote: post #160
burntsushi wrote: post #156

Earlier quoted context omitted.

You can see in my other post that this doesn't always work. For example, Python's regex engine chokes on `\s+$` if you use `re.search`, but works fine if you use `re.match`. It's likely that Perl has an optimization for `\s+$` but not `^\s+|\s+$` (the former regex only ever matches at the end of the string, which is amenable to optimizations).

I did see your other post, and upvoted it. This rule of thumb has served me well between different regex dialects and implementations, but it's not surprising that there are some specific cases that are "broken" for lack of a better word. I haven't done much Python but the documentation for re.search() and re.match() is very clear: use search to find an expression anywhere in a string, use match to find an expression…

Anchors aren't ignored. For re.match, `^abc$` is equivalent to `abc`, so the anchors are just redundant. (N.B. `^^abc$$` will match the same set of strings as `^abc$`.) For re.search, `^abc$` is equivalent to `re.match(^abc$)` but `abc` is equivalent to `re.match(.STAR?abc.STAR?)`.

But yes, this is a very subtle semantic and different regex engines handle it differently. My favorite approach is to always use `.STAR?theregex.STAR?` and don't provide any additional methods. Instead, if the user wants to match from the beginning to the end, they just need to use the well known anchors. (Go's regexp package does this, and Rust borrowed that API choice as well.)

Re: Stack Overflow Outage Postmortem

#168

Post by OJFord » 

    > It took 10 minutes to identify the cause,
Impressive, considering:

    > cause was a malformed post that caused one of our
    > regular expressions to consume high CPU ... called on
    > each home page view ... Since the home page is what our
    > load balancer uses for the health check, the entire site
    > became unavailable since the load balancer took the
    > servers out of rotation.

Re: Stack Overflow Outage Postmortem

#170

Post by nxzero »

xirdstl wrote: post #149
zymhan wrote: post #45

Earlier quoted context omitted.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

Cat laying its head on the spacebar. Mine does that on the Control key all the time.

At least it wasn't a squirrel... http://www.citylab.com/design/2016/01/map-squirrel-cyberatta...
Post reply on HN