Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

251–260 of 335 posts

Re: Stack Overflow Outage Postmortem

#251

Post by ekimekim »

nickpsecurity wrote: post #210
chubot wrote: post #9

Ha! The same bug happened internally at my company. In that case it was a regex matching a URL taking so much CPU as to cause a DOS of a proxy server. I won't be surprised if it's happened to someone here too. This is very timely, because minutes ago, I made a link to Russ Cox's articles in my Kernighan awk repo: https://github.com/andychu/bwk https://swtch.com/~rsc/regexp/regexp1.html If you are not familiar with th…

There's a simple trick the real-time and high-assurance communities use to catch stuff like this: enforce a hard limit for time each task takes to complete. Some you might give leeway to prevent DDOS'ing your users. Many (most?) things have a sane, hard limit that can apply along with a log entry or notification to ops. Keeps one user or task from DDOSing whole system or forces it to fail fast + noticeably. Note a lo…

For the specific example, this is how I dealt with a situation in one service I wrote that takes an arbitrary user-controlled regex (for search).

The program was written in python, and as it turns out the GIL prevents a thread from being able to interrupt another thread that's doing a re.match() call - the entire regex engine is in C so the GIL is held for the entire call.

My solution was to use SIGALRM to set a timer before calling, and have the signal handler raise an exception. This causes the regex call to raise that exception, which I then catch and display an appropriate error.

Re: Stack Overflow Outage Postmortem

#252

Post by skeletonjelly »

Nick-Craver wrote: post #230
skeletonjelly wrote: post #199

Earlier quoted context omitted.

How large was the stack dump? I'm impressed you identified it this quickly considering the time it would take to get this to disk.

We have an internal tool where we can dump stack traces almost instantly by attaching to a running process. We can't open source it due to using Microsoft lab code which was never licensed itself. However, clrmd ( https://github.com/Microsoft/clrmd and https://www.nuget.org/packages/Microsoft.Diagnostics.Runtime ) means an open source version is hopefully on the horizon. As an example, here's the result of me piddlin…

Awesome. Thanks for the response, love your work Nick.

Re: Stack Overflow Outage Postmortem

#253

Post by rixed »

Regex was not the main issue. The main issues were:

1. Rendering a page fails/does not terminate if some non essential subtask (rendering a single code block) fails/does not terminate.

2. They do not try to detect bad data (the way they certainly try to detect bad code)

3. Load balancing based on the rendering time of a single page

Code bugs triggered by bad data will happen again, with or without regular expressions.

Re: Stack Overflow Outage Postmortem

#254

Post by randominion »

zymhan wrote: post #45
Someone1234 wrote: post #36

Earlier quoted context omitted.

Well in this case a post contained 20K whitespaces, so I wouldn't jump to the conclusion that it was a mistake rather than intentional.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

I can tell you how.

A shitty Belkin KVM in certain configurations can allow this to happen.

There's a bug which keeps generating chr(32) characters when you activate the keyboard shortcut (scroll lock twice), and try to switch to another machine.

It will keep pumping out those spaces on whatever fields was selected at the time, so if you take your time before you switch back, you are going to be in for a lot of fun.

Haven't read the link yet, but wanted to share this sooner than later.

Re: Stack Overflow Outage Postmortem

#255

Post by dingo_bat »

solipsism wrote: post #250
dingo_bat wrote: post #234

Earlier quoted context omitted.

From what I understood, trim would work perfectly. It's 200 spaces not a single 200 width character.

You're misreading the regex. \u200c is a single whitespace character. http://www.fileformat.info/info/unicode/char/200c/index.htm

But that's a weird character to put in a comment line! I don't get how this would happen accidentally.

Re: Stack Overflow Outage Postmortem

#256

Post by brongondwana »

Time to pop this old chestnut out:

https://blog.fastmail.com/2014/12/14/on-duty/

"At one stage, we decided to try to avoid having to be woken for some types of failure by using Heartbeat, a high availability solution for Linux, on our frontend servers. The thing is, our servers are actually really reliable, and we found that heartbeat failed more often than our systems - so the end result was reduced reliability! It's counter-intuitive, but automated high-availability often isn't."

One of these days we'll finish our new system and I'll blog about that, which is that the automated systems are allowed to take ONE corrective action without paging, at which point they flag that the system is in compromised state. Any further test failures trigger an immediate wake of the on-call.

Re: Stack Overflow Outage Postmortem

#258

Post by [deleted] »

burntsushi wrote: post #167
mwpmaybe wrote: post #160

Earlier quoted context omitted.

I did see your other post, and upvoted it. This rule of thumb has served me well between different regex dialects and implementations, but it's not surprising that there are some specific cases that are "broken" for lack of a better word. I haven't done much Python but the documentation for re.search() and re.match() is very clear: use search to find an expression anywhere in a string, use match to find an expression…

Anchors aren't ignored. For re.match, `^abc$` is equivalent to `abc`, so the anchors are just redundant. (N.B. `^^abc$$` will match the same set of strings as `^abc$`.) For re.search, `^abc$` is equivalent to `re.match(^abc$)` but `abc` is equivalent to `re.match(.STAR?abc.STAR?)`. But yes, this is a very subtle semantic and different regex engines handle it differently. My favorite approach is to always use `.STAR?t…

[deleted]

Re: Stack Overflow Outage Postmortem

#259

Post by gbrayut »

burntsushi wrote: post #106
barrkel wrote: post #103

Earlier quoted context omitted.

FWIW, the conversion from NFA (non-deterministic, i.e. backtracking) to DFA (deterministic, linear time) can take exponential space. So there's another avenue for DDOS; it's a lot harder to exploit, though, because it requires the attacker to control the input (i.e. regular expression) to the NFA->DFA transformation, rather than merely provide a string that takes a long time for an NFA to recognize.

I'm actually not aware of any popular DFA based engines that suffer from this vulnerability. grep and RE2, for example, build the DFA lazily and cap the size of the DFA such that matching is still linear time even if generating the full DFA would take exponential space. (This is because at most one DFA state is generated for each byte in the input, so technically, you only ever need space for one or two states. In pr…

Random side note... Recognized your name, we use your TOML go package in a bunch of our tools. Thanks and keep up the great work!

Re: Stack Overflow Outage Postmortem

#260

Post by gbrayut »

Someone1234 wrote: post #132
jandrese wrote: post #53

Earlier quoted context omitted.

Most likely answer: The built-in trim isn't Unicode aware or has buggy behavior when dealing with Unicode.

Assuming they're still using ASP.net 4+, it is very unicode aware/safe. I don't know why a developer would reinvent Trim() but I do know it isn't a .Net limitation.

This code was written 5 years ago, and back then the trim function was different.
Post reply on HN