Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

91–100 of 335 posts

Re: Stack Overflow Outage Postmortem

#91

Post by burntsushi »

phasmantistes wrote: post #76
StavrosK wrote: post #29

I don't understand something: the regex expected a space character, followed by the end of the string. If the last character wasn't a space, this could never match. Why did the engine keep backtracking, even though it's easy to figure out that it could never match the regex?

Most simple regular expression evaluators are basically state machines. They hold a few variables: a) what part of the regex am I currently trying to match b) what point in the string am I currently starting at c) how much of the string has this piece of the regex consumed so far Then the state machine basically has three transitions: * if (b+c) terminally matches (a), increment both (a) and (b) and reset (c) * if (b…

If you are really using a state machine and every match in the regex is at the end of the haystack, then you can reverse the state machine and use the same algorithm you described in reverse. :-)

(Rust's regex engine does this.)

Re: Stack Overflow Outage Postmortem

#92

Post by benten10 »

johnwheeler wrote: post #24

ugh. i would've just sat there wondering WTF. then proceed to initiate daily backup recovery.

Haha, really!

Maybe I'm less-experienced (1.5 yrs), but it would have taken me significantly more than the 15 mins(was it?) that it took them to diagnose the problem.

Props to the team, and Thank All 0.33G Gods I was not on call!

Re: Stack Overflow Outage Postmortem

#93

Post by redbeard0x0a »

chiph wrote: post #28
redbeard0x0a wrote: post #13

In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?

I think you've got the right approach - a vertical slice through the app that checks every layer. You want to know if a user can get useful info from your site, and it tracks (separately!) the common path their query would follow. The danger is that the endpoint becomes public knowledge and comes under a DDOS attack. Putting an IP address filter on that endpoint is usually enough to stop that.

The concept I try to go for with that status check is 'Can this node connect to everything so it can successfully respond to http requests'. However my approach wouldn't identify an overloaded server, which might be a good thing if we need to scale up - taking down an overloaded server is just going to make the other servers that much more overloaded.

I'm aways up for hearing about other ways people solve health checks.

Re: Stack Overflow Outage Postmortem

#95

Post by jandrese »

grashalm wrote: post #49

Easy to reproduce [1]. Just remove the a in the end and your timeout disappears. Anybody knows which regex engine they used? [1] http://regexr.com/3drn3

I had a tough time reproducing it in Perl, although I think I managed to get it.

  % perl -v
  This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-gnu-thread-multi
  % ls -l testfile
  125380092 Jul 20 16:41 testfile
  % time ./regextest testfile
  ./regextest testfile  0.36s user 0.03s system 99% cpu 0.398 total
  (Remove the a at the end)
  % time ./regextest testfile
  ./regextest testfile  0.13s user 0.03s system 99% cpu 0.156 total
I had to make a much larger input file than I expected before I was able to really see the difference. Given the O(n^2) nature of the bug, I was expecting it to take much longer.

Re: Stack Overflow Outage Postmortem

#96

Post by AceJohnny2 »

smrtinsert wrote: post #60

"This regular expression has been replaced with a substring function." This should be the title of a book on software engineering.

    This regular expression has been replaced with a substring function.
God I wish all my bugs were this easy to fix and deploy

Re: Stack Overflow Outage Postmortem

#97

Post by surething »

Someone1234 wrote: post #36
dkopi wrote: post #6

Perfect. Awesome bug. Awesome Post Mortem. This was just fun to read. While this might have been caused by mistake - these types of bugs can be (and are) abused by hackers. https://www.owasp.org/index.php/Regular_expression_Denial_of... https://en.wikipedia.org/wiki/ReDoS The post also links to this video: https://vimeo.com/112065252

Well in this case a post contained 20K whitespaces, so I wouldn't jump to the conclusion that it was a mistake rather than intentional.

I think it'd be possible to inject this kind of thing into your code if you're just starting out with vim, aren't cognisant of all commands you invoke, and then copy/paste all code straight into a browser.

Re: Stack Overflow Outage Postmortem

#100

Post by jandrese »

xigency wrote: post #78
chubot wrote: post #9

Ha! The same bug happened internally at my company. In that case it was a regex matching a URL taking so much CPU as to cause a DOS of a proxy server. I won't be surprised if it's happened to someone here too. This is very timely, because minutes ago, I made a link to Russ Cox's articles in my Kernighan awk repo: https://github.com/andychu/bwk https://swtch.com/~rsc/regexp/regexp1.html If you are not familiar with th…

The key quote here is: "Regular expressions are one of computer science's shining examples of how using good theory leads to good programs ..." "Today, regular expressions have also become a shining example of how ignoring good theory leads to bad programs. The regular expression implementations used by today's popular tools are significantly slower than the ones used in many of those thirty-year-old Unix tools." The…

It's a little unfair to complain that they're slower than 30 year old regex engines when the old regex engines were so feature limited that they were nearly useless.
Post reply on HN