Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

191–200 of 335 posts

Re: Stack Overflow Outage Postmortem

#191

Post by chubot »

throwanem wrote: post #122
chubot wrote: post #9

Ha! The same bug happened internally at my company. In that case it was a regex matching a URL taking so much CPU as to cause a DOS of a proxy server. I won't be surprised if it's happened to someone here too. This is very timely, because minutes ago, I made a link to Russ Cox's articles in my Kernighan awk repo: https://github.com/andychu/bwk https://swtch.com/~rsc/regexp/regexp1.html If you are not familiar with th…

The first non-boilerplate-header line in b.c: /* lasciate ogne speranza, voi ch'intrate. */ Well, that's encouraging... ("Abandon all hope, you who enter here.")

Admittedly I haven't stepped through all the code, but it's 958 lines and doesn't look horrible. It somewhat resembles the example C programs here:

https://swtch.com/~rsc/regexp/

In particular there don't seem to be crazy optimizations and non-portable stuff like you see in other production regex implementations and in interpreters. It's pure ANSI C.

The functions are all short -- no 1000 line monsters like I've seen in a lot of old C code lately.

The rsc example code is actually a great example of using pointers in C. People rightly avoid lots of raw pointers for application code, but for this use case, they yield compact and elegant code.

Re: Stack Overflow Outage Postmortem

#192

Post by mattmanser »

judah wrote: post #152
smrtinsert wrote: post #60

"This regular expression has been replaced with a substring function." This should be the title of a book on software engineering.

I've fixed so many bugs using regex, only to have to fix several bugs later. My current stance is, avoid regex if at all possible. Turns out, many of the things we use regex for is possible without. Often times, .Substring, .IndexOf, and using LINQ over strings is sufficient.

Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot.

Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.

Re: Stack Overflow Outage Postmortem

#193

Post by skeletonjelly »

OJFord wrote: post #165
zymhan wrote: post #45

Earlier quoted context omitted.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

It was in a multiline code block, so it just had a tonne of horizontal scroll. See the edit: http://stackoverflow.com/revisions/38484433/2

Still hasn't killed as many characters as GRRM

Re: Stack Overflow Outage Postmortem

#195

Post by skeletonjelly »

gibrown wrote: post #189
kilroy123 wrote: post #23

> It took 10 minutes to identify the cause. I'm impressed they were able to do this so quickly.

Agreed that's impressive debugging for this issue. But... > 10 minutes to roll out the fix That seems very slow to me. 30% of their down time was because their deploy process is slow.

http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-...

FWIW here's a write up on their process

Also I imagine that 10 minutes included dev and testing, not just the deployment part of "rolling out"

Re: Stack Overflow Outage Postmortem

#196

Post by chubot »

xigency wrote: post #114
xigency wrote: post #78

Earlier quoted context omitted.

The key quote here is: "Regular expressions are one of computer science's shining examples of how using good theory leads to good programs ..." "Today, regular expressions have also become a shining example of how ignoring good theory leads to bad programs. The regular expression implementations used by today's popular tools are significantly slower than the ones used in many of those thirty-year-old Unix tools." The…

Unfortunately, there is a hint as to how this has happened: "This strategy is no longer practical: users have come to rely on backreferences for at least occasional use, and backreferences are part of the POSIX standard for regular expressions." What better excuse is there for a poor implementation than standards compliance? In many ways, using regex with backtracking by default is like programming in Lisp without ta…

Yes, I recall that he talks about the multi-engine approach with respect to both PCRE and RE2 somewhere in these articles:

https://swtch.com/~rsc/regexp/

Sorry I don't have the exact reference handy. Apparently the author of PCRE tried to implement an NFA/DFA engine for some regexes as well. And I think RE2 might also have different strategies for different regexes as well, but I forget the details (it might be related to capturing).

Re: Stack Overflow Outage Postmortem

#197

Post by skeletonjelly »

KyleBrandt wrote: post #57
amenod wrote: post #55

Earlier quoted context omitted.

I am genuinly curious: how did you fix it? Did you remove the spaces first and then tried to use substring / trimming with proper testing done, or did you just implement it in place? I have faced similar dilemmas in the past, but I usually go with "put out the fire, then find the correct solution" approach.

Implement in place to put the fire out. Pushed to half the web servers, made sure it fixed the problem, then rolled it out the rest. Coding under fire :P

How did you test this on half? Accessing the single web server directly I'm guessing? Probably not public

Re: Stack Overflow Outage Postmortem

#198

Post by chubot »

mamcx wrote: post #74
chubot wrote: post #9

Ha! The same bug happened internally at my company. In that case it was a regex matching a URL taking so much CPU as to cause a DOS of a proxy server. I won't be surprised if it's happened to someone here too. This is very timely, because minutes ago, I made a link to Russ Cox's articles in my Kernighan awk repo: https://github.com/andychu/bwk https://swtch.com/~rsc/regexp/regexp1.html If you are not familiar with th…

Sadly not much Thompson's libraries are implemented. I have tried to find one for F# but are just toy projects.

It doesn't actually take that many lines of code to implement a linear time NFA engine. Most of the code is actually in the regex compiler. That is, there are only a few actual "instructions" or node types in a regex engine (alternation, concatenation, etc.). The rest is just compiling the bizarre syntax to a those nodes/instructions. (And dealing with Unicode if you need that.)

The whole awk implementation is 958 lines, so it can't be that bad. It supports a fairly featureful regex language (though no capturing AFAICT).

Re: Stack Overflow Outage Postmortem

#199

Post by skeletonjelly »

KyleBrandt wrote: post #65
ww520 wrote: post #46

Earlier quoted context omitted.

System not responsive. Look at the CPU load. Look at the process peaking at 100%. Force dump the stack track of the process couple times. Hmm. All of them stuck in the regex engine. Look back up the stack track to see who calls it. Oh, it's on the home page's text cleansing code. Something like that.

Yes, we took a stackdump and saw the traces leading to that regex.

How large was the stack dump? I'm impressed you identified it this quickly considering the time it would take to get this to disk.

Re: Stack Overflow Outage Postmortem

#200

Post by gibrown »

skeletonjelly wrote: post #195
gibrown wrote: post #189

Earlier quoted context omitted.

Agreed that's impressive debugging for this issue. But... > 10 minutes to roll out the fix That seems very slow to me. 30% of their down time was because their deploy process is slow.

http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-... FWIW here's a write up on their process Also I imagine that 10 minutes included dev and testing, not just the deployment part of "rolling out"

Seemed to say that 14 minutes were spent writing the code to fix it which I assumed meant testing it, but ya not entirely clear.

Maybe "deploy" means the "Deploy" section of this article: http://highscalability.com/blog/2014/7/21/stackoverflow-upda...

Seems to target only being able to "deploy 5 times a day". I guess maybe the build time is the limiting factor.

Post reply on HN