Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

211–220 of 335 posts

Re: Stack Overflow Outage Postmortem

#211

Post by Niksko »

flukus wrote: post #202
mattmanser wrote: post #192

Earlier quoted context omitted.

Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot. Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.

How are they hard to test? Given input x, expect output y. It's one of the easiest things in the world to test.

The point of this post is that even though the regex behaves correctly (input x produces expected output y), you also need to consider performance constraints.

Without fuzzing, it's going to be pretty difficult to come up with enough test cases to thoroughly test a regex.

Re: Stack Overflow Outage Postmortem

#212

Post by lazyant »

"the entire site became unavailable since the load balancer took the servers out of rotation." I don't care about the regexp, this is bad SRE, you can't just take servers out of rotation without some compensation action.

Never mind that it looks like all web servers where taken out of rotation, even one server down could cause a cascading effect (more traffic directed to the healthy ones that end up dying, in a traffic-based failure). One action for example after n servers have gone down, (besides getting up other m servers) is to put (at least some) servers in a more basic mode (read only/static, some features disabled), not guaranteed but that could have prevented this and other type of down times.

Re: Stack Overflow Outage Postmortem

#213

Post by mattmanser »

flukus wrote: post #202
mattmanser wrote: post #192

Earlier quoted context omitted.

Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot. Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.

How are they hard to test? Given input x, expect output y. It's one of the easiest things in the world to test.

Yeah, testing what you expect in regexes is super easy. Edge case testing is not though. Just like exactly what happened in the post this discussion is about..

Re: Stack Overflow Outage Postmortem

#214

Post by carterehsmith »

onetwotree wrote: post #118

It seems like there should be a way to determine whether a regex can be compiled using the classic O(n) DFA algorithm or with whatever madness PCREs use to support backtracking and so on. Anybody know if any regex engines attempt this? Obviously you can still shoot yourself in the foot, but it's somewhat more difficult to do so in a situation like this where the regex in question "looks" cheap.

As an alternative, in Java, there is FindBugs plugin "find-sec-bugs" that flags potentially-long-running regexes.

In my current project, it found two of those in Google Zxing library.

Re: Stack Overflow Outage Postmortem

#215

Post by DasIch »

mattmanser wrote: post #192
judah wrote: post #152

Earlier quoted context omitted.

I've fixed so many bugs using regex, only to have to fix several bugs later. My current stance is, avoid regex if at all possible. Turns out, many of the things we use regex for is possible without. Often times, .Substring, .IndexOf, and using LINQ over strings is sufficient.

Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot. Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.

Regexes are perfectly fine. Awful implementations that accept patterns that aren't regular expressions without complaint and provide little to no tools to look at underlying automata or to step through them during execution are the problem.

It's quite an amazing problem to have honestly because it really shouldn't be a problem to create a proper implementation. You can learn the theory behind regular expression in a few hours and know pretty much everything there is to know.

Re: Stack Overflow Outage Postmortem

#217

Post by hinkley »

OJFord wrote: post #165
zymhan wrote: post #45

Earlier quoted context omitted.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

It was in a multiline code block, so it just had a tonne of horizontal scroll. See the edit: http://stackoverflow.com/revisions/38484433/2

If he had used tabs instead of spaces this wouldn't have been a problem.

Re: Stack Overflow Outage Postmortem

#218

Post by fermuch »

emn13 wrote: post #104
xigency wrote: post #78

Earlier quoted context omitted.

The key quote here is: "Regular expressions are one of computer science's shining examples of how using good theory leads to good programs ..." "Today, regular expressions have also become a shining example of how ignoring good theory leads to bad programs. The regular expression implementations used by today's popular tools are significantly slower than the ones used in many of those thirty-year-old Unix tools." The…

So you have the classic NFA and DFA engines, and these more modern NSH (non-deterministic space heater) engines. At least they keep you warm. Sometimes. How can it not be a feature that you can heat more space the more spaces you feed it?

> How can it not be a feature that you can heat more space the more spaces you feed it?

You just made my day. Thank you.

Re: Stack Overflow Outage Postmortem

#219

Post by fermuch »

lazyant wrote: post #212

"the entire site became unavailable since the load balancer took the servers out of rotation." I don't care about the regexp, this is bad SRE, you can't just take servers out of rotation without some compensation action. Never mind that it looks like all web servers where taken out of rotation, even one server down could cause a cascading effect (more traffic directed to the healthy ones that end up dying, in a traff…

I was thinking the same while reading that part, and I also find it very scary. Why would you even _let_ the load balancer take off _all_ of your servers?

Re: Stack Overflow Outage Postmortem

#220

Post by solipsism »

lilbobbytables wrote: post #194
smrtinsert wrote: post #60

"This regular expression has been replaced with a substring function." This should be the title of a book on software engineering.

What exactly is a substring function, and what makes it different than a regex?

My guess is they're searching for the first non-whitespace character, reverse-searching for the last non-whitespace character, and then using String.Substring to return only what's in the middle.

As to why they're not using String.Trim (https://msdn.microsoft.com/en-us/library/t97s7bs3(v=vs.110)....), maybe it's because String.Trim doesn't seem to know about the 200c whitespace character.

Post reply on HN