Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

51–60 of 335 posts

Re: Stack Overflow Outage Postmortem

#51

Post by gortok »

mh- wrote: post #44
wheaties wrote: post #21

Earlier quoted context omitted.

SO is I/O bound most of the time. If you've set up your system to handle high workloads of I/O bound traffic, then hitting CPU bounds throws a real wrench in your cogs. To put this another way, SO is one of the most traffic'd sites on the internet. So a page that's loaded 10k+ times a second is going to push that number much, much, higher. If the CPU can't clear 10k+ req in under the regular time it takes, everything…

> To put this another way, SO is one of the most traffic'd sites on the internet. I've seen this said several times here, but never bothered to ask.. by what measure is this true?

Alexa has it at #50:

http://www.alexa.com/siteinfo/stackoverflow.com

Quantcast has a similar ranking for it: https://www.quantcast.com/stackoverflow.com#trafficCard

Re: Stack Overflow Outage Postmortem

#52

Post by revelation »

gortok wrote: post #47
revelation wrote: post #39

Earlier quoted context omitted.

Or just reject this input. The server isn't the time and place to do data cleanup. Can always fix this on the frontend with JavaScript for free if it's an actual problem.

You can't trust that the user has JavaScript enabled; and since you can't trust user input anyway, you'd have to do this server side.

"You can't trust user input" means to not fudge with user data in the first place!

If your code cares about unicode spaces at the beginning of user input, there is your problem.

(Also, you missed the reject part)

Re: Stack Overflow Outage Postmortem

#55

Post by amenod »

KyleBrandt wrote: post #17
laurencei wrote: post #2

Could this has been a deliberate/malicious act? Why else would someone post 20,000 consecutive characters of whitespace on a comment line? Also, the "homepage" of StackOverflow does not show any 'comments' - it is just the top questions? Why was the page loading any comments in the first place?

We don't think it was intentional. Maybe copy and paste, or something an editor did. To clarify the "comment" was not a Stack Overflow comment, but rather a comment in a code block inside a question.

I am genuinly curious: how did you fix it? Did you remove the spaces first and then tried to use substring / trimming with proper testing done, or did you just implement it in place? I have faced similar dilemmas in the past, but I usually go with "put out the fire, then find the correct solution" approach.

Re: Stack Overflow Outage Postmortem

#56

Post by jandrese »

zymhan wrote: post #45
Someone1234 wrote: post #36

Earlier quoted context omitted.

Well in this case a post contained 20K whitespaces, so I wouldn't jump to the conclusion that it was a mistake rather than intentional.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

Browsers typically collapse whitespace, so it probably would render as a single space.

Re: Stack Overflow Outage Postmortem

#57

Post by KyleBrandt »

amenod wrote: post #55
KyleBrandt wrote: post #17

Earlier quoted context omitted.

We don't think it was intentional. Maybe copy and paste, or something an editor did. To clarify the "comment" was not a Stack Overflow comment, but rather a comment in a code block inside a question.

I am genuinly curious: how did you fix it? Did you remove the spaces first and then tried to use substring / trimming with proper testing done, or did you just implement it in place? I have faced similar dilemmas in the past, but I usually go with "put out the fire, then find the correct solution" approach.

Implement in place to put the fire out. Pushed to half the web servers, made sure it fixed the problem, then rolled it out the rest. Coding under fire :P

Re: Stack Overflow Outage Postmortem

#58

Post by mason55 »

bluedino wrote: post #19
laurencei wrote: post #2

Could this has been a deliberate/malicious act? Why else would someone post 20,000 consecutive characters of whitespace on a comment line? Also, the "homepage" of StackOverflow does not show any 'comments' - it is just the top questions? Why was the page loading any comments in the first place?

Imagine if they had only posted 10,000 or 15,000 characters, and it just slowed the site down. How fast would it have been noticed? Hours? Days?

Good monitoring (which I expect the SO guys to have) would have triggered on a spike in 95% or 99% response times so probably almost as quickly

Re: Stack Overflow Outage Postmortem

#59

Post by unethical_ban »

sklivvz1971 wrote: post #34
unethical_ban wrote: post #26

Not understanding why backtracking happened. Once it hit a non space, non end character, move on. Nothing before can match the regex.

It finds a space, tries to match the rest of the string, fails, rolls back to it and goes forward one, finds a space... The trick is that there's no guarantee, in general, that a match failing at character N is due to character N, so the regex engine backtracks.

I guess I need to buff up on my automata, but I would have thought that for positional delimiters, there would be optimizations. If the pattern is \s+$, then I can look from the back of the line, see if there is a space, and if so, go backward until I find a non-space character.
Post reply on HN