Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

11–20 of 335 posts

Re: Stack Overflow Outage Postmortem

#11

> So the Regex engine has to perform a “character belongs to a certain character class” check (plus some additional things) 20,000+19,999+19,998+…+3+2+1 = 199,990,000 times, and that takes a while. 199,990,000 isn't really all that many. I'm a little surprised it didn't just cause a momentary blip in performance. edit: whoops, i guess that's per page load

[deleted]

Re: Stack Overflow Outage Postmortem

#12

Could this has been a deliberate/malicious act? Why else would someone post 20,000 consecutive characters of whitespace on a comment line? Also, the "homepage" of StackOverflow does not show any 'comments' - it is just the top questions? Why was the page loading any comments in the first place?

on a site like stackoverflow, I would think they get malicious requests like that all the time. Even if it wasn't malicious (perhaps an otherwise benign script gone amuk), I bet something like this happens very often.

It would be interesting to find out if something like this goes into their automated regression tests. :)

Re: Stack Overflow Outage Postmortem

#13
In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate.

Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?

Re: Stack Overflow Outage Postmortem

#15

> So the Regex engine has to perform a “character belongs to a certain character class” check (plus some additional things) 20,000+19,999+19,998+…+3+2+1 = 199,990,000 times, and that takes a while. 199,990,000 isn't really all that many. I'm a little surprised it didn't just cause a momentary blip in performance. edit: whoops, i guess that's per page load

199,990,000 * # of people visiting StackOverflow home page

The result is likely at least 1,000* 199,990,000 at any given point.

Re: Stack Overflow Outage Postmortem

#17

Could this has been a deliberate/malicious act? Why else would someone post 20,000 consecutive characters of whitespace on a comment line? Also, the "homepage" of StackOverflow does not show any 'comments' - it is just the top questions? Why was the page loading any comments in the first place?

We don't think it was intentional. Maybe copy and paste, or something an editor did.

To clarify the "comment" was not a Stack Overflow comment, but rather a comment in a code block inside a question.

Re: Stack Overflow Outage Postmortem

#19

Could this has been a deliberate/malicious act? Why else would someone post 20,000 consecutive characters of whitespace on a comment line? Also, the "homepage" of StackOverflow does not show any 'comments' - it is just the top questions? Why was the page loading any comments in the first place?

Imagine if they had only posted 10,000 or 15,000 characters, and it just slowed the site down. How fast would it have been noticed? Hours? Days?

Re: Stack Overflow Outage Postmortem

#20

> So the Regex engine has to perform a “character belongs to a certain character class” check (plus some additional things) 20,000+19,999+19,998+…+3+2+1 = 199,990,000 times, and that takes a while. 199,990,000 isn't really all that many. I'm a little surprised it didn't just cause a momentary blip in performance. edit: whoops, i guess that's per page load

Per homepage load.
Post reply on HN