Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

21–30 of 335 posts

Re: Stack Overflow Outage Postmortem

#21

Post by wheaties »

monochromatic wrote: post #10

> So the Regex engine has to perform a “character belongs to a certain character class” check (plus some additional things) 20,000+19,999+19,998+…+3+2+1 = 199,990,000 times, and that takes a while. 199,990,000 isn't really all that many. I'm a little surprised it didn't just cause a momentary blip in performance. edit: whoops, i guess that's per page load

SO is I/O bound most of the time. If you've set up your system to handle high workloads of I/O bound traffic, then hitting CPU bounds throws a real wrench in your cogs.

To put this another way, SO is one of the most traffic'd sites on the internet. So a page that's loaded 10k+ times a second is going to push that number much, much, higher. If the CPU can't clear 10k+ req in under the regular time it takes, everything starts to back up.

Re: Stack Overflow Outage Postmortem

#22

Post by [deleted] »

monochromatic wrote: post #10

> So the Regex engine has to perform a “character belongs to a certain character class” check (plus some additional things) 20,000+19,999+19,998+…+3+2+1 = 199,990,000 times, and that takes a while. 199,990,000 isn't really all that many. I'm a little surprised it didn't just cause a momentary blip in performance. edit: whoops, i guess that's per page load

[deleted]

Re: Stack Overflow Outage Postmortem

#25

Post by jrbancel »

yeukhon wrote: post #5

This seems like a hard-to-expect edge case for real. I think catching edge case is needed (means more rigorous testing). This is the equivalence of algorithm complexity analysis. How bad can my algorithm be? But regular expression, to be honest, is usually something I hardly think about performance. I don't know about others, but most of the my input are small enough. How big of an input should I test? If I were to d…

> But regular expression, to be honest, is usually something I hardly think about performance.

This is actually not an uncommon problem. I recently experienced a backend system going down because of catastrophic backtracking. There is a reason why proper regex libraries have a timeout on the match methods.

For example: https://msdn.microsoft.com/en-us/library/hh160204(v=vs.110)....

Re: Stack Overflow Outage Postmortem

#27

Post by allannienhuis »

monochromatic wrote: post #10

> So the Regex engine has to perform a “character belongs to a certain character class” check (plus some additional things) 20,000+19,999+19,998+…+3+2+1 = 199,990,000 times, and that takes a while. 199,990,000 isn't really all that many. I'm a little surprised it didn't just cause a momentary blip in performance. edit: whoops, i guess that's per page load

They mentioned that the drop in performance caused the load balancer to remove the servers from rotation. They said that the site would have otherwise been pretty functional. So the resolution to the issue (aside from fixing the regex) was to update the configuration of the loadbalancer/healthcheck to avoid this type of issue.

Re: Stack Overflow Outage Postmortem

#28

Post by chiph »

redbeard0x0a wrote: post #13

In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?

I think you've got the right approach - a vertical slice through the app that checks every layer. You want to know if a user can get useful info from your site, and it tracks (separately!) the common path their query would follow.

The danger is that the endpoint becomes public knowledge and comes under a DDOS attack. Putting an IP address filter on that endpoint is usually enough to stop that.

Re: Stack Overflow Outage Postmortem

#29

Post by StavrosK »

I don't understand something: the regex expected a space character, followed by the end of the string. If the last character wasn't a space, this could never match. Why did the engine keep backtracking, even though it's easy to figure out that it could never match the regex?
Post reply on HN