Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

201–210 of 335 posts

Re: Stack Overflow Outage Postmortem

#201

In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?

I'd be VERY careful about including external dependancies in an HTTP health check which results in a web server being removed from service - it's usually an invitation for cascading failures.

1) If you do have a back-end failure, this setup can cloud the root cause during recovery because your downstream web servers are down as well.

2) Transitory back-end failures can cascade, and your health checks can make this worse as things flap around. You need to be VERY careful with your timers and retry logic, and tune them appropriately. eg, how does `/status` handle a single TCP reset from the DB? When does it decide to timeout if the DB doesn't respond? It's hard to get this right - network errors vs protocol errors vs timeouts are often handled differently at different layers of the controller, and the load balancer has its own retries and timeouts.

3) If this health check is being used for an API, recovery can be more difficult because of thundering herds. eg, 5/5 API servers go down because of a temporary DB failure. Queues and requests backup. 1/5 API servers is restored to service a bit earlier than the others, and immediately goes down again because it's not degrading gracefully under heavy load. Wash/rinse/repeat until you figure out how to load shed.

4) Are you absolutely positive that your app is worthless without a DB/redis/backend-baz, and every endpoint is broken? Ideally, the app will degrade gracefully if certain request can be served from cache, or don't require the failed backend.

5) The failure case where this type of thing might be useful (network partition affecting DB connectivity from a subset of your webservers) isn't very common in the environments I've been involved with.

A far more productive approach if you absolutely need this sort of thing is to implement circuit breakers[1] at the app level, and explicitly make certain tripped breakers a hard failure for the /status endpoint. This has the advantage of not depending solely on synthetic queries, and having very explicit restoration logic.

For load balancer health checks, I prefer ones that hit the controller and expect a 200, and nothing more. (The DB and overall health of the service cluster are monitored through more organic instrumentation.)

I was laughing to myself when I read the Stack Overflow post mortem because I have a healthy pile of healthcheck-too-expensive war stories, and their experience resonated.

[1] http://martinfowler.com/bliki/CircuitBreaker.html

Re: Stack Overflow Outage Postmortem

#202
post #152

Earlier quoted context omitted.

I've fixed so many bugs using regex, only to have to fix several bugs later. My current stance is, avoid regex if at all possible. Turns out, many of the things we use regex for is possible without. Often times, .Substring, .IndexOf, and using LINQ over strings is sufficient.

Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot. Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.

How are they hard to test? Given input x, expect output y. It's one of the easiest things in the world to test.

Re: Stack Overflow Outage Postmortem

#203
post #158
post #153

Earlier quoted context omitted.

Well, StackOverflow devs are able to cheat and load up the site on their local machine if they want to

I'm not sure I like the idea of being able to connect to a prod db from a dev instance, but whatever floats your boat.

Who says they have to? The can just restore a recent backup.

Re: Stack Overflow Outage Postmortem

#204
post #158
post #153

Earlier quoted context omitted.

Well, StackOverflow devs are able to cheat and load up the site on their local machine if they want to

I'm not sure I like the idea of being able to connect to a prod db from a dev instance, but whatever floats your boat.

I doubt they'd allow that either - but database exports are a thing or they could just query the database or elasticsearch themselves.

Re: Stack Overflow Outage Postmortem

#208

Earlier quoted context omitted.

I frequently complete eBay feedback 'comments' fields[0] with a variety of Unicode spaces and Firefox at least doesn't seem to collapse them. [0] eBay insists on something being entered and when it was a routine transaction with a vendor I seldom have anything useful to say.

Not sure if you're a frequent eBayer, but the common behavior in that community is just "A+++++++++" with some arbitrary number of "+" indicating that everything went fine.

He's no doubt observed that and is using whitespace to rail against the ridiculousness of that convention.

Re: Stack Overflow Outage Postmortem

#210
post #9

Ha! The same bug happened internally at my company. In that case it was a regex matching a URL taking so much CPU as to cause a DOS of a proxy server. I won't be surprised if it's happened to someone here too. This is very timely, because minutes ago, I made a link to Russ Cox's articles in my Kernighan awk repo: https://github.com/andychu/bwk https://swtch.com/~rsc/regexp/regexp1.html If you are not familiar with th…

There's a simple trick the real-time and high-assurance communities use to catch stuff like this: enforce a hard limit for time each task takes to complete. Some you might give leeway to prevent DDOS'ing your users. Many (most?) things have a sane, hard limit that can apply along with a log entry or notification to ops. Keeps one user or task from DDOSing whole system or forces it to fail fast + noticeably.

Note a lot of developers use this trick anymore but it's quite powerful. Should get more consideration. Not to mention the real solution of choosing tools or algorithms with predictable, reliable behavior. Those definitely exist for the task they're performing as other commenters noted.

EDIT to add modern example from high-assurance field that shows how awesome it can be (esp see Github link):

https://leepike.github.io/Copilot/

Post reply on HN