TIL what language Stack Overflow is written in.
what language is that? Couldn't find it in the post.
Stack Overflow Outage Postmortem
221–230 of 335 posts
Re: Stack Overflow Outage Postmortem
#222Earlier quoted context omitted.
> It seems like there should be a way to determine whether a regex can be compiled using the classic O(n) DFA algorithm or with whatever madness PCREs use to support backtracking and so on. Obviously. If the "regex" includes a backreference, it requires backtracking. If it includes only regular operations (I can't call any other nonregular operations that people might expect to mind), it doesn't. This is information…
I wonder why regex libraries don't do this then? Isn't /\s+$/ backreference free? Or does including the anchor change that?
It's backreference free. Backtracking to process it is just bad design.
Here's a state machine that recognizes /^.*\s+$/:
1: initial state (non-matching)
2: matching state
all states transition to 2 on \s, and to 1 on \S. Done.
Taking this a little further, we can observe that since the transition table is identical for every state, there's no real need to have states at all -- you get the same effect by just checking the last character.
Re: Stack Overflow Outage Postmortem
#223Earlier quoted context omitted.
Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot. Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.
How are they hard to test? Given input x, expect output y. It's one of the easiest things in the world to test.
Where do you think you are, right now?
Re: Stack Overflow Outage Postmortem
#224In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?
I'd be VERY careful about including external dependancies in an HTTP health check which results in a web server being removed from service - it's usually an invitation for cascading failures. 1) If you do have a back-end failure, this setup can cloud the root cause during recovery because your downstream web servers are down as well. 2) Transitory back-end failures can cascade, and your health checks can make this wo…
You make a good point about health checks. Its easy to conflate single node health with overall service health. The LB check should really be about "can this node serve HTTP traffic", even if the non-health check requests can only send back 5xx responses because their dependencies are failing.
I've also had the opposite problem where the health check would answer 200 OK, but the real app itself had a corrupted internal state because of a poor design/threading bug. If the health check had been the home page the node would have been pulled from the LB. While a more in-depth health check would have helped here, I think its better to alert/monitor/kill any node with a higher than normal error rate and leave the health check simple.
Re: Stack Overflow Outage Postmortem
#225Re: Stack Overflow Outage Postmortem
#226Earlier quoted context omitted.
I think you've got the right approach - a vertical slice through the app that checks every layer. You want to know if a user can get useful info from your site, and it tracks (separately!) the common path their query would follow. The danger is that the endpoint becomes public knowledge and comes under a DDOS attack. Putting an IP address filter on that endpoint is usually enough to stop that.
The concept I try to go for with that status check is 'Can this node connect to everything so it can successfully respond to http requests'. However my approach wouldn't identify an overloaded server, which might be a good thing if we need to scale up - taking down an overloaded server is just going to make the other servers that much more overloaded. I'm aways up for hearing about other ways people solve health chec…
> taking down an overloaded server is just going to make the other servers that much more overloaded
Emphatically agree, but it's important in the first place to design and deploy your infrastructure such that basic increases of scale are accounted for - prevention is the most important piece of the puzzle. Get that right and an overloaded server is symptomatic of something else, in which case taking down access to the unruly resource is first priority.
IMO, the big takeaway here is that they were load balancing simply by only hitting the top level - selectivity is somewhat tedious to build but worth it in the long run.
Re: Stack Overflow Outage Postmortem
#227Re: Stack Overflow Outage Postmortem
#228"This regular expression has been replaced with a substring function." This should be the title of a book on software engineering.
"I had a problem, solved it with RegExp, now I have two problems"
There's quite a nice history to that one:
http://regex.info/blog/2006-09-15/247
It dates back to August 12, 1997!
(Perhaps someone should encourage jwz to have a 20th birthday party for that at DNA?)
Re: Stack Overflow Outage Postmortem
#229I think this might have been the post they quoted. http://stackoverflow.com/questions/38484433/in-corona-sdk-ho...
Yep.
Re: Stack Overflow Outage Postmortem
#230Earlier quoted context omitted.
Yes, we took a stackdump and saw the traces leading to that regex.
How large was the stack dump? I'm impressed you identified it this quickly considering the time it would take to get this to disk.
We have an internal tool where we can dump stack traces almost instantly by attaching to a running process. We can't open source it due to using Microsoft lab code which was never licensed itself. However, clrmd (https://github.com/Microsoft/clrmd and https://www.nuget.org/packages/Microsoft.Diagnostics.Runtime) means an open source version is hopefully on the horizon. As an example, here's the result of me piddling in 2 hours with LinqPad: https://gist.github.com/NickCraver/d6292c0c7f93767686e8c5c89... Process attachment is in the clrmd roadmap, but isn't stable yet.