Earlier quoted context omitted.
You're misreading the regex. \u200c is a single whitespace character. http://www.fileformat.info/info/unicode/char/200c/index.htm
But that's a weird character to put in a comment line! I don't get how this would happen accidentally.
Stack Overflow Outage Postmortem
271–280 of 335 posts
Re: Stack Overflow Outage Postmortem
#272Re: Stack Overflow Outage Postmortem
#273Earlier quoted context omitted.
It was in a multiline code block, so it just had a tonne of horizontal scroll. See the edit: http://stackoverflow.com/revisions/38484433/2
Good find! Where did you see this? Or was it in the comments via Nick Craver?
> The malformed post contained roughly 20,000 consecutive
> characters of whitespace on a comment line that started
> with -- play happy sound for player to enjoy. For us,
> the sound was not happy.
So I just searched: https://www.google.co.uk/search?q="play+happy+sound+for+player+to+enjoy"+site%3Astackoverflow.comRe: Stack Overflow Outage Postmortem
#274Re: Stack Overflow Outage Postmortem
#275Re: Stack Overflow Outage Postmortem
#276In the past, I have done Load Balancer status checks against a special /status endpoint. I queried all the connected services (i.e. DB, Redis, etc) with a super fast query (i.e. `SELECT version();`). Monitoring CPU/MEM usage for scaling was separate. Comparing this to checking the home page, what is the best way to setup a health check for your load balancers?
I'd be VERY careful about including external dependancies in an HTTP health check which results in a web server being removed from service - it's usually an invitation for cascading failures. 1) If you do have a back-end failure, this setup can cloud the root cause during recovery because your downstream web servers are down as well. 2) Transitory back-end failures can cascade, and your health checks can make this wo…
> 4) Are you absolutely positive that your app is worthless without a DB/redis/backend-baz, and every endpoint is broken?
Yes, if that's the case you should not respond "unable to serve". However, you should probably fail setup check and not start serving traffic if you're a newly provisioned node.
But!
I wouldn't call this cascading failure more than error propagation. If a critical dependency is down making the node essential worthless, it should be removed. This can be caused by things like network partitions and whatnot - as you say by 5. You say "it's not likely" which is sadly a common statement, but when it does happen it can be nasty - and the bigger environment the often it happens.
Cascading failure usually (in my experience anyway) refers to having the actual failover cause more outages. In the case of failing /status I can only see this happening if the architecture of the system is broken anyway - or the load is genuinely too high for the failure that just occurred.
You say: "The DB and overall health of the service cluster are monitored through more organic instrumentation". What is acting on that? What happens when you do get that rare network partition and half of your web nodes cannot reach the database?
Re: Stack Overflow Outage Postmortem
#277I always cringe when I see regex used for such simple string checks. In fact, Stackoverflow is full of accepted answers that "solve" problems that way.
Re: Stack Overflow Outage Postmortem
#278Re: Stack Overflow Outage Postmortem
#279Earlier quoted context omitted.
The key quote here is: "Regular expressions are one of computer science's shining examples of how using good theory leads to good programs ..." "Today, regular expressions have also become a shining example of how ignoring good theory leads to bad programs. The regular expression implementations used by today's popular tools are significantly slower than the ones used in many of those thirty-year-old Unix tools." The…
FWIW, the conversion from NFA (non-deterministic, i.e. backtracking) to DFA (deterministic, linear time) can take exponential space. So there's another avenue for DDOS; it's a lot harder to exploit, though, because it requires the attacker to control the input (i.e. regular expression) to the NFA->DFA transformation, rather than merely provide a string that takes a long time for an NFA to recognize.
Re: Stack Overflow Outage Postmortem
#280"This regular expression has been replaced with a substring function." This should be the title of a book on software engineering.
I've fixed so many bugs using regex, only to have to fix several bugs later. My current stance is, avoid regex if at all possible. Turns out, many of the things we use regex for is possible without. Often times, .Substring, .IndexOf, and using LINQ over strings is sufficient.