Agreed that's impressive debugging for this issue. But... > 10 minutes to roll out the fix That seems very slow to me. 30% of their down time was because their deploy process is slow.
http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-... FWIW here's a write up on their process Also I imagine that 10 minutes included dev and testing, not just the deployment part of "rolling out"
Correct. 10 minutes was from checkin to all servers built out, including a dev and staging tier.
What exactly is a substring function, and what makes it different than a regex?
My guess is they're searching for the first non-whitespace character, reverse-searching for the last non-whitespace character, and then using String.Substring to return only what's in the middle. As to why they're not using String.Trim ( https://msdn.microsoft.com/en-us/library/t97s7bs3(v=vs.110).... ), maybe it's because String.Trim doesn't seem to know about the 200c whitespace character.
From what I understood, trim would work perfectly. It's 200 spaces not a single 200 width character.
My rephrasing of their follow-up actions: * "Audit our regular expressions and post validation workflow for any similar issues" * ==> "Not even people who've worked for years on the guts of regex engines can easily predict the runtime of a given regex, but somehow our engineers will be expected to do that". * "Add controls to our load balancer to disable the healthcheck – as we believe everything but the home page wo…
I don't think they're disabling health checks in general, just adding a way to temporarily disable checks when they are misbehaving.
Correct. I'll be adding this functionality into Opserver so that we can override the health check in HAProxy when we know better.
A few months ago, a Stack Overflow representative asked me if their presence at a dev conference was justified. My positive answer more or less revolved around the importance SO took in the daily life of programmers everywhere. If only she was there to witness the effect of a 34 minute downtime on an open space full of mobile/back/front developers.
Google's cached versions of SO answers are generally up to date, no? Nice try, lazy developers!
Alternatively, you can refer to StackPrinter [0] (especially if arriving via a search engine).
The lesson seems to be "Always run trim() before running regex" and "validate content as much as possible before running regex".
Or just reject this input. The server isn't the time and place to do data cleanup. Can always fix this on the frontend with JavaScript for free if it's an actual problem.
> Can always fix this on the frontend with JavaScript for free
Nice to see what people think of offloading work to users' systems. This kind of thinking leads to slow ensures with fucked up scrolling.
Ha! The same bug happened internally at my company. In that case it was a regex matching a URL taking so much CPU as to cause a DOS of a proxy server. I won't be surprised if it's happened to someone here too. This is very timely, because minutes ago, I made a link to Russ Cox's articles in my Kernighan awk repo: https://github.com/andychu/bwk https://swtch.com/~rsc/regexp/regexp1.html If you are not familiar with th…
There's a simple trick the real-time and high-assurance communities use to catch stuff like this: enforce a hard limit for time each task takes to complete. Some you might give leeway to prevent DDOS'ing your users. Many (most?) things have a sane, hard limit that can apply along with a log entry or notification to ops. Keeps one user or task from DDOSing whole system or forces it to fail fast + noticeably. Note a lo…
You can only safely abort a "task" if the task has specifically be designed that way, or if the task is a process.