[deleted]
Stack Overflow Outage Postmortem
321–330 of 335 posts
Re: Stack Overflow Outage Postmortem
#322Earlier quoted context omitted.
I've fixed so many bugs using regex, only to have to fix several bugs later. My current stance is, avoid regex if at all possible. Turns out, many of the things we use regex for is possible without. Often times, .Substring, .IndexOf, and using LINQ over strings is sufficient.
Regexes are almost always a massive code smell. They should almost never be used, bad idea, bad implementation, hard to grok, hard to debug, hard to test, hard to spot. Whoever came up with them has surely been given the same honorary place in hell with Jon Postel, who invented the utterly disastrous "be liberal in what you accept", that has plagued all web developers for the last 20 years.
>"be liberal in what you accept"
... is a part of so-called unix philosophy, the CAUSE of the internet. The fact that many web developers forgot to become a programmer is just a fun fact changing nothing.
Re: Stack Overflow Outage Postmortem
#323"the entire site became unavailable since the load balancer took the servers out of rotation." I don't care about the regexp, this is bad SRE, you can't just take servers out of rotation without some compensation action. Never mind that it looks like all web servers where taken out of rotation, even one server down could cause a cascading effect (more traffic directed to the healthy ones that end up dying, in a traff…
I have the same concern - including with our own system. As far as I know, all LBs are designed with this faulty logic. That's the case with our in-house F5 BigIPs, and I believe also for AWS's ELBs.
Sure, it makes sense to cut malfunctioning servers out of the production pool when discovered. But at the point where you've dumped out every production server, you've just created the worst possible situation - you're absolutely guaranteed not to be able to serve any traffic. At the point you get to zero functioning servers, a better response would be to restore all servers to the pool and just hope. That would be bad, but less so.
What we've done on our internal systems is to set things up so that when the last server is dropped, it fails over to a passive pool that is really just the normal production pool with no health checks. But I'm not aware of any LB that supports this directly, we had to glue together some ugly configurations to get this to work.
Re: Stack Overflow Outage Postmortem
#324Earlier quoted context omitted.
Oh totally. I assumed that unicode bs immediately. And anyone would make this mistake easily. That's the point -- gotta have it imprinted in the brains, that regexes are for finding things in files, not for your production code. I've used them myself, but I'd like to think that when i type that regex in i stop and thing whether i will be feeding raw user inputs into it.
Many times regexes are more clear and therefore less bug prone than any non regex alternative. They have their use. Even in production.
example please
Re: Stack Overflow Outage Postmortem
#325Earlier quoted context omitted.
They're only missing one feature: back references. This feature is not needed for the majority of regular expressions, so the 30 year old engines are actually remarkably useful and don't have these pathologies. And actually it was the introduction of back references (in Perl) that causes you to have to implement backtracking regular expression engines.
In addition to back references, Perl also has forward and back assertions, IF/ELSE, and recursion IIRC. I'm not sure if anyone actually USES those features, but they are there. Python adopted back references as well as forward and back assertions, but not the other stuff. Perl really turned regular expressions into something else entirely...
Yes, I forgot about assertions. I've never _really_ used them, but I remember they were useful once or twice when wrangling with a particularly ugly format problem.
Re: Stack Overflow Outage Postmortem
#326Earlier quoted context omitted.
How are they hard to test? Given input x, expect output y. It's one of the easiest things in the world to test.
The point of this post is that even though the regex behaves correctly (input x produces expected output y), you also need to consider performance constraints. Without fuzzing, it's going to be pretty difficult to come up with enough test cases to thoroughly test a regex.
Wouldn't the same apply to a custom method? Why would code using a combination of indexOf(), substring() and trim() be any less foolproof on arbitrary data?
Re: Stack Overflow Outage Postmortem
#327Earlier quoted context omitted.
How are they hard to test? Given input x, expect output y. It's one of the easiest things in the world to test.
The point of this post is that even though the regex behaves correctly (input x produces expected output y), you also need to consider performance constraints. Without fuzzing, it's going to be pretty difficult to come up with enough test cases to thoroughly test a regex.
[deleted]
Re: Stack Overflow Outage Postmortem
#328Earlier quoted context omitted.
Many times regexes are more clear and therefore less bug prone than any non regex alternative. They have their use. Even in production.
example please
Compressing multiple forms of non-unicode whitespace to single space. Used for cleaning text from input fields that often contains unwanted characters from copy/paste.
The regexp for this is simply \s+
Re: Stack Overflow Outage Postmortem
#329Hmmn, if it was one request, how did it cause other web servers in the farm go down?
Re: Stack Overflow Outage Postmortem
#330I remember the day I learned that Python's "re" module uses backtracking for non-extended regexes. My tests covered lots of corner cases in the regex logic, but were too short for me to notice the performance penalty. Luckily I only caused a partial outage in production. I actually got to talk to Raymond Hettinger (Python core team) about why re uses a potentially exponential-time algorithm for regexes when there is…
I started writing a replacement in Python that doesn't use backtracking. https://github.com/cyphar/redone
I finished (but never optimized it): https://bitbucket.org/devin.jeanpierre/re0