Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow Outage Postmortem

stackstatus.net

Post reply on HN
Print view View original

141–150 of 335 posts

Re: Stack Overflow Outage Postmortem

#141

Post by lnanek2 »

avar wrote: post #129

My rephrasing of their follow-up actions: * "Audit our regular expressions and post validation workflow for any similar issues" * ==> "Not even people who've worked for years on the guts of regex engines can easily predict the runtime of a given regex, but somehow our engineers will be expected to do that". * "Add controls to our load balancer to disable the healthcheck – as we believe everything but the home page wo…

I think an audit on regex usage would be fine in this case because it is silly to be using an expensive regex on the server for every home page display. It should be done when the data item is created, not when it is displayed.

Even if they need to support full round trip back to the originally entered data for editing purposes, they could have an extra column in the table for that purpose only, or they could case the displayed output for the post in something like Redis.

I assume, just like those PHP forums of old, they change so much about the page based on the logged in user they can't actually fully cache the home page for all users.

So an audit should just lead them to remove the regex on display anyway, not try to figure out the run time of it.

Re: Stack Overflow Outage Postmortem

#142

Post by StevePerkins »

msoad wrote: post #119
smrtinsert wrote: post #60

"This regular expression has been replaced with a substring function." This should be the title of a book on software engineering.

"I had a problem, solved it with RegExp, now I have two problems"

Heh... as a Java developer, my favorite version of that joke is where you solve the problem with Java, and now you have an `AbstractProblemFactoryFactory`.

Re: Stack Overflow Outage Postmortem

#143

Post by olympus »

dingaling wrote: post #130
jandrese wrote: post #56

Earlier quoted context omitted.

Browsers typically collapse whitespace, so it probably would render as a single space.

I frequently complete eBay feedback 'comments' fields[0] with a variety of Unicode spaces and Firefox at least doesn't seem to collapse them. [0] eBay insists on something being entered and when it was a routine transaction with a vendor I seldom have anything useful to say.

Not sure if you're a frequent eBayer, but the common behavior in that community is just "A+++++++++" with some arbitrary number of "+" indicating that everything went fine.

Re: Stack Overflow Outage Postmortem

#145

Post by mwpmaybe »

This is why I always do:

  s/^\s+//;
  s/\s+$//;
Instead of:

  s/^\s+|\s+$//;
Weirdly, I've "known" this since I started writing Perl in the mid-'90. Not sure where I originally read it (or was told it). Funny how that works.

I try to write my regexes such that they anchor at the front of the strong or the back, or they describe the whole string; never an either-or anchoring type situation like this example.

Spaces at beginning of string (100,000 iterations):

             Rate onestep twostep
  onestep 62500/s      --     -2%
  twostep 63694/s      2%      --

  real	0m3.093s
  user	0m3.066s
  sys	0m0.018s
Spaces at end of string (100,000 iterations):

             Rate twostep onestep
  twostep 55249/s      --     -9%
  onestep 60976/s     10%      --

  real	0m3.453s
  user	0m3.421s
  sys	0m0.022s
Spaces in middle of string (only 500 iterations because I don't want to sit here for four hours):

             Rate onestep twostep
  onestep  7.11/s      --   -100%
  twostep 16667/s 234333%      --

  real	1m10.741s
  user	1m10.207s
  sys	0m0.228s

Re: Stack Overflow Outage Postmortem

#146

Post by shdon »

undersuit wrote: post #37
fweespeech wrote: post #18

The lesson seems to be "Always run trim() before running regex" and "validate content as much as possible before running regex".

Trim would not have worked, the post started with '-- play happy sound for player to enjoy', had 20000 characters of whitespace, and then some other character.

Which also wouldn't have been caught by the regex, because it was designed to do the same as Trim. Given the input, all that whitespace was not actually supposed to be removed and the regex worked, it was just a degenerate case that slowed it down to a crawl. Trim would not slow down on this input.

Re: Stack Overflow Outage Postmortem

#147

Post by gnud »

jandrese wrote: post #53
revelation wrote: post #30

They implemented trim with a regex? Neither Java nor .NET do that. The postmortem here should probably be "why are you reimplementing trim".

Most likely answer: The built-in trim isn't Unicode aware or has buggy behavior when dealing with Unicode.

.NET builtin trim lets you specify which 'char' values to trim [1]. .NET is notoriously tied to UTF-16 (a char is 16 bit), and you have to handle surrogate pairs very carefully, but I can't really imagine that being any better with the built-in System.Text.RegularExpressions.Regex

1: https://msdn.microsoft.com/en-us/library/d4tt83f9(v=vs.110)....

Re: Stack Overflow Outage Postmortem

#148

Post by mark-r »

alexbecker wrote: post #77

I remember the day I learned that Python's "re" module uses backtracking for non-extended regexes. My tests covered lots of corner cases in the regex logic, but were too short for me to notice the performance penalty. Luckily I only caused a partial outage in production. I actually got to talk to Raymond Hettinger (Python core team) about why re uses a potentially exponential-time algorithm for regexes when there is…

I vaguely remember a replacement for Python's re module, but I can't remember the name, and Googling for it is an exercise in frustration.

Edit: it's "regex" - https://pypi.python.org/pypi/regex. I have no idea if it behaves better with backtracking, a cursory glance at the documentation doesn't indicate any changes in that area.

Re: Stack Overflow Outage Postmortem

#149

Post by xirdstl »

zymhan wrote: post #45
Someone1234 wrote: post #36

Earlier quoted context omitted.

Well in this case a post contained 20K whitespaces, so I wouldn't jump to the conclusion that it was a mistake rather than intentional.

Yeah, I'm trying to figure out how you even get 20,000 spaces into a Stack Exchange post, and how it would render in your browser.

Cat laying its head on the spacebar.

Mine does that on the Control key all the time.

Re: Stack Overflow Outage Postmortem

#150

Post by thaumasiotes »

onetwotree wrote: post #118

It seems like there should be a way to determine whether a regex can be compiled using the classic O(n) DFA algorithm or with whatever madness PCREs use to support backtracking and so on. Anybody know if any regex engines attempt this? Obviously you can still shoot yourself in the foot, but it's somewhat more difficult to do so in a situation like this where the regex in question "looks" cheap.

> It seems like there should be a way to determine whether a regex can be compiled using the classic O(n) DFA algorithm or with whatever madness PCREs use to support backtracking and so on.

Obviously. If the "regex" includes a backreference, it requires backtracking. If it includes only regular operations (I can't call any other nonregular operations that people might expect to mind), it doesn't. This is information that can be trivially surfaced by a parser.

Post reply on HN