Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The dangers of single line regular expressions

greg.molnar.io

Post reply on HN
Print view View original

91–100 of 133 posts

Re: The dangers of single line regular expressions

#91

Post by JonChesterfield »

wlesieutre wrote: post #68
JonChesterfield wrote: post #64

Regular expressions make me sad about our industry. If you read the early papers, you get a very clear language for pattern matching on sequences. They have really nice properties - the compilation to finite automata gives you decidable equality and decidable minimisation. As in you can compile equivalent regex to exactly the same state machine however they were expressed. At some point perl happened and that seems t…

Swift's RegexBuilder DSL from a couple years ago gets away from the illegible subset of ASCII. Easy to explode into a lot lines, but I'd rather have a 50 line RegexBuilder implementation than try to keep track of what the equivalent single-line version is doing. Especially if you ever have to come back to it later and understand it again. And if you ever make revisions in RegexBuilder you have useful diffs instead of…

An alternative to seeking better language APIs.

Parsing regex then pretty-printing the parse tree as s-expressions is very legible. You can also print the parse tree as the original syntax. Postfix will work better for some people, I like the lispy look for parse trees.

Most regex are similar syntax over a parse tree with different parts missing, if you keep track of roughly what features the current engine has in your head the sema checking a real compiler should do could be deferred or incomplete.

Some coding standards will want redundant escapes because that is considered more readable, could put that logic in the pretty-printer.

That's sort of suggesting using your IDE to translate the thing back and forth on the fly instead of persuading colleagues to stop writing in the obfuscated format.

Re: The dangers of single line regular expressions

#95

Post by tsimionescu »

ec109685 wrote: post #53

Escape the output based on the context a string is being used in versus trying to sanitize for all use cases on input. This will guarantee that you’re safe no matter how a piece of content is used tomorrow (just need a new escaping function for that content type), and prevent awkward things like not letting users use “unsafe” strings as input. JSX and XHP are example templating systems that understand context and esc…

The output is not the problem here, it is the input. And, if you can get away with, accepting a small set of known-safe characters is much safer than accepting any character and hoping it will be properly escaped at every level.

When the user hands you a string and you then pass this down to other bits of code, you can't know if it will be used in an SQL query, a regex, in an error message that will be rendered into HTML, etc.

Ideally all layers of your code would handle user input with the utmost care, but that is often very hard to achieve. If you take user input and use it in a regex, it's easy to regex-escape it, but it's much harder to remember that now this whole regex is user input and can't be safely used to, say, construct an SQL query. And even if you remember to properly escape it in the SQL query, it may show up in the returned result, and now if you display that result, you need to be careful to escape it before passing it to some HTML engine.

But then none of this works if you did intend to have some SQL syntax in the regex, or some HTML snippets in the DB: you'd need to make all of these technologies aware of which parts of the expressions are safe and which are tainted by user input.

And this is all just to prevent code injection type attacks. I haven't even discussed more subtle attacks, like using Unicode look-like characters to confuse other users.

Re: The dangers of single line regular expressions

#96

Post by tsimionescu »

ec109685 wrote: post #87
Jerrrry wrote: post #59

Earlier quoted context omitted.

>If a user wants their title to be “hello%0a%3C%25%3D%20File.open%28%27flag.txt%27%29.read%20%25%3E”, so be it. that's a good way to horizontally propagate/reflect XSS and other Code As Data vulnerabilities. better to strip the known-bad/problematic characters https://en.wikipedia.org/wiki/Code_as_data

The known problematic characters are different in json, xml, css, html content, html attributes, MySQL, etc. Unless you have output escaping, it is hard to ensure everything gets caught, no matter how the data enters the system.

Sure, but there is a common set of safe characters that are guaranteed not to cause problems in any of these: the set described by the regex [a-zA-Z0-9 -]. If you can limit user input to this set, you'll drastically reduce the risk of code injection regardless of the stack below you.

Re: The dangers of single line regular expressions

#98

Post by phyzome »

dwheeler wrote: post #49
neilk wrote: post #6

In my experience `$` does reliably mean end of string for regular expressions, unless you specifically ask for "multiline" mode. Ruby seems to be in multiline mode all the time? $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foobar") else "no"' yes $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\nbar") else "no"' no $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\…

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulner…

Interesting that a trailing newline is accepted. Not as bad as what's in the post, at least. Definitely worth breaking out which languages do which of those, though! Python, for instance, only accepts a trailing newline but not additional chars beyond that.

I don't think Java should be in your first list, though? Pattern.matches("^foo$", "foo\n") returns false.

Re: The dangers of single line regular expressions

#99

Post by phyzome »

Jerrrry wrote: post #59
ec109685 wrote: post #53

Escape the output based on the context a string is being used in versus trying to sanitize for all use cases on input. This will guarantee that you’re safe no matter how a piece of content is used tomorrow (just need a new escaping function for that content type), and prevent awkward things like not letting users use “unsafe” strings as input. JSX and XHP are example templating systems that understand context and esc…

>If a user wants their title to be “hello%0a%3C%25%3D%20File.open%28%27flag.txt%27%29.read%20%25%3E”, so be it. that's a good way to horizontally propagate/reflect XSS and other Code As Data vulnerabilities. better to strip the known-bad/problematic characters https://en.wikipedia.org/wiki/Code_as_data

And that's how you end up pissing off users with apostrophes in their names.

Re: The dangers of single line regular expressions

#100

Post by phyzome »

JonChesterfield wrote: post #64

Regular expressions make me sad about our industry. If you read the early papers, you get a very clear language for pattern matching on sequences. They have really nice properties - the compilation to finite automata gives you decidable equality and decidable minimisation. As in you can compile equivalent regex to exactly the same state machine however they were expressed. At some point perl happened and that seems t…

Am I just unusual in really liking the usual regex syntax? (I mean, other than how every engine has a slightly different variation on it.) This might just be a matter of familiarity, but I find the s-expression versions harder to read, despite having worked in a Lisp for more than 10 years.
Post reply on HN