Live data from Hacker News

The dangers of single line regular expressions

greg.molnar.io

81–90 of 133 posts

Re: The dangers of single line regular expressions

#81
post #11

This was interesting and new to me, but as other commenters indicate, part of the problem is that we're trying to find the bad thing rather than trying to verify it is the good thing There's a related concept of "failing open vs failing closed" (fail open: fire exit, fail closed: ranch gate) In Jurassic park (amazing book/film to understand system failures), when the power goes out, the fence is functionally an open…

[deleted]

Re: The dangers of single line regular expressions

#82
post #11

This was interesting and new to me, but as other commenters indicate, part of the problem is that we're trying to find the bad thing rather than trying to verify it is the good thing There's a related concept of "failing open vs failing closed" (fail open: fire exit, fail closed: ranch gate) In Jurassic park (amazing book/film to understand system failures), when the power goes out, the fence is functionally an open…

I don't think this is a good example, because the regex does just that: it doesn't try to filter out bad input, it specifically only accepts known good input. If the regex did what it was meant to do, only allowing strings composed of ascii letters and numbers, and space, than the code would have not been exploitable.

Re: The dangers of single line regular expressions

#86
post #6

In my experience `$` does reliably mean end of string for regular expressions, unless you specifically ask for "multiline" mode. Ruby seems to be in multiline mode all the time? $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foobar") else "no"' yes $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\nbar") else "no"' no $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\…

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulner…

`$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(?I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1].

Edit: also quoting from [2]:

> By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence.

[1] https://github.com/scala-js/scala-js/blob/eb160f1ef113794999...

[2] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pa...

Re: The dangers of single line regular expressions

#87
post #59

Escape the output based on the context a string is being used in versus trying to sanitize for all use cases on input. This will guarantee that you’re safe no matter how a piece of content is used tomorrow (just need a new escaping function for that content type), and prevent awkward things like not letting users use “unsafe” strings as input. JSX and XHP are example templating systems that understand context and esc…

>If a user wants their title to be “hello%0a%3C%25%3D%20File.open%28%27flag.txt%27%29.read%20%25%3E”, so be it. that's a good way to horizontally propagate/reflect XSS and other Code As Data vulnerabilities. better to strip the known-bad/problematic characters https://en.wikipedia.org/wiki/Code_as_data

The known problematic characters are different in json, xml, css, html content, html attributes, MySQL, etc. Unless you have output escaping, it is hard to ensure everything gets caught, no matter how the data enters the system.

Re: The dangers of single line regular expressions

#89
post #13

I pretty much always consider regex expressions as the wrong solution. They're notoriously hard to get right. There's a whole lot of faulty expressions out there for validating email addresses. I prefer to do less validation and let it fail. If the email address is wrong, whatever service you're using for sending emails will just reject it. If you really do need to validate email addresses, use something somebody els…

Regex works very well for what it was originally designed: describing/validating regular languages. It can work ok if your language is simple and almost regular. They work very badly for validating non-regulars languages, even when extensions are added Perl-style to support that. And, unfortunately, most structured formats you might care to valdiate are in fact not regular languages at all.

Email addresses in particular are surprisingly complicated and far from being regular languages. I don't know how commonly real servers support the full feature set, but even if they just support non-ascii names they quickly become a pain.

Re: The dangers of single line regular expressions

#90
post #86

Earlier quoted context omitted.

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulner…

`$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(? I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1]. Edit: also quoting from [2]: > By default, the regular expressions ^ and $ ignore line terminators and only match at the beginn…

OK it seems they changed the doc since. In the docs for JDK 21 we read instead [1]:

> If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters.

Looks like I have some code to fix.

[1] https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi...

Post reply on HN