Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The dangers of single line regular expressions

greg.molnar.io

Post reply on HN
Print view View original

121–130 of 133 posts

Re: The dangers of single line regular expressions

#121

Post by js2 »

medstrom wrote: post #61
js2 wrote: post #39

Earlier quoted context omitted.

The potential trouble with $ (even in single-line mode) is that it matches the end of a string BOTH with AND without a newline at the end. If you're using it to ensure the string has no newline before doing something with it, this can lead to trouble. $ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo\n") else "no")' yes…

Correct me if I'm wrong, but if you extract a capture group (^foo$), you would get "foo" without the "\n", right? If so, it is not "matching the end of a string" at all. Just end of line. That's exactly as expected in single-line mode, so it's good. May mismatch your expectations in multi-line mode though.

That's right. It all depends on what you're doing with the input string after the match. The point is to be aware of the nuance and to communicate that clearly in the code in cases where it matters.

Re: The dangers of single line regular expressions

#122

Post by int_19h »

tsimionescu wrote: post #95
ec109685 wrote: post #53

Escape the output based on the context a string is being used in versus trying to sanitize for all use cases on input. This will guarantee that you’re safe no matter how a piece of content is used tomorrow (just need a new escaping function for that content type), and prevent awkward things like not letting users use “unsafe” strings as input. JSX and XHP are example templating systems that understand context and esc…

The output is not the problem here, it is the input. And, if you can get away with, accepting a small set of known-safe characters is much safer than accepting any character and hoping it will be properly escaped at every level. When the user hands you a string and you then pass this down to other bits of code, you can't know if it will be used in an SQL query, a regex, in an error message that will be rendered into…

> accepting a small set of known-safe characters is much safer than accepting any character and hoping it will be properly escaped at every level

It's also how you end up with apps that people can't use because they reject their perfectly valid legal name, address etc.

Re: The dangers of single line regular expressions

#123

Post by tsimionescu »

int_19h wrote: post #122
tsimionescu wrote: post #95

Earlier quoted context omitted.

The output is not the problem here, it is the input. And, if you can get away with, accepting a small set of known-safe characters is much safer than accepting any character and hoping it will be properly escaped at every level. When the user hands you a string and you then pass this down to other bits of code, you can't know if it will be used in an SQL query, a regex, in an error message that will be rendered into…

> accepting a small set of known-safe characters is much safer than accepting any character and hoping it will be properly escaped at every level It's also how you end up with apps that people can't use because they reject their perfectly valid legal name, address etc.

Absolutely, that's why I said "if you can get away with it". There are situations where this is ok - for example, a company user ID. Even for names, it is often perfectly fine to require someone to spell their name with Latin characters.

For example, even if your name is officially 鳥山 in Japan, you will have to spell it out as Toriyama when you leave Japan, both in formal and informal settings, on paper just as much as in electronic forms, since no one would be able to understand it otherwise. And similarly, if your name is Smith, in Japan you will often have to spell it (and sometimes even pronounce it) スミス.

Re: The dangers of single line regular expressions

#124

Post by tsimionescu »

ec109685 wrote: post #117
tsimionescu wrote: post #95

Earlier quoted context omitted.

The output is not the problem here, it is the input. And, if you can get away with, accepting a small set of known-safe characters is much safer than accepting any character and hoping it will be properly escaped at every level. When the user hands you a string and you then pass this down to other bits of code, you can't know if it will be used in an SQL query, a regex, in an error message that will be rendered into…

What was actually invalid about that input? Why shouldn’t that html escaped string be shown as is to the user? I guess I would tweak my first comment and say input filtering is not enough. You must do output filtering to truly be safe.

I don't get how you use the term "output" here. However you put it, the problem was feeding the user's input to ERB.new(). The most general solution would have been something that accepted any string, but escaped it properly for ERB.new, I think we all agree on that. But that escaping still needs to be done on the input, not the output of ERB.new. If they were able to inject code there, the output doesn't even matter: you've already lost by the time you get a return, the malicious payload has already run, it doesn't matter what its output was.

Re: The dangers of single line regular expressions

#125

Post by ec109685 »

tsimionescu wrote: post #124
ec109685 wrote: post #117

Earlier quoted context omitted.

What was actually invalid about that input? Why shouldn’t that html escaped string be shown as is to the user? I guess I would tweak my first comment and say input filtering is not enough. You must do output filtering to truly be safe.

I don't get how you use the term "output" here. However you put it, the problem was feeding the user's input to ERB.new(). The most general solution would have been something that accepted any string, but escaped it properly for ERB.new, I think we all agree on that. But that escaping still needs to be done on the input, not the output of ERB.new. If they were able to inject code there, the output doesn't even matter…

My point was that the web page that the researcher compromised takes input from user and creates a neon version of it.

It should be just fine to pass in any character to the site, so a regex deny list is the wrong approach.

Re: The dangers of single line regular expressions

#127

Post by banish-m4 »

I once had to explain this class of security vulnerability to IC5-IC7 senior engineers.

0. There is no universal regex language but many.

1. Perl-like ones (Ruby, Perl, and PCRE1/2) contain additional hidden traps.

2. You must vigorously match untrusted input to assume it to include invalid unicode, control characters, and other oddities.

3. You should replicate frontend and backend validations to ensure they are always exactly consistent and correct, preferably through fuzzing and/or property testing.

Re: The dangers of single line regular expressions

#128

Post by _wire_ »

sublinear wrote: post #126

> Hire me for a penetration test When does the blogspam end?

Aye.

"Consider every ambiguity of technology as a personal marketing opportunity."

The true topic at hand is that text substitution in scripted services is an eternal hazard of code injection.

The point that regex-based input sanitization doesn't work because everyone misunderstands the token semantics for string termination is made to look like a marvelous mitigation, but this teaching on regex is distracting from an unavoidable hazard of scripting.

Good news for the contractor: he appears like Jesus to shine the Lord's light on the sin of the fathers while dancing by the moral hazard of the priesthood.

Elsewhere another instance of the OP is a service provider pushing business solutions based on the ease of use of scripted service frameworks ("Input sanitization is as simple as a regex!)

These hazards are going to get much worse as AI merges the causes of and solutions to these ambiguities into the same semantic mush.

Re: The dangers of single line regular expressions

#129

Post by dwheeler »

jasonlotito wrote: post #120
dwheeler wrote: post #49

Earlier quoted context omitted.

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulner…

So, you are technically correct when you say PHP accepts a trailing newline, but it doesn't mean it refutes the comment and the context we are discussing. This is easily demonstrated with an example. versus versus Which all makes sense, as by default PHP doesn't operate in multiline mode. So, by default, PHP is not going to fall prey to the same problem being discussed here. In addition, the first \n would be apart o…

I think we're using different definitions for "end of string".

In PHP, the following is considered true:

> var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n"));

That is clear proof that "$" does NOT just match the end of the string; it also accepts an extra newline at the end of the string. In PHP you need to use \z if you want to match the end of the string, or use the "D" flag when using "$".

That definition of "$" is often reasonable when you read files a line-at-a-time from a file, which is why Perl changed its definition. However, PHP is often used for server-side web applications. In this case, you are often NOT reading a line-at-a-time from a file. In such cases, allowing an extra newline at the end could be disastrous. The MediaWiki code (written in PHP) deals with this by adding the "D" flag when it uses "$", but I'm not sure it always uses it, and I doubt all PHP programs use this flag when they should.

Re: The dangers of single line regular expressions

#130

Post by dwheeler »

phyzome wrote: post #98
dwheeler wrote: post #49

Earlier quoted context omitted.

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulner…

Interesting that a trailing newline is accepted. Not as bad as what's in the post, at least. Definitely worth breaking out which languages do which of those, though! Python, for instance, only accepts a trailing newline but not additional chars beyond that. I don't think Java should be in your first list, though? Pattern.matches("^foo$", "foo\n") returns false.

Which version of Java (JDK) are you using? Which implementation?

If that's true, then I fear the answer for Java may vary. The O'Reilly book on Regular Expressions, and the JDK documentation for version 21, say clearly that $ permits an optional \n at the end. The Java 8 documentation is murky, and maybe Java 8 is different.

Post reply on HN