Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

231–240 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#231

Post by jfhufl »

jfhufl wrote: post #226
jewel wrote: post #174

This has security implications! Example exploitable ruby code: unless person_id =~ /^\d+$/ abort "Bad person ID" end sql = "select * from people where person_id = #{person_id}" In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.

Unsure what you mean? $ ruby -e 'x = "25" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "a25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' no Also, you'd want to use something that parameterizes the query with '?' (I use the Sequel gem) instead of just stuffing it into a sql string.

Well, learned something today after reading a bit further in the thread:

    ruby -e 'x = "a\n25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end'
    yes
Good to know.

Re: Regex character "$" doesn't mean "end-of-string"

#232

Post by cortesoft »

amelius wrote: post #214
FranOntanaya wrote: post #210

Earlier quoted context omitted.

I don't think anyone that writes regex would feel specially challenged by using the Alt+ | Ctrl+Shift+u key combos for unicode entry. Having to escape less things in a pattern would be nice.

Also, code is read more often than it is written.

People say this all the time, but is it really always true? I have a ton of code that I wrote, that just works, and I never really look at it again, at least not with the level of inspection that requires parsing the regex in my head.

Re: Regex character "$" doesn't mean "end-of-string"

#233

Post by keybored »

mbrubeck wrote: post #190
wtetzner wrote: post #152

Earlier quoted context omitted.

So if you have "A" in a file with no newline, there are no lines in that file?

$ echo -n "A" | wc --lines 0

Yep. since wc(1) apparently strictly adheres to what a newline-terminated text file is. This is why plaintext files should end with a newline. :)

https://stackoverflow.com/questions/729692/why-should-text-f...

Re: Regex character "$" doesn't mean "end-of-string"

#234

Post by burntsushi »

danbruc wrote: post #229
burntsushi wrote: post #182

Earlier quoted context omitted.

re.search does not accept a "line." It accepts a "string." There is no pretext in which re.search is meant to only accept a single line. And giving it a `string` with multiple new lines doesn't necessarily mean you want to enable multi-line mode. They are orthogonal things. > Both ^ and $ always match at start or end of lines This is trivially not true, as I showed in my previous example. The haystack `cat\n\n` conta…

re.search does not accept a "line." It accepts a "string." There is no pretext in which re.search is meant to only accept a single line. Sure, it takes a string which might be a line or multiple or whatever. Does not change the fact that $ matches at the end of a line. If you want the end of the string, use \Z. This is trivially not true, as I showed in my previous example. The haystack `cat\n\n` contains two lines a…

> In multi-line mode it matches, in single-line mode it does not because there is a newline between cat and the end of the line. A newline is only a terminating newline if it is the last character, the newline after cat is not a terminating newline. You need cat\n$ or cat\n\n to match.

This only makes sense if re.search accepted a line to search. It doesn't. It accepts an arbitrary string.

I don't think this conversation is going anywhere. Your description of the semantics seems inconsistent and incomprehensible to me.

> A newline is only a terminating newline if it is the last character, the newline after cat is not a terminating newline. You need cat\n$ or cat\n\n to match.

The first `\n` in `cat\n\n` is a terminating newline. There just happens to be one after it.

Like I said, your description makes sense if the input is meant to be interpreted as a single line. And in some contexts (like line oriented CLI tools), that can make sense. But that's not the case here. So your description makes no sense at all to me.

Re: Regex character "$" doesn't mean "end-of-string"

#235

Post by keybored »

knome wrote: post #196
amelius wrote: post #194

Earlier quoted context omitted.

What is driving me nuts is that we have Unicode now, so there is no need to use common characters like $ or ^ to denote special regex state transitions.

the idea of changing a decades old convention to instead use, as I assume you are implying, some character that requires special entry, is beyond silly.

It’s not that silly. You constantly get into escape conundrums because you need to use a metacharacter which is also a metacharacter three levels deep in some embedding.

(But that might not solve that problem? Maybe the problem is mostly about using same-character delimiters for strings.)

And I guess that’s why Perl is so flexible with regards to delimiters and such.

Re: Regex character "$" doesn't mean "end-of-string"

#236

Post by deaddodo »

Gormo wrote: post #184
akdev1l wrote: post #142

Earlier quoted context omitted.

Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.

Suddenly the DOS/Windows solution of using \r\n instead of just \n seems to offer some advantages.

The "Windows way" is the "right way" for a few reasons.

This is definitely not one of them.

Re: Regex character "$" doesn't mean "end-of-string"

#237

Post by halostatue »

jfhufl wrote: post #226
jewel wrote: post #174

This has security implications! Example exploitable ruby code: unless person_id =~ /^\d+$/ abort "Bad person ID" end sql = "select * from people where person_id = #{person_id}" In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.

Unsure what you mean? $ ruby -e 'x = "25" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "a25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' no Also, you'd want to use something that parameterizes the query with '?' (I use the Sequel gem) instead of just stuffing it into a sql string.

You need to make your regex multi-line (`/^\d+$/m`), but that isn't the problem shown. Your query will be searching for `25\n`, not `25` despite your pre-check that it’s a good value.

The second line should always be no, which if you use `\A\d+\z`, it will be.

Re: Regex character "$" doesn't mean "end-of-string"

#238

Post by davidw »

wccrawford wrote: post #65
Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

It's kind of driving me nuts that the article says ^ is "start of string" when it's actually "start of line", just like $ is "end of line". \A is apparently "start of string" like \Z is "end of string".

What with unicode, it'd be fun to have Α and Ω available to make our regexps that much more readable...

Re: Regex character "$" doesn't mean "end-of-string"

#239

Post by keybored »

jerf wrote: post #177
jasonjayr wrote: post #33

Earlier quoted context omitted.

The three big ones I know of are POSIX, Perl/PCRE(aka Perl-Compatible Regular Expression), and Go came along and added used re2, which is a bit different from the first too. A lot of systems implemented PCRE, including JavaScript, since Perl extended the POSIX system with many useful extensions. IIRC, re2 tries to reign in on some of the performance issues and quirks the original systems had, while implementing the w…

POSIX and PCRE are arguably redundant. They both support backreferences, which puts very significant constraints on their implementations. PCRE is at least functionally a superset of POSIX, whether or not there's some quirky thing POSIX supports that PCRE does not. re2 adds a legitimate option to the menu of using NDFAs, which have the disadvantage of not supporting backreferences, but have the advantage of having co…

> RE engines don't quite engender the same emotions as programming languages as a whole, but this is not cheerleading, this is a sober engineering assessment.

Good on you.

Re: Regex character "$" doesn't mean "end-of-string"

#240

Post by aftbit »

Wait, in non-multiline mode, it only matches _one_ trailing newline? And not any other whitespace, including \r or \r\n? That is indeed surprising behavior. Why? Why not just make it end of string like the author expected?

    >>> import re
    >>> bool(re.search('abc$', 'abc'))
    True
    >>> bool(re.search('abc$', 'abc\n'))
    True
    >>> bool(re.search('abc$', 'abc\n\n'))
    False
    >>> bool(re.search('abc$', 'abc '))
    False
    >>> bool(re.search('abc$', 'abc\t'))
    False
    >>> bool(re.search('abc$', 'abc\r'))
    False
    >>> bool(re.search('abc$', 'abc\r\n'))
    False
Post reply on HN