Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

181–190 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#181

Post by SAI_Peregrinus »

POSIX regexes and Python regexes are different. In general, you need to reference the regex documentation for your implementation, since the syntax is not universal.

Per POSIX chapter 9[1]:

9.2 … "The use of regular expressions is generally associated with text processing. REs (BREs and EREs) operate on text strings; that is, zero or more characters followed by an end-of-string delimiter (typically NUL). Some utilities employing regular expressions limit the processing to lines; that is, zero or more characters followed by a ."

and 9.3.8 … "A ( '$' ) shall be an anchor when used as the last character of an entire BRE. The implementation may treat a as an anchor when used as the last character of a subexpression. The shall anchor the expression (or optionally subexpression) to the end of the string being matched; the can be said to match the end-of-string following the last character."

combine to mean that $ may match the end of string OR the end of the line, and it's up to the utility (or mode) to define which. Most of the common utilities (grep, sed, awk, Python, etc) treat it as end of line by default, since they operate on lines by default.

THERE IS NO SINGLE UNIVERSAL REGULAR EXPRESSION SYNTAX. You cannot reliably read or write regular expressions without knowing which language & options are being used.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

Re: Regex character "$" doesn't mean "end-of-string"

#182

Post by burntsushi »

danbruc wrote: post #158
burntsushi wrote: post #118

Earlier quoted context omitted.

> Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $ No, you're not, except for this weird corner case where `$` can match before the last `\n` in a string. It's not just any `\n` that non-multiline `$` can match before. It's when it's the last `\n` in the string. See: >>> re.search('cat$', 'cat\n') >>> re.search('cat$', 'cat\n\n') >>> This is weird behavior. I…

But that is exactly what it means, the end of the line is before the terminating newline or at the end of the string if there is no terminating newline. Both ^ and $ always match at start or end of lines, \A and \Z match at the start or end of the string. The difference between multi-line and not is whether or not internal newlines end and start lines, it does not change the semantics from end of line to end of strin…

re.search does not accept a "line." It accepts a "string." There is no pretext in which re.search is meant to only accept a single line. And giving it a `string` with multiple new lines doesn't necessarily mean you want to enable multi-line mode. They are orthogonal things.

> Both ^ and $ always match at start or end of lines

This is trivially not true, as I showed in my previous example. The haystack `cat\n\n` contains two lines and the regex `cat$` says it should match `cat` followed by the "end of a line" according to your definition. Yet it does not match `cat` followed by the end of a line in `cat\n\n`. And it does not do so in Python or in any other regex engine.

You're trying to square a circle here. It can't be done.

Can you make sense of, historically, why this choice of semantics was made? Sure. I bet you can. But I can still evaluate the choice on its own merits today. And I did when I made the regex crate.

> but I am not convinced that this would be an easy and obvious choice to have breaking change.

Rust's regex crate, Go's regexp package and RE2 all reject this whacky behavior. As the regex crate maintainer, I don't think I've ever seen anyone complain. Not once. This to me suggests that, at minimum, making `$` and `\z` equivalent in non-multiline mode is a reasonable choice. I would also argue it is the better and more sensible approach.

Whether other regex engines should have a breaking change or not to change the meaning of `$` is an entirely different question completely. That is neither here nor there. They absolutely will not be able to make such a change, for many good reasons.

Re: Regex character "$" doesn't mean "end-of-string"

#183

Post by Izkata »

absoluteunit1 wrote: post #110
Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

I’ve always thought that as well; mostly due to Vim though. ^ - takes you to start of line $ - takes you to end of line

^ actually takes you to the first non-whitespace character in the line in vim. For start of line you want 0

Re: Regex character "$" doesn't mean "end-of-string"

#184

Post by Gormo »

akdev1l wrote: post #142
Bjartr wrote: post #140

Earlier quoted context omitted.

The line delimiter is a newline. If you have a file containing `A\nB\nC` in a file, the file is three lines long. I guess it could be argued that a file containing `A\nB\nC\n` has four lines, with the fourth having zero length. That a regex is applying to an in memory string vs a file doesn't feel to me like it should have different semantics. Digging into the history a little, it looks like regexes were popularized…

Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.

Suddenly the DOS/Windows solution of using \r\n instead of just \n seems to offer some advantages.

Re: Regex character "$" doesn't mean "end-of-string"

#186

Post by burntsushi »

jerf wrote: post #177
jasonjayr wrote: post #33

Earlier quoted context omitted.

The three big ones I know of are POSIX, Perl/PCRE(aka Perl-Compatible Regular Expression), and Go came along and added used re2, which is a bit different from the first too. A lot of systems implemented PCRE, including JavaScript, since Perl extended the POSIX system with many useful extensions. IIRC, re2 tries to reign in on some of the performance issues and quirks the original systems had, while implementing the w…

POSIX and PCRE are arguably redundant. They both support backreferences, which puts very significant constraints on their implementations. PCRE is at least functionally a superset of POSIX, whether or not there's some quirky thing POSIX supports that PCRE does not. re2 adds a legitimate option to the menu of using NDFAs, which have the disadvantage of not supporting backreferences, but have the advantage of having co…

I'll add two notes to this:

* Finite automata based regex engines don't necessarily have to be slower than backtracking engines like PCRE. Go's regexp is in practice slower in a lot of cases, but this is more a property of its implementation than its concept. See: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa... --- Given "sufficient" implementation effort (~several person years of development work), backtrackers and finite automata engines can both perform very well, with one beating the other in some cases but not in others. It depends.

* Fun fact is that if you're iterating over all matches in a haystack (e.g., Go's `FindAll` routines), then you're susceptible to O(m * n^2) search time. This applies to all regex engines that implement some kind of leftmost match priority. See https://github.com/BurntSushi/rebar?tab=readme-ov-file#quadr... for a more detailed elaboration on this point.

Re: Regex character "$" doesn't mean "end-of-string"

#187

Post by jerf »

burntsushi wrote: post #186
jerf wrote: post #177

Earlier quoted context omitted.

POSIX and PCRE are arguably redundant. They both support backreferences, which puts very significant constraints on their implementations. PCRE is at least functionally a superset of POSIX, whether or not there's some quirky thing POSIX supports that PCRE does not. re2 adds a legitimate option to the menu of using NDFAs, which have the disadvantage of not supporting backreferences, but have the advantage of having co…

I'll add two notes to this: * Finite automata based regex engines don't necessarily have to be slower than backtracking engines like PCRE. Go's regexp is in practice slower in a lot of cases, but this is more a property of its implementation than its concept. See: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa... --- Given "sufficient" implementation effort (~several person years of development work), b…

Excellent, thank you.

Re: Regex character "$" doesn't mean "end-of-string"

#188

Post by Izkata »

Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

Same here; when I saw the title I was like "well obviously not, where did you hear that?"

In nearly two decades of using regex I think this might be the first time I've heard of $ being end of string. It's always been end of line for me.

Re: Regex character "$" doesn't mean "end-of-string"

#189

Post by Terretta »

perlgeek wrote: post #68

Raku (formerly Perl 6) has picked ^ and $ for start-of-string and end-of-string, and has introduced ^^ and $$ for start-of-line and end-of-line. No multi line mode is available or necessary. (There's also \h for horizontal and \v for vertical whitespace) That's one of the benefits of a complete rethink/rewrite, you can learn from the fact that the old behavior surprised people.

And this is why this curmudgeon can't use Perl 6[^1]. It randomly shuffles the line noise we learned over decades.

It seems so obvious that's the opposite of what they should have defaulted to, that it clearly should have been ^ and $ for lines, and ^^ and $$ for the string, since like ((1)(2)(3)):

^^line1$\n^line2$\n^line3$\n$

[1]: That, and it's not anywhere, while Perl 5 is everywhere.

Re: Regex character "$" doesn't mean "end-of-string"

#190

Post by mbrubeck »

wtetzner wrote: post #152
akdev1l wrote: post #142

Earlier quoted context omitted.

Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.

So if you have "A" in a file with no newline, there are no lines in that file?

    $ echo -n "A" | wc --lines
    0
Post reply on HN