Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

321–330 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#321

Post by burntsushi »

danbruc wrote: post #293
burntsushi wrote: post #290

Earlier quoted context omitted.

The docs do not say what you're saying. Your phrasing is completely different, and the part where "if ^/$ are in the pattern then the haystack is treated as a single line" is completely made up. As far as I can tell, that's your rationalization for how to make sense of this behavior. But it is not a story supported by the actual regex engine docs. The actual docs say, "^ matches only at the beginning of the string, a…

You are right, it is my wording, I replaced end of string or before newline as the last character with end of line because that is what this means. You could also write that into the documentation but then you would have to also explain what end of line means. And I will grant you that I might be wrong, that the behavior is only accidentally identical to matching the end of a line but that the true reason for it is d…

> I guess you want the regex engine

Ignoring compatibility concerns, I would want the regex engine to behave the same way RE2, Go's regexp package and Rust's regex engine behave. I remember specifically considering Cox's decision ~10 years ago when writing the initial implementation of the regex crate. I thought Perl's (and Python's) behavior on this point was whacky then and I still think it's whacky now. So I followed RE2's semantics.

The OP is right to be surprised by this. And folks will continue to be surprised by it for eternity because it's an extremely subtle corner case that doesn't have a consistent story explaining its behavior. (I know you have proffered one, but I don't find it consistent in the context of a general purpose regex engine that searches arbitrary strings and not just lines.)

Of course, compatibility is a trump card here. I've acknowledged that. Changing this behavior now would be too hard. The best you could probably do is some kind of migration, where you provide the more "sensible" behavior behind an opt-in flag. And then maybe Python 4 enables it by default. But it's a lot of churn, and while people will continue to be confounded by this so long as the behavior exists, it probably isn't a Huge & Common Deal In Practice. So it may not be worth fixing. But if you're starting from scratch? Yes, please don't implement $ this way. It should match the end of the string when 'm' is disabled and the end of any line (including end of string and possibly being Unicode aware, depending on how much you care about that) when 'm' is enabled.

Re: Regex character "$" doesn't mean "end-of-string"

#322

Post by Izkata »

qu4z-2 wrote: post #312
kataklasm wrote: post #284

Earlier quoted context omitted.

I don't have (n)vi(m) open right now but I think this only applies to prepending spaces. For prepending tabs, 0 will take you to the first non-tab character as well.

Vim takes me to the first character in the line (the first tab), but displays the cursor on the last gridsquare the tab's width covers.

If you have "set list" to make non-space whitespace visible, it'll go to the leftmost position. I did it long ago along with "set listchars=trail:.,tab:>-" so I can see not only where tabs are, but also their size/alignment without causing the text to shift.

Re: Regex character "$" doesn't mean "end-of-string"

#323

Post by sltkr »

Izkata wrote: post #320
sltkr wrote: post #315

Earlier quoted context omitted.

> Python's behavior [..] is the common behavior. The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $). Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behav…

> - in vim, /^M/ does not match any lines But /\n/ does

Thanks for the correction! That's interesting.

Re: Regex character "$" doesn't mean "end-of-string"

#324

Post by emporas »

int_19h wrote: post #313
cortesoft wrote: post #230

Earlier quoted context omitted.

I write regexes all the time, and I don't know if I would be CHALLENGED by that, but it would be annoying. Escaping things is trivial, and since you do it all the time it is not anything extra to learn. Having to remember bespoke keystrokes for each character is a lot more to learn.

Regexes are one case where I think it's already extremely unbalanced wrt being easy to write but hard to read. Using stuff like special Unicode chars for this would make them harder to write but easier to read, which sounds like a fair deal to me. In general, I'd say that regexes should take time and effort to write, just because it's oh-so-easy to write something that kinda sorta works but has massive footguns. I wo…

Regexes originate from Perl, or they were popularized by Perl if i got this right. In Perl readable code is not ranked as one of it's top 100 priorities. Regexes could originate from J and situation could be even worse though!

Re: Regex character "$" doesn't mean "end-of-string"

#325

Post by dbdudbdiddjc »

frame_ranger wrote: post #259
Izkata wrote: post #188

Earlier quoted context omitted.

Same here; when I saw the title I was like "well obviously not, where did you hear that?" In nearly two decades of using regex I think this might be the first time I've heard of $ being end of string. It's always been end of line for me.

You couldn’t write a post like this if you didn’t start with a strawman.

It’s not a straw man, it’s accurate in many contexts.

Re: Regex character "$" doesn't mean "end-of-string"

#326

Post by gordonfish »

toyg wrote: post #138
tyingq wrote: post #112

Seems odd to leave Perl off the list, given it's regex related. Here's the explanation for $ in the perlre docs: $ Match the end of the string (or before newline at the end of the string; or before any newline if /m is used)

Yeah, omitting what is arguably the language most associated with regexes seems a bit of an oversight. I guess it shows how far off the radar Perl currently is.

> I guess it shows how far off the radar Perl currently is.

This is a serious misconception. Perl is far, far from dead. The constant activity of the gargantuan CPAN library more than demonstrates very much the opposite.

I would say Perl and its community has done quite well considering it hasn't had the same mountain of corporate funds thrust into it like more highlighted have. Mainstream ain't everything.

Re: Regex character "$" doesn't mean "end-of-string"

#327

Post by gordonfish »

TillE wrote: post #228
toyg wrote: post #138

Earlier quoted context omitted.

Yeah, omitting what is arguably the language most associated with regexes seems a bit of an oversight. I guess it shows how far off the radar Perl currently is.

PHP uses PCRE, so it more or less serves as a stand-in for Perl in this case.

Sort of, though PCRE is not a 100% replacement for regex in Perl proper; the former lacks some features of the latter.

Re: Regex character "$" doesn't mean "end-of-string"

#328

Post by al_borland »

danbruc wrote: post #121
masklinn wrote: post #117

Earlier quoted context omitted.

> It is start and end of line. You seem to have redefined “line” as “not a line”. > The confusion I’m sure redefining “line” as “nothing like what anyone reasonable would interpret as a line” will help a lot and right clear up the confusion.

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.

I suppose this is why certain config files are strict about ending with a newline, without it, the last line wouldn’t technically be a line?

Re: Regex character "$" doesn't mean "end-of-string"

#329

Post by zeehio »

danbruc wrote: post #121
masklinn wrote: post #117

Earlier quoted context omitted.

> It is start and end of line. You seem to have redefined “line” as “not a line”. > The confusion I’m sure redefining “line” as “nothing like what anyone reasonable would interpret as a line” will help a lot and right clear up the confusion.

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.

I'm now amused by the idea of a malicious compliance linter telling me "you have an issue at your code, on line NaN"

Re: Regex character "$" doesn't mean "end-of-string"

#330

Post by IshKebab »

danbruc wrote: post #293
burntsushi wrote: post #290

Earlier quoted context omitted.

The docs do not say what you're saying. Your phrasing is completely different, and the part where "if ^/$ are in the pattern then the haystack is treated as a single line" is completely made up. As far as I can tell, that's your rationalization for how to make sense of this behavior. But it is not a story supported by the actual regex engine docs. The actual docs say, "^ matches only at the beginning of the string, a…

You are right, it is my wording, I replaced end of string or before newline as the last character with end of line because that is what this means. You could also write that into the documentation but then you would have to also explain what end of line means. And I will grant you that I might be wrong, that the behavior is only accidentally identical to matching the end of a line but that the true reason for it is d…

Dunno if you noticed who you are debating with here... :-D
Post reply on HN