Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

281–290 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#281

Post by burntsushi »

danbruc wrote: post #265
burntsushi wrote: post #234

Earlier quoted context omitted.

> In multi-line mode it matches, in single-line mode it does not because there is a newline between cat and the end of the line. A newline is only a terminating newline if it is the last character, the newline after cat is not a terminating newline. You need cat\n$ or cat\n\n to match. This only makes sense if re.search accepted a line to search. It doesn't. It accepts an arbitrary string. I don't think this conversa…

This only makes sense if re.search accepted a line to search. It doesn't. It accepts an arbitrary string. Which is fine because lines are a subset of strings. And whether you want your input treated as a line or a string is decided by your pattern, use ^ and $ and it will be treated as a line, use \A and \Z and it will be treated as a string. The first `\n` in `cat\n\n` is a terminating newline. There just happens to…

> And whether you want your input treated as a line or a string is decided by your pattern, use ^ and $ and it will be treated as a line, use \A and \Z and it will be treated as a string.

Where is this semantic explained in the `re` module docs?

This is totally and completely made up as far as I can tell.

This also seems entirely consistent with my rebuttal:

Me: What you're saying makes sense if condition foo holds.

You: Condition foo holds.

This is uninteresting to me because I see no reason to believe that condition foo holds. Where condition foo is "the input to re.search is expected to be a single line." Or more precisely, apparently, "the input to re.search is expected to be a single line when either ^ or $ appear in the pattern." That is totally bonkers.

> but it explains why things behave the way they do

Firstly, I am not debating with you about the historical reasoning for this. Secondly, I am providing a commentary on the semantics themselves (they suck) and also on your explanation of them in today's context (it doesn't make sense). Thirdly, I am not making a prescriptive argument that established regex engines should change their behavior in any way.

If you're looking to explain why this semantic is the way it is, then I'd expect writing from the original implementors of it. Probably in Perl. I wouldn't at all be surprised if this was an "oops" or if it was implemented in a strictly-line-oriented context, and then someone else decided to keep it unthinkingly when they moved to a non-line-oriented context. From there, compatibility takes over as a reason for why it's with us today.

Re: Regex character "$" doesn't mean "end-of-string"

#282

Post by lolc »

berkes wrote: post #95
username_my1 wrote: post #61

Earlier quoted context omitted.

regex is useful but chatgpt is amazing at it, so why spend a minute keeping such useless knowledge in mind. if you know where to find something no point in knowing it.

> if you know where to find something no point in knowing it. Nonsense. And you know it. First, you need to know what to find, before knowing where to find it. And knowing what to find requires intricate knowledge of the thing. Not intricate implementation details, but enough to point yourself in the right direction. Secondly, you need to know why to find thing X and not thing Y. If anything, ChatGPT is even worse th…

Funny for me there have been instances where the LLM did push back. I had a plan of how to solve something and tasked the LLM with a draft implementation. It kept producing another solution which I kept rejecting and specifying more details so it wouldn't stray. In the end I had to accept that my solution couldn't work, and that the proposed one was acceptable. It's going to happen again, because it often comes up with inferior solutions so I'm not very open to the reverse situation.

Re: Regex character "$" doesn't mean "end-of-string"

#283

Post by HideousKojima »

keybored wrote: post #244
jbverschoor wrote: post #37

Earlier quoted context omitted.

The whole \r is archaic. It doesn't even behave properly in most cases. Just use \n everywhere and bite the lemon for a short while to fix your problems. And if you believe \r\n is the way to go, please make sure \n\r also works as they should have the same results. (or \r\n\r\r\r\r for that matter)

Why did they even decide to use two characters for the end of line? Seems bizarre. I could have imagined that `\r` and `\n` was a tossup. But why both?

Typewriters is why

Re: Regex character "$" doesn't mean "end-of-string"

#284

Post by kataklasm »

Izkata wrote: post #183
absoluteunit1 wrote: post #110

Earlier quoted context omitted.

I’ve always thought that as well; mostly due to Vim though. ^ - takes you to start of line $ - takes you to end of line

^ actually takes you to the first non-whitespace character in the line in vim. For start of line you want 0

I don't have (n)vi(m) open right now but I think this only applies to prepending spaces. For prepending tabs, 0 will take you to the first non-tab character as well.

Re: Regex character "$" doesn't mean "end-of-string"

#285

Post by layer8 »

dumbo-octopus wrote: post #277
layer8 wrote: post #153

Earlier quoted context omitted.

I generally agree, but the two consecutive dots (or leading/trailing dots) are an example that would very likely be a typo and that you wouldn’t particularly want to send. Similar for unbalanced quotes, angle brackets, and other grammar elements.

I wonder whether simply (regex) replacing a sequence of .'s with a single one as part of a post-processing step would be effective.

That would be bad form, IMO. The user may have typed john..kennedy@example.com by mistake instead of john.f.kennedy@example.com, and now you’ll be sending their email to john.kennedy@example.com. Similar for leading or trailing dots. You can’t just decide what a user probably meant, when they type in something invalid.

Re: Regex character "$" doesn't mean "end-of-string"

#286

Post by dr-smooth »

jfhufl wrote: post #226
jewel wrote: post #174

This has security implications! Example exploitable ruby code: unless person_id =~ /^\d+$/ abort "Bad person ID" end sql = "select * from people where person_id = #{person_id}" In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.

Unsure what you mean? $ ruby -e 'x = "25" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "a25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' no Also, you'd want to use something that parameterizes the query with '?' (I use the Sequel gem) instead of just stuffing it into a sql string.

    $ ruby -e 'x = "25\n; delete from people" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end'
    yes

Re: Regex character "$" doesn't mean "end-of-string"

#287

Post by danbruc »

burntsushi wrote: post #281
danbruc wrote: post #265

Earlier quoted context omitted.

This only makes sense if re.search accepted a line to search. It doesn't. It accepts an arbitrary string. Which is fine because lines are a subset of strings. And whether you want your input treated as a line or a string is decided by your pattern, use ^ and $ and it will be treated as a line, use \A and \Z and it will be treated as a string. The first `\n` in `cat\n\n` is a terminating newline. There just happens to…

> And whether you want your input treated as a line or a string is decided by your pattern, use ^ and $ and it will be treated as a line, use \A and \Z and it will be treated as a string. Where is this semantic explained in the `re` module docs? This is totally and completely made up as far as I can tell. This also seems entirely consistent with my rebuttal: Me: What you're saying makes sense if condition foo holds.…

I quoted the section from the Python module here. [1]

If you do not specify multi-line, bar$ matches a lines ending in bar, either foobar\n or foobar if the terminating newline has been removed or does not exist. If you specify multi-line, then it will also match at every bar\n within the string. So it either treats your input as a single line or as multiple lines. You can of course not specify multi-line and still pass in a string with additional newlines within the string, but then those newlines will be treated more or less as any other character, bar$ will not match bar\n\n. The exception is that dot will not match them except you set the single-line/dot-all flag, bar\n$ will match bar\n\n but bar.$ will not unless you specify the single-line/dot-all flag.

I would even agree with you that it seems a bit weird. If you have a proper line without additional newlines in the middle, then multi-line behaves exactly like not multi-line. Not multi-line only behaves differently if you confront it with multiple lines and I have no good idea how you would end up in a situation where you have multiple lines and want to treat them as one unit but still treat the entire thing as if it was a line.

[1] https://news.ycombinator.com/item?id=39765086

Re: Regex character "$" doesn't mean "end-of-string"

#288

Post by librasteve »

I am surprised that the OP does not include perl5 in their table.

In raku (aka perl6) Regexes were reinvented by Larry Wall (the creator of perl which made perlRE the de facto regex standard)

Here's what he does with $:

(https://docs.raku.org/language/regexes#Start_of_string_and_e...)

* The $ anchor only matches at the end of the string

* The $$ anchor matches at the end of a logical line. That is, before a newline character, or at the end of the string when the last character is not a newline character.

Re: Regex character "$" doesn't mean "end-of-string"

#289

Post by HideousKojima »

jbverschoor wrote: post #262
mnau wrote: post #256

Earlier quoted context omitted.

Likely compatibility bugs going back decades (70s?). Probably with some terminal/teletype. \r - returned teletype head to the start of a line \n - move paper one line down > The sequence CR+LF was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line…

It’s similar to an old school typewriter. The handle does 2 things: return and feed. You can also just return by not pulling all the way or the other way around depending on the design

Which also let you do strikethrough and similar effects by typing over a line you already typed

Re: Regex character "$" doesn't mean "end-of-string"

#290

Post by burntsushi »

danbruc wrote: post #287
burntsushi wrote: post #281

Earlier quoted context omitted.

> And whether you want your input treated as a line or a string is decided by your pattern, use ^ and $ and it will be treated as a line, use \A and \Z and it will be treated as a string. Where is this semantic explained in the `re` module docs? This is totally and completely made up as far as I can tell. This also seems entirely consistent with my rebuttal: Me: What you're saying makes sense if condition foo holds.…

I quoted the section from the Python module here. [1] If you do not specify multi-line, bar$ matches a lines ending in bar, either foobar\n or foobar if the terminating newline has been removed or does not exist. If you specify multi-line, then it will also match at every bar\n within the string. So it either treats your input as a single line or as multiple lines. You can of course not specify multi-line and still p…

The docs do not say what you're saying. Your phrasing is completely different, and the part where "if ^/$ are in the pattern then the haystack is treated as a single line" is completely made up. As far as I can tell, that's your rationalization for how to make sense of this behavior. But it is not a story supported by the actual regex engine docs. The actual docs say, "^ matches only at the beginning of the string, and $ matches only at the end of the string and immediately before the newline (if any) at the end of the string." The docs do not say, "the string is treated as a single line when ^/$ are used in the pattern." That's your phrasing, not anyone else's. That's your story, not theirs.

I still have not seen anything from you that makes sense of the behavior that `cat$` does not match `cat\n\n`. Like, I realize you've tried to explain it. But your explanation does not make sense. That's because the behavior is strange.

The only actual way to explain the behavior of $ is what the `re` docs say: it either matches at the end of the string or just before a `\n` that appears at the end of the string. That's it.

Post reply on HN