Regex character "$" doesn't mean "end-of-string" • Hacker News Board

Re: Regex character "$" doesn't mean "end-of-string"

#111

Post by danbruc » Wed, Mar 20, 2024, 11:36 AM UTC

masklinn wrote: post #83

wccrawford wrote: post #65

Earlier quoted context omitted.

It's kind of driving me nuts that the article says ^ is "start of string" when it's actually "start of line", just like $ is "end of line". \A is apparently "start of string" like \Z is "end of string".

It’s not start of line though, unless the engine is in multiline mode. Here is the documentation for Python’s re for instance: > Matches the start of the string, and in MULTILINE mode also matches immediately after each newline. Or JavaScript: > An input boundary is the start or end of the string; or, if the m flag is set, the start or end of a line. \A and \Z are start/end of input regardless of mode… when they’re a…

It is start and end of line. [1]

Usually ^ matches only at the beginning of the string, and $ matches only at the end of the string and immediately before the newline (if any) at the end of the string. When this flag is specified, ^ matches at the beginning of the string and at the beginning of each line within the string, immediately following each newline. Similarly, the $ metacharacter matches either at the end of the string and at the end of each line (immediately preceding each newline).

In single-line [2] mode, the line starts at the start of the string and ends at the end of the line where the end of the line is either the end of the string if there is no terminating newline or just before the final newline if there is a terminating newline.

In multi-line mode a new line starts at the start of the string and after each newline and ends before each newline or at the end of the string if the last line has no terminating newline.

The confusion is that people think that they are in string-mode if they are not in multi-line mode but they are not, they are in single-line mode, ^ and $ still use the semantics of lines and a terminating newline, if present, is still not part of the content of the line.

With \n\n\n in single-line mode the non-greedy ^(\n+?)$ will capture only two of the newlines, the third one will be eaten by the $. If you make it greedy ^(\n+)$ will capture all three newlines. So arguably the implementations that do not match cat\n with cat$ are the broken ones.

[1] https://docs.python.org/3/howto/regex.html#more-metacharacte...

[2] I am using single-line to mean not multi-line for convenience even though single-line already has a different meaning.

Re: Regex character "$" doesn't mean "end-of-string"

#112

Post by tyingq » Wed, Mar 20, 2024, 11:36 AM UTC

Seems odd to leave Perl off the list, given it's regex related.

Here's the explanation for $ in the perlre docs:

  $   Match the end of the string                 
      (or before newline at the end of the      
      string; or before any newline if /m is     
      used)

Re: Regex character "$" doesn't mean "end-of-string"

#113

Post by burntsushi » Wed, Mar 20, 2024, 11:43 AM UTC

ghusbands wrote: post #18

> Note: The table of data was gathered from regex101.com, I didn't test using the actual runtimes. Has anyone confirmed this behaviour directly against the runtimes/languages? Newlines at the end of a string are certainly something that could get lost in transit inside an online service involving multiple runtimes.

Yes, and with more regex engines: https://github.com/BurntSushi/rebar/blob/177f5d55e916964b9c4...

Beyond what's in the OP, that includes RE2, Hyperscan, D's std.regex, ICU, Perl, Python's third party `regex` package, and `regress`.

Re: Regex character "$" doesn't mean "end-of-string"

#114

Post by riffraff » Wed, Mar 20, 2024, 11:43 AM UTC

pjc50 wrote: post #106

bregma wrote: post #70

Earlier quoted context omitted.

The ISO/IEC 14882 C++ standard library mandates [0] implementations for six de jure standard regex grammars: IEEE Std 1003.1-2008 (POSIX) [1] BRE, ERE, awk, grep, and egrep and ECMA-262 EcmaScript 3 [2]. So, yes, at least someone (me) considers regex to be standardized in several published de jure standards. [0] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3690.pdf#chapter.28 [1] https://pubs.opengroup.o…

"At least six different standards" is an XKCD comic, not a standard.

"The nice thing about standards is that you have so many to choose from." - Andrew Tanenbaum (or Grace Hopper)

Re: Regex character "$" doesn't mean "end-of-string"

#115

Post by masklinn » Wed, Mar 20, 2024, 11:46 AM UTC

eastbound wrote: post #109

masklinn wrote: post #83

Earlier quoted context omitted.

It’s not start of line though, unless the engine is in multiline mode. Here is the documentation for Python’s re for instance: > Matches the start of the string, and in MULTILINE mode also matches immediately after each newline. Or JavaScript: > An input boundary is the start or end of the string; or, if the m flag is set, the start or end of a line. \A and \Z are start/end of input regardless of mode… when they’re a…

Probably a vulnerability issue. Programmers would leave multiline mode on by mistake, then validate that some string only contain ^[a-Z]*$… only for the string to have an \n and an SQL injection on the second line.

> Probably a vulnerability issue.

No? It’s a semantics decision.

Re: Regex character "$" doesn't mean "end-of-string"

#116

Post by bonki » Wed, Mar 20, 2024, 11:47 AM UTC

rhd wrote: post #86

thecatspaw wrote: post #82

Earlier quoted context omitted.

what does gpt say how we should validate email addresses?

chatgpt-4: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566... chatgpt-3.5: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436...

Not good at all, but a little better than expected. I use + in email addresses prominently and there are so many websites who don't even allow that...

Re: Regex character "$" doesn't mean "end-of-string"

#117

Post by masklinn » Wed, Mar 20, 2024, 11:49 AM UTC

danbruc wrote: post #111

masklinn wrote: post #83

Earlier quoted context omitted.

It’s not start of line though, unless the engine is in multiline mode. Here is the documentation for Python’s re for instance: > Matches the start of the string, and in MULTILINE mode also matches immediately after each newline. Or JavaScript: > An input boundary is the start or end of the string; or, if the m flag is set, the start or end of a line. \A and \Z are start/end of input regardless of mode… when they’re a…

It is start and end of line. [1] Usually ^ matches only at the beginning of the string, and $ matches only at the end of the string and immediately before the newline (if any) at the end of the string. When this flag is specified, ^ matches at the beginning of the string and at the beginning of each line within the string, immediately following each newline. Similarly, the $ metacharacter matches either at the end of…

> It is start and end of line.

You seem to have redefined “line” as “not a line”.

> The confusion

I’m sure redefining “line” as “nothing like what anyone reasonable would interpret as a line” will help a lot and right clear up the confusion.

Re: Regex character "$" doesn't mean "end-of-string"

#118

Post by burntsushi » Wed, Mar 20, 2024, 11:50 AM UTC

danbruc wrote: post #99

fauigerzigerk wrote: post #76

Earlier quoted context omitted.

Why is this relevant when multi-line is disabled?

Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $, the newline at the end is still not part of the content. You have to use \A and \Z if you want to treat all characters as a string instead of one or multiple lines.

> Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $

No, you're not, except for this weird corner case where `$` can match before the last `\n` in a string. It's not just any `\n` that non-multiline `$` can match before. It's when it's the last `\n` in the string. See:

    >>> re.search('cat$', 'cat\n')
    
    >>> re.search('cat$', 'cat\n\n')
    >>>

This is weird behavior. I assume this is why RE2 didn't copy this. And it's certainly why I followed RE2 with Rust's regex crate. Non-multiline `$` should only match at the end of the string. It should not be line-aware. In regex engines like Python where it has the behavior above, it is only "partially" line-aware, and only in the sense that it treats the last `\n` as special.

Re: Regex character "$" doesn't mean "end-of-string"

#119

Post by zaxomi » Wed, Mar 20, 2024, 11:51 AM UTC

rhd wrote: post #86

thecatspaw wrote: post #82

Earlier quoted context omitted.

what does gpt say how we should validate email addresses?

chatgpt-4: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566... chatgpt-3.5: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436...

Remember to first punycode the domain part of an email address before trying to validate it, or it will not work with internationalized domain names.

Re: Regex character "$" doesn't mean "end-of-string"

#120

Post by hnlmorg » Wed, Mar 20, 2024, 11:55 AM UTC

da39a3ee wrote: post #84

ykonstant wrote: post #74

Earlier quoted context omitted.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

You don't have to be an expert; you should very rarely be using regexes so complex that you can't understand them.

...and if you can understand them then you clearly understand regex enough not to need ChatGPT to write them