Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

81–90 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#81

Post by kqr »

Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

I'm the same, but now that I try in Perl, sure enough, $ seems to default to being a positive lookahead assertion for the end of the string. It does not match and consume an EOL character.

Only in multiline mode does it match EOL characters, but it does still not appear to consume them. In fact, I cannot construct a regex that captures the last character of one line, then consumes the newline, and then captures the first character of the next line, while using $. The capture group simply ends at $.

Re: Regex character "$" doesn't mean "end-of-string"

#82

Post by thecatspaw »

ykonstant wrote: post #74
username_my1 wrote: post #61

Earlier quoted context omitted.

regex is useful but chatgpt is amazing at it, so why spend a minute keeping such useless knowledge in mind. if you know where to find something no point in knowing it.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

what does gpt say how we should validate email addresses?

Re: Regex character "$" doesn't mean "end-of-string"

#83

Post by masklinn »

wccrawford wrote: post #65
Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

It's kind of driving me nuts that the article says ^ is "start of string" when it's actually "start of line", just like $ is "end of line". \A is apparently "start of string" like \Z is "end of string".

It’s not start of line though, unless the engine is in multiline mode. Here is the documentation for Python’s re for instance:

> Matches the start of the string, and in MULTILINE mode also matches immediately after each newline.

Or JavaScript:

> An input boundary is the start or end of the string; or, if the m flag is set, the start or end of a line.

\A and \Z are start/end of input regardless of mode… when they’re available, that’s not the case of all engines.

Re: Regex character "$" doesn't mean "end-of-string"

#84

Post by da39a3ee »

ykonstant wrote: post #74
username_my1 wrote: post #61

Earlier quoted context omitted.

regex is useful but chatgpt is amazing at it, so why spend a minute keeping such useless knowledge in mind. if you know where to find something no point in knowing it.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

You don't have to be an expert; you should very rarely be using regexes so complex that you can't understand them.

Re: Regex character "$" doesn't mean "end-of-string"

#85

Post by colimbarna »

romwell wrote: post #64
fooofw wrote: post #55

Earlier quoted context omitted.

Fortunately, it's pretty simple to test. $ printf 'Line with EOL\nLine without EOL' | grep 'EOL$' Line with EOL Line without EOL $ grep --version | head -n1 grep (GNU grep) 3.8

The line does end with the file, so it's logically consistent. It's not matching the newline character after all.

Yes exactly, they match the end of a line, not a newline character. Some examples from documentation:

man 7 regex: '$' (matching the null string at the end of a line)

pcre2pattern: The circumflex and dollar metacharacters are zero-width assertions. That is, they test for a particular condition being true without consuming any characters from the subject string. These two metacharacters are concerned with matching the starts and ends of lines. ... The dollar character is an assertion that is true only if the current matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default), unless PCRE2_NOTEOL is set. Note, however, that it does not actually match the newline. Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

Re: Regex character "$" doesn't mean "end-of-string"

#86

Post by rhd »

thecatspaw wrote: post #82
ykonstant wrote: post #74

Earlier quoted context omitted.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

what does gpt say how we should validate email addresses?

chatgpt-4:

^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566...

chatgpt-3.5:

^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436...

Re: Regex character "$" doesn't mean "end-of-string"

#87

Post by criley2 »

thecatspaw wrote: post #82
ykonstant wrote: post #74

Earlier quoted context omitted.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

what does gpt say how we should validate email addresses?

Prompt:

'I'm writing a nodejs javascript application and I need a regex to validate emails in my server. Can you write a regex that will safely and efficiently match emails?'

GPT4 / Gemini Advanced / Claude 3 Sonnet

GPT4: `const emailRegex = /^[a-zA-Z0-9._-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/;` Full answser: https://justpaste.it/cg4cl

Gemini Advanced: `const emailRegex = /^[a-zA-Z0-9.!#$%&'+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)$/;` Full answer: https://justpaste.it/589a5

Claude 3: `const emailRegex = /^([a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})$/;` Full answer: https://justpaste.it/82r2v

Re: Regex character "$" doesn't mean "end-of-string"

#88

Post by susam »

jwilk wrote: post #48
susam wrote: post #20

Earlier quoted context omitted.

POSIX specifies two flavours of regular expressions: basic regular expressions (BRE) and extended regular expressions (ERE). There are subtle differences between the two and ERE supports more features than BRE. For example, what is written as a\(bc\)\{3\}d in BRE is written as a(bc){3}d in ERE. See https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... for more details. The regular expression engines availab…

> what is written as \(f..\)\1 in BRE is written as (f..)\1 in ERE Oddly, there are no backreferences in POSIX EREs.

You are right indeed. Looked at the specification again and indeed there is no back-reference in POSIX ERE.

Quoting from https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...>:

> It was suggested that, in addition to interval expressions, back-references ( '\n' ) should also be added to EREs. This was rejected by the standard developers as likely to decrease consensus.

Updated my comment to present a better example that avoids back-references. Thanks!

Re: Regex character "$" doesn't mean "end-of-string"

#89

Post by danbruc »

People are confused about strings and lines. A string is a sequence of characters, a line can be two different things. If you consider the newline a line terminator, then a line is a sequence of non-newline characters - possibly zero - plus a newline. If there is no new-line at the end, then it is not a [complete] line. That is what POSIX uses. If you consider the newline a line separator, then a line is a sequence of non-newline characters - possibly zero. In either case, the content of the line ends before the newline, either because the newline terminates the line or because it separates the line from the next. [1]

The semantics of ^ and $ is based on lines - whether single-line or multi-line mode. For string based semantics - which you could also think of as entire file if you are dealing with files - use \A and \Z or their equivalents.

[1] Both interpretations have their merits. If you transmit text over a serial connection, it is useful to have a newline as line terminator so that you know when you received a complete line. If you put text into text files, it might arguably be easier to look at a newline as a line separator because then you can not have a invalid last line. On the other hand having line terminators in text files allows you to detect incompletely written lines.

Re: Regex character "$" doesn't mean "end-of-string"

#90

Post by noirscape »

IshKebab wrote: post #9
Izmaki wrote: post #4

The new-line character is an actual character "at the end" of the string though so it makes sense that $ would include the new-line character in multi-line matching.

Yes and every implementation gets that right. The point was when multi-line matching is disabled and only Javascript, Go and Rust get that right. I'm not too surprised by PHP and Python getting it wrong. Java and C# is a slight surprise though.

It's not wrong actually. It's the difference between BRE and ERE, which are the two different POSIX standards that define regex. In BRE the $ should always match the end of the string (the spec specifically says it should match the string terminator since "newlines aren't special characters"), while the ERE spec says it should match until the end of the line.

The real issue is that no language nowadays "just" implements BRE or ERE since both specs are lacking in features.

Most languages instead implement some variant of Perl's regex instead (often called PCRE regex because of the C library that brought Perl's regex to C), which as far as I can tell isn't standardized, so you get these subtle differences between implementations.

Post reply on HN