Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

261–270 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#261

Post by mtmk »

ydant wrote: post #98
xlii wrote: post #8

Regexp was one of the first things I truly internalized years ago when I was discovering Perl (which still lives in a cozy place in my heart due to a lovely “Camel” book). Today most important bit of information is knowledge that implementations differ and I made a habit of pulling reference sheet for a thing I work with. E.g. Emacs Regexp annoyingly doesn’t have word in form of “\w” but uses “\s_-“ (or something no…

Exactly the same here, re: Perl. My brain thinks in Perl's regex language and then I have to translate the inconsistent bits to the language I'm using. Especially in the shell - I'm way more likely to just drop a perl into the pipeline instead of trying to remember how sed/grep/awk (GNU or BSD?) prefer their regex.

hah, I'm the same too, straight to 'perl -lne'. I believe that was one of Larry Wall's goals when creating Perl:

> Perl is kind of designed to make awk and sed semi-obsolete.

https://github.com/Perl/perl5/commit/8d063cd8

Re: Regex character "$" doesn't mean "end-of-string"

#262

Post by jbverschoor »

mnau wrote: post #256
keybored wrote: post #244

Earlier quoted context omitted.

Why did they even decide to use two characters for the end of line? Seems bizarre. I could have imagined that `\r` and `\n` was a tossup. But why both?

Likely compatibility bugs going back decades (70s?). Probably with some terminal/teletype. \r - returned teletype head to the start of a line \n - move paper one line down > The sequence CR+LF was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line…

It’s similar to an old school typewriter.

The handle does 2 things: return and feed. You can also just return by not pulling all the way or the other way around depending on the design

Re: Regex character "$" doesn't mean "end-of-string"

#263

Post by Izkata »

samatman wrote: post #192
Gormo wrote: post #184

Earlier quoted context omitted.

Suddenly the DOS/Windows solution of using \r\n instead of just \n seems to offer some advantages.

This does precisely nothing to solve the ambiguity issue when a final line lacks a newline. The representation of that newline isn't relevant to the problem.

It's actually slightly worse: Windows defines newline as a delimiter, not a terminator. So this:

  foo\nbar\n
Would be 2 lines in *nix and 3 lines in windows.

Re: Regex character "$" doesn't mean "end-of-string"

#264

Post by [deleted] »

danbruc wrote: post #121
masklinn wrote: post #117

Earlier quoted context omitted.

> It is start and end of line. You seem to have redefined “line” as “not a line”. > The confusion I’m sure redefining “line” as “nothing like what anyone reasonable would interpret as a line” will help a lot and right clear up the confusion.

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.

[deleted]

Re: Regex character "$" doesn't mean "end-of-string"

#265

Post by danbruc »

burntsushi wrote: post #234
danbruc wrote: post #229

Earlier quoted context omitted.

re.search does not accept a "line." It accepts a "string." There is no pretext in which re.search is meant to only accept a single line. Sure, it takes a string which might be a line or multiple or whatever. Does not change the fact that $ matches at the end of a line. If you want the end of the string, use \Z. This is trivially not true, as I showed in my previous example. The haystack `cat\n\n` contains two lines a…

> In multi-line mode it matches, in single-line mode it does not because there is a newline between cat and the end of the line. A newline is only a terminating newline if it is the last character, the newline after cat is not a terminating newline. You need cat\n$ or cat\n\n to match. This only makes sense if re.search accepted a line to search. It doesn't. It accepts an arbitrary string. I don't think this conversa…

This only makes sense if re.search accepted a line to search. It doesn't. It accepts an arbitrary string.

Which is fine because lines are a subset of strings. And whether you want your input treated as a line or a string is decided by your pattern, use ^ and $ and it will be treated as a line, use \A and \Z and it will be treated as a string.

The first `\n` in `cat\n\n` is a terminating newline. There just happens to be one after it.

Look at where this is coming from. You do line-based stuff, there is either no newline at all or there is exactly one newline at the end. You do file-based stuff, there are many newlines. In both cases the behavior of ^ and $ makes perfect sense.

Now you come along with cat\n\n which clearly falls into the file-based stuff category as it has more than one newline in it but you also insist that it is not multiple lines. If it is not multiple lines, then only the last character can be a newline, otherwise it would be multiple lines.

And I get it, yes, you can throw arbitrary strings at a regular expression, this line-based processing is not everything, but it explains why things behave the way they do. And that is also why people added \A and \Z. And I understand that ^ and $ are much nicer and much better known than \A and \Z. Maybe the best option would be to have a separate flag that makes them synonymous with \A and \Z and this could maybe even be the default.

Re: Regex character "$" doesn't mean "end-of-string"

#266

Post by masklinn »

danbruc wrote: post #121
masklinn wrote: post #117

Earlier quoted context omitted.

> It is start and end of line. You seem to have redefined “line” as “not a line”. > The confusion I’m sure redefining “line” as “nothing like what anyone reasonable would interpret as a line” will help a lot and right clear up the confusion.

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.

> a line is a sequence of non-newline characters

Works for me.

How do you square that with your assertion that in your invention of "single-line mode" you implicitly define "line" as matching \n\n?

Re: Regex character "$" doesn't mean "end-of-string"

#267

Post by michaelt »

Izkata wrote: post #188
Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

Same here; when I saw the title I was like "well obviously not, where did you hear that?" In nearly two decades of using regex I think this might be the first time I've heard of $ being end of string. It's always been end of line for me.

Take a look at, for example, these stackoverflow answers about a regex to validate and e-mail address: https://stackoverflow.com/a/8829363

These people are I think not intending to say a newline character is permitted at the end of an e-mail address.

(Of course people using 'grep' would have different expectations for obvious reasons)

Re: Regex character "$" doesn't mean "end-of-string"

#268

Post by skeaker »

thecatspaw wrote: post #82
ykonstant wrote: post #74

Earlier quoted context omitted.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

what does gpt say how we should validate email addresses?

There really ought to be a regex repository of common use cases like these so we don't have to reinvent the wheel or dig up a random codebase that we hope is correct to copy from every time.

Re: Regex character "$" doesn't mean "end-of-string"

#269

Post by LK5ZJwMwgBbHuVI »

wtetzner wrote: post #152
akdev1l wrote: post #142

Earlier quoted context omitted.

Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.

So if you have "A" in a file with no newline, there are no lines in that file?

Why don't you go ask?

    $ echo -n foo | wc -l
    0

Re: Regex character "$" doesn't mean "end-of-string"

#270

Post by LK5ZJwMwgBbHuVI »

joshjje wrote: post #257
xyzzy_plugh wrote: post #175

Earlier quoted context omitted.

No, it is valid for a file to have content but no lines. Semantically many libraries treat that as a line because while \n means "the end of the last line" having just adds additional complexity the user has to handle to read the remaining input. But by the book it's not "a line". If I said "ten buckets of water" does that mean ten full buckets? Or does a bucket with a drop in it count as "a bucket of water?" If I as…

Thats beyond ridiculous. Most languages when you are reading a line from a file, and it doesn't have a \n terminator, its going to give you that line, not say, oops, this isn't a line sorry.

That's a relatively recent invention compared to tools like `wc` (or your favorite `sh` for that matter). See also: https://perldoc.perl.org/functions/chop wherein the norm was "just cut off the last character of the line, it will always be a newline"
Post reply on HN