Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

311–320 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#311
post #184

Earlier quoted context omitted.

Suddenly the DOS/Windows solution of using \r\n instead of just \n seems to offer some advantages.

The "Windows way" is the "right way" for a few reasons. This is definitely not one of them.

Which are the valid reasons, legacy meanings of those characters aside?

Re: Regex character "$" doesn't mean "end-of-string"

#312
post #183

Earlier quoted context omitted.

^ actually takes you to the first non-whitespace character in the line in vim. For start of line you want 0

I don't have (n)vi(m) open right now but I think this only applies to prepending spaces. For prepending tabs, 0 will take you to the first non-tab character as well.

Vim takes me to the first character in the line (the first tab), but displays the cursor on the last gridsquare the tab's width covers.

Re: Regex character "$" doesn't mean "end-of-string"

#313

Earlier quoted context omitted.

I don't think anyone that writes regex would feel specially challenged by using the Alt+ | Ctrl+Shift+u key combos for unicode entry. Having to escape less things in a pattern would be nice.

I write regexes all the time, and I don't know if I would be CHALLENGED by that, but it would be annoying. Escaping things is trivial, and since you do it all the time it is not anything extra to learn. Having to remember bespoke keystrokes for each character is a lot more to learn.

Regexes are one case where I think it's already extremely unbalanced wrt being easy to write but hard to read. Using stuff like special Unicode chars for this would make them harder to write but easier to read, which sounds like a fair deal to me. In general, I'd say that regexes should take time and effort to write, just because it's oh-so-easy to write something that kinda sorta works but has massive footguns.

I would also imagine that, if this became the norm, IDEs would quickly standardize around common notation - probably actually based on existing regex symbols and escapes - to quickly input that, similar to TeX-like notation for inputting math. So if you're inside a regex literal, you'd type, say, \A, and the editor itself would automatically replace it with the Unicode sigil for beginning-of-string.

Re: Regex character "$" doesn't mean "end-of-string"

#314

Earlier quoted context omitted.

In theory yes, in practice no. What you have gained is that the regex is now much easier to read.

> In theory yes, in practice no. That's like "in theory we need 4 bytes to represent Unicode, but in practice 3 bytes is fine" ( glances at universally-maligned utf8mb3 )

It's not really an issue if the string you're matching might have those characters. It's an issue if the regex you are matching that string might need to match those characters verbatim. Which is actually pretty common with ()[]$ when you're matching phone numbers, prices etc - so you end up having to escape a lot, and regex is less readable especially if it also has to use those same characters as regex operators. On the other hand, it would be very uncommon to want to literally match, say, ⦑⦒ or ⟦⟧.

Re: Regex character "$" doesn't mean "end-of-string"

#315
post #303

Earlier quoted context omitted.

Python violates that definition however, by allowing internal newlines in strings. For example /^c[^a]t$/ matches "c\nt\n", but according to POSIX that's not a line. I suspect the real reason for Python's behavior starts with the early decision to include the terminating newline in the string returned by IOBase.readline(). Python's peculiar choice has some minor advantages: you can distinguish between files that do a…

Python's behavior is not a hack, it is the common behavior. $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. But as you said, you can have additional newlines inside of the string which is also the common behavior and not specific to python. Personally I think of this as you just assume that the string is a single line and m…

> Python's behavior [..] is the common behavior.

The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $).

Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behavior.

> $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line.

Yes, that is Python's behavior (and PHP's, Java's, etc.). You're just describing it; not motivating why it has to work that way or why it's more correct than the obvious alternative of only matching the end of the string.

Subjectively, I find it odd that /^cat$/ matches not just the obvious string "cat" but also the string "cat\n". And I think historically, it didn't. I tried several common tools that predate Python:

  - awk 'BEGIN { print ("cat\n" ~ /^cat$/) }' prints 0
  - in GNU ed, /^M/ does not match any lines
  - in vim, /^M/ does not match any lines
  - sed -n '/\n/p' does not print any lines
  - grep -P '\n' does not match any lines
  - (I wanted to try `grep -E` too but I don't know how to escape a newline)
  - perl -e 'print ("cat\n" =~ /^cat$/)' prints 1
So the consensus seems to be that the classic UNIX line-based tools match the regex against the line excluding the newline terminator (which makes sense since it isn't part of the content of that line) and therefore $ only needs to match the end of the string.

The odd one out is Perl: it seems to have introduced the idea that $ can match a newline at the end of the string, probably for similar reasons as Python. All of this suggests to me that allowing $ to match both "\n" and "" at the end of the string was a hack designed to make it easier to deal with strings without control characters and string that end with a single newline.

Re: Regex character "$" doesn't mean "end-of-string"

#316

Earlier quoted context omitted.

It's kind of driving me nuts that the article says ^ is "start of string" when it's actually "start of line", just like $ is "end of line". \A is apparently "start of string" like \Z is "end of string".

What is driving me nuts is that we have Unicode now, so there is no need to use common characters like $ or ^ to denote special regex state transitions.

The problem with using an eggplant to denote end of string is backwards compatibility.

Re: Regex character "$" doesn't mean "end-of-string"

#317

If you really want to learn regex, you'll have a hard time piecing it all together via blog posts. Brad Freidl's Mastering Regular Expressions is a good book to read if you want to stop being surprised/lost. I'll admit I stopped at the dive into DFA/NFA engine details.

Writing a string -> NFA -> DFA grep-like tool is one of my most memorable college projects. Had a lot of fun with that, and decades later I ended up reusing some of the concepts for a work project.

Re: Regex character "$" doesn't mean "end-of-string"

#318
post #297

Was any regex documentation unclear on this? Some libraries have modes that change the semantics of ^ and $ but I’ve always found their use to be rather clear. It’s the grouping and look ahead/behind modifiers that I’ve always found hard to understand (at times).

This is a feature that seems so painfully obvious in the abstract that I’d wager most have never read the documentation. I’ve been a regex user since the early 90s and I’ve never thought about this.

Re: Regex character "$" doesn't mean "end-of-string"

#319

I think there's a big opportunity to re-write Regex as a SQL-type language. It's too bad I don't feel like trying.

I am 99% confident I’ve seen a post on HN about someone creating a much more verbose regex replacement. I have no idea how to find it.

Here’s an interesting project for typed regular expressions: https://news.ycombinator.com/item?id=12292389

Re: Regex character "$" doesn't mean "end-of-string"

#320
post #315

Earlier quoted context omitted.

Python's behavior is not a hack, it is the common behavior. $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. But as you said, you can have additional newlines inside of the string which is also the common behavior and not specific to python. Personally I think of this as you just assume that the string is a single line and m…

> Python's behavior [..] is the common behavior. The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $). Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behav…

> - in vim, /^M/ does not match any lines

But /\n/ does

Post reply on HN