Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

Post reply on HN
Print view View original

251–260 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#251

Post by mnau »

da39a3ee wrote: post #84
ykonstant wrote: post #74

Earlier quoted context omitted.

Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.

You don't have to be an expert; you should very rarely be using regexes so complex that you can't understand them.

Even simple regexs can be problematic, e.g. Gitlab RCE bug through ExifTools

https://devcraft.io/2021/05/04/exiftool-arbitrary-code-execu...

> "a\ > ""

> The second quote was not escaped because in the regex $tok =~ /(\\+)$/ the $ will match the end of a string, but also match before a newline at the end of a string, so the code thinks that the quote is being escaped when it’s escaping the newline.

Re: Regex character "$" doesn't mean "end-of-string"

#253

Post by joshjje »

akdev1l wrote: post #142
Bjartr wrote: post #140

Earlier quoted context omitted.

The line delimiter is a newline. If you have a file containing `A\nB\nC` in a file, the file is three lines long. I guess it could be argued that a file containing `A\nB\nC\n` has four lines, with the fourth having zero length. That a regex is applying to an in memory string vs a file doesn't feel to me like it should have different semantics. Digging into the history a little, it looks like regexes were popularized…

Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.

“A\n” is two lines.

Re: Regex character "$" doesn't mean "end-of-string"

#254

Post by mnau »

jewel wrote: post #174

This has security implications! Example exploitable ruby code: unless person_id =~ /^\d+$/ abort "Bad person ID" end sql = "select * from people where person_id = #{person_id}" In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.

Practical Gitlab RCE that involved end of line regex in ExifTools:

https://devcraft.io/2021/05/04/exiftool-arbitrary-code-execu...

Re: Regex character "$" doesn't mean "end-of-string"

#255

Post by nomel »

cortesoft wrote: post #227
rerdavies wrote: post #217

Earlier quoted context omitted.

Technically, that is one of two possible interpretations, and you seem to have invented a "by definition" out of thin air. Very very technically a "newline" character indicates the start of a new line, which is why it is not called the "end-of-line" character.

I mean, the person you are responding to didn't invent the definition out of thin air... the POSIX standard did: 3.206 Line A sequence of zero or more non- characters plus a terminating character. https://pubs.opengroup.org/onlinepubs/9699919799.2018edition...

Posix getline() includes EOF as a line terminator:

    getline() reads an entire line from stream, storing the address
       of the buffer containing the text into *lineptr.  The buffer is
       null-terminated and includes the newline character, if one was
       found.
    ...
    ... a delimiter character is not added if one was
       not present in the input before end of file was reached.
EOF seems same as end-of-string.

Re: Regex character "$" doesn't mean "end-of-string"

#256

Post by mnau »

keybored wrote: post #244
jbverschoor wrote: post #37

Earlier quoted context omitted.

The whole \r is archaic. It doesn't even behave properly in most cases. Just use \n everywhere and bite the lemon for a short while to fix your problems. And if you believe \r\n is the way to go, please make sure \n\r also works as they should have the same results. (or \r\n\r\r\r\r for that matter)

Why did they even decide to use two characters for the end of line? Seems bizarre. I could have imagined that `\r` and `\n` was a tossup. But why both?

Likely compatibility bugs going back decades (70s?). Probably with some terminal/teletype.

\r - returned teletype head to the start of a line

\n - move paper one line down

> The sequence CR+LF was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line. The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in time to print the next character. Any character printed after a CR would often print as a smudge in the middle of the page while the print head was still moving the carriage back to the first position. "The solution was to make the newline two characters: CR to move the carriage to column one, and LF to move the paper up."[2] In fact, it was often necessary to send extra padding characters—extraneous CRs or NULs—which are ignored but give the print head time to move to the left margin. Many early video displays also required multiple character times to scroll the display.

https://en.wikipedia.org/wiki/Newline

Re: Regex character "$" doesn't mean "end-of-string"

#257

Post by joshjje »

xyzzy_plugh wrote: post #175
rovr138 wrote: post #165

Earlier quoted context omitted.

> Yes, that is a file with zero lines that ends with an "incomplete line". It's a file with zero complete lines. But it has 1 line, that's incomplete, right? The file starts empty. Anything in it starts "a line". So it's 1 incomplete line. I hate weird states.

No, it is valid for a file to have content but no lines. Semantically many libraries treat that as a line because while \n means "the end of the last line" having just adds additional complexity the user has to handle to read the remaining input. But by the book it's not "a line". If I said "ten buckets of water" does that mean ten full buckets? Or does a bucket with a drop in it count as "a bucket of water?" If I as…

Thats beyond ridiculous. Most languages when you are reading a line from a file, and it doesn't have a \n terminator, its going to give you that line, not say, oops, this isn't a line sorry.

Re: Regex character "$" doesn't mean "end-of-string"

#258

Post by vitiral »

denzquix wrote: post #211
vitiral wrote: post #202

In Lua it's only the start/end of the string > A pattern is a sequence of pattern items. A caret '^' at the beginning of a pattern anchors the match at the beginning of the subject string. A '$' at the end of a pattern anchors the match at the end of the subject string. At other positions, '^' and '$' have no special meaning and represent themselves. https://www.lua.org/manual/5.3/manual.html#6.4.1 Lua's pattern matc…

> In Lua it's only the start/end of the string There's an additional caveat: if you use the optional "init" parameter to specify an offset into the string to start matching, the ^ anchor will match at that offset , which may or may not be what you expect.

That is a good point, and something I've actually (personally) used quite a bit when writing parsers

Re: Regex character "$" doesn't mean "end-of-string"

#259

Post by frame_ranger »

Izkata wrote: post #188
Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

Same here; when I saw the title I was like "well obviously not, where did you hear that?" In nearly two decades of using regex I think this might be the first time I've heard of $ being end of string. It's always been end of line for me.

You couldn’t write a post like this if you didn’t start with a strawman.

Re: Regex character "$" doesn't mean "end-of-string"

#260

Post by cerved »

Karellen wrote: post #28

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of…

In `sed` it's end of string.

String is usually end of line, but not if you use stuff like `N`, to manipulate multi-line strings

Post reply on HN