Live data from Hacker News

Regex character "$" doesn't mean "end-of-string"

sethmlarson.dev

171–180 of 382 posts

Re: Regex character "$" doesn't mean "end-of-string"

#171
post #58

Earlier quoted context omitted.

"I will hire you anyway, but I will pay you less" Regex, useful in any job...

regex is useful but chatgpt is amazing at it, so why spend a minute keeping such useless knowledge in mind. if you know where to find something no point in knowing it.

This is something ChatGPT would say.

Re: Regex character "$" doesn't mean "end-of-string"

#172
post #87

Earlier quoted context omitted.

what does gpt say how we should validate email addresses?

Prompt: 'I'm writing a nodejs javascript application and I need a regex to validate emails in my server. Can you write a regex that will safely and efficiently match emails?' GPT4 / Gemini Advanced / Claude 3 Sonnet GPT4: `const emailRegex = /^[a-zA-Z0-9._-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/;` Full answser: https://justpaste.it/cg4cl Gemini Advanced: `const emailRegex = /^[a-zA-Z0-9.!#$%&' +/=?^_`{|}~-]+@[a-zA-Z0-9](?:…

Whereas email more or less lasts forever (mailbox contents), and has to be backwards compatible with older versions back to (at least) RFC 821/822, or those before. It also allows almost any character (when escaped at 821 level) in the host or domain part (domain names allow any byte value).

So a Internet email address match pattern has to be: "..*@..*", anything else can reject otherwise valid addresses.

That however does not account for earlier source routed addresses, not the old style UUCP bang paths. However those can probably be ignored for newly generated email.

I regularly use an email address with a "+" in the host part. When I used qmail, I often used addresses like: "foo-a/b-bar-tat@DOMAIN". Mainly for auto filtering received messages from mailing lists.

Re: Regex character "$" doesn't mean "end-of-string"

#173

In 30 years of developing software I don’t think I ever used multi-line regexp even once.

Definitely not common, but if you are parsing a text file you're going to use it a lot (say, you're writing a JS parser).

You really shouldn't use a lot of regexes for parsing code.

They go only on the tokenizer, if they go somewhere at all.

Re: Regex character "$" doesn't mean "end-of-string"

#174
This has security implications! Example exploitable ruby code:

  unless person_id =~ /^\d+$/
    abort "Bad person ID"
  end
  sql = "select * from people where person_id = #{person_id}"
In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.

Re: Regex character "$" doesn't mean "end-of-string"

#175
post #157

Earlier quoted context omitted.

Yes, that is a file with zero lines that ends with an "incomplete line". Processing of such files by standard line-oriented utilities is undefined in the opengroup spec. So, for instance, the effect of "grep"ping such a file is not defined. Heck, even "cat"ting such a file gives non-ideal results, such as colliding with the regular shell prompt. For this reason, a lot of software projects I work on check and correct…

> Yes, that is a file with zero lines that ends with an "incomplete line". It's a file with zero complete lines. But it has 1 line, that's incomplete, right? The file starts empty. Anything in it starts "a line". So it's 1 incomplete line. I hate weird states.

No, it is valid for a file to have content but no lines.

Semantically many libraries treat that as a line because while \n means "the end of the last line" having just adds additional complexity the user has to handle to read the remaining input. But by the book it's not "a line".

If I said "ten buckets of water" does that mean ten full buckets? Or does a bucket with a drop in it count as "a bucket of water?" If I asked for ten buckets of water and you brought me nine and one half-full, is that acceptable? What about ten half-full buckets?

A line ends in a newline. A file with no newlines in it has no lines.

Re: Regex character "$" doesn't mean "end-of-string"

#176
post #150
post #97

Earlier quoted context omitted.

…which both excludes addresses allowed by the RFC and includes addresses disallowed by the RFC. (For example, the RFC disallows two consecutive dots in the local-part.)

I take the descriptivist approach to email validation, rather than the prescriptivist. I know an email has to have a domain name after the @ so I know where to send it. I also know it has to have something before the @ so the domain’s email server knows how to handle it. But do I care if the email server is supports sub addresses, characters outside of the commonly supported range (eg quotation marks and spaces), or…

The HTML email regex validation [1] is probably the best rule to use for validating an email address in most user applications. It prohibits IP address domain literals (which the emailcore people have basically said is of limited utility [2]), and quoted strings in the localpart. Its biggest fault is allowing multiple dots to appear next to each other, which is a lot of faff to put in a regex when you already have to individually spell out every special character in atext.

[1] https://html.spec.whatwg.org/multipage/input.html#email-stat...

[2] https://datatracker.ietf.org/doc/draft-ietf-emailcore-as/

Re: Regex character "$" doesn't mean "end-of-string"

#177
post #12

Does anyone consider RegEx to be standardised? Moving to a new context is always a relearning exercise in my experience.

The three big ones I know of are POSIX, Perl/PCRE(aka Perl-Compatible Regular Expression), and Go came along and added used re2, which is a bit different from the first too. A lot of systems implemented PCRE, including JavaScript, since Perl extended the POSIX system with many useful extensions. IIRC, re2 tries to reign in on some of the performance issues and quirks the original systems had, while implementing the w…

POSIX and PCRE are arguably redundant. They both support backreferences, which puts very significant constraints on their implementations. PCRE is at least functionally a superset of POSIX, whether or not there's some quirky thing POSIX supports that PCRE does not.

re2 adds a legitimate option to the menu of using NDFAs, which have the disadvantage of not supporting backreferences, but have the advantage of having constrained complexity of scanning a string. This does not come for free; you can conceivably end up with a compiled regexp of very large size with an NDFA approach, but most of the time you won't. The result may be generally slower than a PCRE-type approach, but it can also end up safer because you can be confident that there isn't a pathological input string for a given regexp that will go exponential.

This is one of those cases where ~99% of the time, it doesn't really matter which you choose, but at the scale of the Entire Programming World, both options need to be available. I've got some security applications where I legitimately prefer the re2 implementation in Go because it is advantageous to be confident that the REs I write have no pathological cases in the arbitrary input they face. PCRE can be necessary in certain high-performance cases, as long as you can be sure you're not going to get that pathological input.

RE engines don't quite engender the same emotions as programming languages as a whole, but this is not cheerleading, this is a sober engineering assessment. I use both styles in my code. I've even got one unlucky exe I've been working with lately that has both, because it rather irreducibly has the requirements for both. Professionally annoying, but not actually a problem.

Re: Regex character "$" doesn't mean "end-of-string"

#178

Earlier quoted context omitted.

Definitely not common, but if you are parsing a text file you're going to use it a lot (say, you're writing a JS parser).

You really shouldn't use a lot of regexes for parsing code. They go only on the tokenizer, if they go somewhere at all.

Agreed, it's more about quick and dirty ad hoc capture than full-fledged parser though (like when you want to extract certain object when scraping).

Re: Regex character "$" doesn't mean "end-of-string"

#179
post #157

Earlier quoted context omitted.

Yes, that is a file with zero lines that ends with an "incomplete line". Processing of such files by standard line-oriented utilities is undefined in the opengroup spec. So, for instance, the effect of "grep"ping such a file is not defined. Heck, even "cat"ting such a file gives non-ideal results, such as colliding with the regular shell prompt. For this reason, a lot of software projects I work on check and correct…

> Yes, that is a file with zero lines that ends with an "incomplete line". It's a file with zero complete lines. But it has 1 line, that's incomplete, right? The file starts empty. Anything in it starts "a line". So it's 1 incomplete line. I hate weird states.

No, a line is defined as a sequence of characters (bytes?) with a line terminator at the end.

Technically as per posix a file as you describe is actually a binary file without any lines. Basically just random binary data that happens to kind of look like a line.

Re: Regex character "$" doesn't mean "end-of-string"

#180
post #21

Special misery case: Visual Studio supports regex search, where '$' matches \n. The end of line character is usually the standard Windows \r\n. Yes, that means if you want to really match the end of line you have to match "\r$". So broken.

The whole \r is archaic. It doesn't even behave properly in most cases. Just use \n everywhere and bite the lemon for a short while to fix your problems. And if you believe \r\n is the way to go, please make sure \n\r also works as they should have the same results. (or \r\n\r\r\r\r for that matter)

But without \r how am I supposed to print to my typewriter over serial cable? Only half-joking, that's the setup my family had in the early 90's.
Post reply on HN