Live data from Hacker News

Major European payment processor can't send email to Google Workspace users

atha.io

301–310 of 443 posts

Re: Major European payment processor can't send email to Google Workspace users

#301
post #282

Earlier quoted context omitted.

> SHOULD is a requirement. I once had a job where reading standards documents was my bread and butter. SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL. My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justi…

Maybe the standards documents you are used to differ from RFCs, but here is the official language: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. SHOULD is effectively REQUIRED unless it conflicts with another stan…

Nope, it's exactly what it says: RECOMMENDED.

Any time any document (standards or otherwise) says something is recommended, then of course you should think it through before going against the recommendation. Going from their verbiage to:

> SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.

is a fairly big leap.

Re: Major European payment processor can't send email to Google Workspace users

#302

Earlier quoted context omitted.

I just don't understand how you get from the text you pasted to "required". Nowhere does it say that anything is effectively required. Words have meaning.

[flagged]

I don't even know how you got to "used twice" tbh. Both your own comment AND the post you quoted from only have a single "must".

The only thing that text demands is understanding and carefully weighing the implications. If, having done that, you conclude that you don't want to then there is absolutely nothing in the spec stopping you. Maybe the spec would have been better off putting more stuff in SHALL and less in SHOULD, but as written that is definitely not the case.

Re: Major European payment processor can't send email to Google Workspace users

#303

Earlier quoted context omitted.

RFC language is expmicltly defined in 2119[0]. Any other interpretation is incorrect. [0] https://www.rfc-editor.org/rfc/rfc2119

Thank you for that. So should is optional, people!

Pulling exact quotes out, SHOULD means "there may exist valid reasons in particular circumstances to ignore a particular item" while MAY means "an item is truly optional."

I don't think this can be interpreted as simply "should is optional".

Re: Major European payment processor can't send email to Google Workspace users

#304

> Viva.com's outgoing verification emails lack a Message-ID header, a requirement that has been part of the Internet Message Format specification (RFC 5322) since 2008 > ... > `Message-ID` is one of the most basic required headers in email. Section 3.6. of the RFC in question ( https://www.rfc-editor.org/rfc/rfc5322.html ) says: +----------------+--------+------------+----------------------------+ | Field | Min | Max…

As indicated in the RFC, it uses another RFC[0] to define those words. Here's the relevant excerpt from that one: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. [0] https://www.rfc-editor.org/rfc/rfc2119

yeah, but that RFC isn't the only relevant document

Mail RFCs do not cover at all spam detection and malicious mail rejection, but it's a thing every large mail provider has and you really have to care about when producing automated mails all looking similar. And large mail providers like google tend to document what "base line" of additional requirements they have for accepting (automated mail). Having a Message-Id is in there, and in pretty much any larger mail providers documentation about that topic. Tbh. I have worked with mail before a bunch of years ago and the need for Message-Id was back then really no hidden gotcha but pretty well known.

and the design space mail provides is larger then any client could reasonable support (like it's really a huge mess covering docent of standards which allow all kind of nonsense and hypothetical use cases practical unsupported), so you anyway have to look at "what everyone does" and only then make sure it's also RFC compatible, instead of starting with the RFC. That was a painful lessen to learn.

In addition there are some de-facto standards not pinned down in any RFC, like e.g.:

- Message-Id being required for any automated mails by many mail providers (through how bad the consequences are if you don't have it diverges largely).

- You can't punycode encode the local part of an email address (it would be a different email), and there is no standard way (as far as I remember) to convert non us-ascii local parts to us-ascii. This is based on the fact that iff your mail server allows you to have non us-ascii local parts it should also support "internationalized mail" (SMTPUTF8 and co.). But it's a semi industry standard to give the user with an unicode local part also the mail with the punycode encoding of the local part so it often "just works" and dev are frequently surprised when it fails to work...

- You can have quoted text in local part, like whitespaces. But most of the industry decided to not give users such mail addresses so you can see it as soft deprecated and using it is asking for trouble.

- Attachements. The MIME encoding allows a lot of different ways to put mails together and doesn't force a specific semantic interpretation by mail clients. As such you if you naively use it you might run into surprises how/if your attachments or embedding(s) are displayed. Through today embedded resources often are either not done or uses data URLs. Again which ways work well and which don't is somewhat an industry standard and not in any RFC.

- A lot of different ways to encode Unicode to us-ascii. If you produce any mails you probably should by default use the latest revision (where encoding is often just not needed as things are utf8), but might need a fallback with it often being fully unclear if . But if you are a client you probably have to support older versions. And in some parts of the world/business segments usage of very very old mail servers is a thing which is a major pain if you run into it.

so quoting that something isn't strictly required by mail RFCs is kinda pointless as even many things explicitly allowed won't work well in practice

As a rule of thump: If you can afford it test you system will all widely used mail providers as if you where an external customers. And redo the tests yearly.

oh and as a bonus, if you mails looks too similar to known phishing mails it will also just disappear. That seems irrelevant, but e.g. if you use Keycloak with the default mail templates for password reset and co. there is a high chance of your mails ending up in spam or not even being delivered as scammers have used Keycloak for their means, too. And that isn't just a case for Keycloak but any "decently widely used open source software producing mails and having default templates". So you pretty much always need to change the default templates (you will do so anyway for branding, but skipping it in the earliest stages of a startup where branding might still be in flux isn't that uncommon either).

Re: Major European payment processor can't send email to Google Workspace users

#305

Maybe that's something to report to the "European System of Financial Supervision" or some other EU government agency. They even have a Whistleblowing link at the bottom of their website: https://www.bankingsupervision.europa.eu/about/esfs/html/ind...

no, please read the article

they forgot to include a message-id

something the RFC standard recommend but doesn't require

but it being required is a de-facto industry standard for sending automated mails

and is clearly documented by support sides of large mail providers (like Google)

the mail standards only defines what parts you can put together, but widely fail to define how this parts can be interpreted, what are sensible combinations, etc.

and they don't cover spam/suspicious mail detection at all

so you can't just go by RFC, you need to read up on what all larger mail providers have as additional requirements (which mostly are the same, and Message-Id being the most common dominator) and then hope that another provider you didn't read up one doesn't have some other surprising rule (which doesn't tent to be the case if you don't do anything surprising, but it sucks anyway).

Re: Major European payment processor can't send email to Google Workspace users

#306
Veeery interesting. I have a personal domain that forwards to a @gmail.com account. There are several companies I interact with, from banks to retail to just about anything, that send mail to my personal domain that never arrives in my Gmail account, but I can see on the hosted mailserver. For those companies I have to use my @gmail.com address instead for the mail to get through.

Maybe there are more companies than we think that send malformed emails?

Re: Major European payment processor can't send email to Google Workspace users

#307
post #6

The specific bug is annoying, but that there's no way to report such a thing is an exact hallmark of our current corposphere.

Google's Postmaster Tools site has a "Report deliverabilty issue" link at the bottom left navigation column. https://postmaster.google.com/v2/sender_compliance

it isn't a google bug

Message-Id being required for automated mails is a de-facto industry standard

while the consequences differ between mail provider, it missing will also make it much more likely for mail to be reject or put into the spam folder

It's also well known. Pretty much viva engineers fucked up doing proper research.

Now to be fair:

- it sucks that you can't just implement the RFC(s)

- the standards suck, docent of different RFCs overlapping and replacing each other and referencing often older versions of other RFCs, with docents of ways to do the same things of which only some can be used reliable in practice and a common gaps in the standards about edge cases or about the "higher level semantics" of constructs.

- so overall mail seems very simple at first but if you want to automated send mails reliable internationally it's a total pain and Message-Id is just the head of the iceberg.

Re: Major European payment processor can't send email to Google Workspace users

#308
post #142

> Viva.com's outgoing verification emails lack a Message-ID header, a requirement that has been part of the Internet Message Format specification (RFC 5322) since 2008 > ... > `Message-ID` is one of the most basic required headers in email. Section 3.6. of the RFC in question ( https://www.rfc-editor.org/rfc/rfc5322.html ) says: +----------------+--------+------------+----------------------------+ | Field | Min | Max…

SHOULD is a requirement. It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case. "I don't want to" is not a valid excuse, "I don't see a reason to" isn't either. IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/…

SHOULD is not MUST

Re: Major European payment processor can't send email to Google Workspace users

#309
post #47

> For viva.com's engineering team, in case this reaches you: add a Message-ID header to your outgoing transactional emails. Don't know what they're using for sending emails, but that's something that should be handled by their email service provider, unless they're hosting their own email servers.

Interestingly the MX record of via.com points to Google, but their verification emails could come from anywhere of course. The IP address in the log is also a Google IP, although that could also be the receiving IP.

message-id is (or at least was ~5-10 years ago) only required for automated mails, i.e. if you send a mail which looks mostly the same to a lot of users

so if you (ab-)use protocols for mail clients to send automated mails they might very well not add Message-Id. In general many mail providers have both a way to send a mail where "they do some clever parts" (like adding Message-Id) and interfaces where they mostly just send what you give them. It's possible that they migrated from one solution which did automatically add Message-Id to one using interfaces where it doesn't happen without realizing that there is a mismatch in this solutions do implicitly add.

Re: Major European payment processor can't send email to Google Workspace users

#310
post #282

Earlier quoted context omitted.

> SHOULD is a requirement. I once had a job where reading standards documents was my bread and butter. SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL. My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justi…

Maybe the standards documents you are used to differ from RFCs, but here is the official language: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. SHOULD is effectively REQUIRED unless it conflicts with another stan…

required means it must exist, not that it may or may not exist depending on the reason
Post reply on HN