> A portion of this code implemented a SMTP client. If I wanted to root cause this, the real problem is right there. Implementing protocols correctly is hard and bugs like in the post are common. A properly implemented SMTP client library, like one you would pull off the shelf, would accept text and encode it properly per the SMTP protocol, regardless of where the periods were in the input. The templating layer shoul…
The curious case of the missing period
101–110 of 201 posts
Re: The curious case of the missing period
#102So... SMTP client misses basic SMTP functionality that fits in the 20 lines summary. Good luck having it handle any of the SMTP craziness that isn't on the short introduction to the protocol.
When you roll your own library duplicating something that likely already exists, you own it all. And that doesn't include all of the vendor-specific nonconforming edge-cases.
I guess when you roll your own library, you get an obligation to read the introduction of the 10 pages long standard.
Or the usage example.
Re: The curious case of the missing period
#103> A portion of this code implemented a SMTP client. If I wanted to root cause this, the real problem is right there. Implementing protocols correctly is hard and bugs like in the post are common. A properly implemented SMTP client library, like one you would pull off the shelf, would accept text and encode it properly per the SMTP protocol, regardless of where the periods were in the input. The templating layer shoul…
The real problem is that SMTP is a "plain-text" protocol that includes in-band signaling. It literally happened because SMTP defined "a line that only contains a single period in it" as a control sequence and not a literal line that only contains a single period in it.
SMTP is an example of an unnecessarily complex design, and the implementation bugs reflect it. SMTP shouldn't be hard for someone to correctly implement by themselves (even though I agree that people shouldn't be re-inventing the wheel).
Re: The curious case of the missing period
#104Why would a cron job that sends e-mails need to implement its own SMTP client??? You just use the mail program from mailutils or whatever. Just from a point of view of deliverability, developing bare bones SMTP interaction over a socket is a nonstarter. You can't just connect to random mail exchange hosts directly and send mail these days. A solution has to be capable of connecting to a specific SMTP forwarding host…
There's no universe where I'd trust that there's already some binary on the system to send mail for me, but I definitely wouldn't roll my own. For any reasonable language, there's a multitude of SMTP client libraries available if there's not already one in the stdlib.
I'd give up instantly and outsource it to a SaaS. I'm no expert on email but I know enough to know it's a PITA and managing delivery is a thing, and you have to make sure your DNS is configured right out the receiver will just reject it, there's reputations to consider, you probably want to throttle how fast you send it.. ugh.
Re: The curious case of the missing period
#105> This meant some customers received emails informing them their new premium was now $2700 instead of $27.00. there's a secondary issue here, why in the world would you auto split a monetary value across a numeric decimal indicator? why would you split lines at all for this use case?
As mentioned, the SMTP protocol only allows for 1000 bytes of data per line. The author also mentions that they are sending html emails, which ignore line breaks. So a message intended to be sent by an SMTP client: DATA Hello customer, [978 characters] 27.00 Was erroneously formated into: DATA Hello customer, [978 characters] 27 .00 . The period after 27 will be removed. And this is how the html will be rendered. Hel…
but html does not ignore line breaks. when part of body text, a run of whitespace (including newline) becomes a single whitespace when rendered.
so splitting 27.00 on the . becomes 27 00, because the CRLF is significant to the client.
you would want to split at whitespace, not at any other character -- unless you had a 999+ string of non-whitespace of course.
perhaps the author didn't know or didn't realize or thought it insignificant to his point that in addition there was a quoted-printable encoding, in which case i believe the trailing/mandatory CRLF can be made non significant for client rendering. personally i still would have split on actual whitespace. (well, i wouldn't have written an smtp client in the first place.)
Re: The curious case of the missing period
#106Why would a cron job that sends e-mails need to implement its own SMTP client??? You just use the mail program from mailutils or whatever. Just from a point of view of deliverability, developing bare bones SMTP interaction over a socket is a nonstarter. You can't just connect to random mail exchange hosts directly and send mail these days. A solution has to be capable of connecting to a specific SMTP forwarding host…
There's no universe where I'd trust that there's already some binary on the system to send mail for me, but I definitely wouldn't roll my own. For any reasonable language, there's a multitude of SMTP client libraries available if there's not already one in the stdlib.
They are relying on a cron program being there, so ...
You can require it and it gets provisioned.
Re: The curious case of the missing period
#107Why would a cron job that sends e-mails need to implement its own SMTP client??? You just use the mail program from mailutils or whatever. Just from a point of view of deliverability, developing bare bones SMTP interaction over a socket is a nonstarter. You can't just connect to random mail exchange hosts directly and send mail these days. A solution has to be capable of connecting to a specific SMTP forwarding host…
There's actually a lot of things that embed a SMTP client for sending mail that should use a host MTA. The reason is that the user who actually wants to use "the thing" is often just about able to enter an SMTP server, but there's no way "the thing" can trust that a properly-configured sender MTA like sendmail is configured. Companies have huge fleets of servers that can't send system mail, and it's often not really…
The SMTP client needs to be configured. E.g. if there is a particular SMTP forwarding host that must be used, you have somehow get the "mail" utility to use that, and you must somehow any get any given SMTP library to also use it.
Re: The curious case of the missing period
#108> A portion of this code implemented a SMTP client. If I wanted to root cause this, the real problem is right there. Implementing protocols correctly is hard and bugs like in the post are common. A properly implemented SMTP client library, like one you would pull off the shelf, would accept text and encode it properly per the SMTP protocol, regardless of where the periods were in the input. The templating layer shoul…
The real problem is that SMTP is a "plain-text" protocol that includes in-band signaling. It literally happened because SMTP defined "a line that only contains a single period in it" as a control sequence and not a literal line that only contains a single period in it. SMTP is an example of an unnecessarily complex design, and the implementation bugs reflect it. SMTP shouldn't be hard for someone to correctly impleme…
Why didn't they use actual control sequences instead of text chars, ASCII had plenty of those?
Re: The curious case of the missing period
#109I suspect a lot of people are no longer being taught these fundamental protocols by manual interaction with a terminal, since that's what SMTP seems to have been originally intended for; and as someone who actually made use of that for a nontrivial amount of time, the "single dot on a line" to end a message has been permanently etched into my memory.
Relatedly, escaping somehow seems to be a foreign concept for a lot of programmers, who wouldn't ever see the above situation and ask themselves "but what if I want to send an email with a line containing a single dot?" yet another large group of them finds it perfectly logical and easy to understand.
Re: The curious case of the missing period
#110Earlier quoted context omitted.
Agreed. But this is SMTP, I have no doubt there are gateways out there that will reencode the mail and put the dot back in the wrong place, eg in the name of wrapping all URLs behind a phishing-warning-page
I think the point here is that any subsequent implementation handling the message may well not be susceptible to the problem thus avoiding the erroneous behavior.
It can handle it fine in that sense, but still unquote it before sending it over to the next node. Which might have the bug.