Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The curious case of the missing period

tjaart.substack.com

Post reply on HN
Print view View original

141–150 of 201 posts

Re: The curious case of the missing period

#141

Post by leni536 »

partdavid wrote: post #65
kazinator wrote: post #34

Why would a cron job that sends e-mails need to implement its own SMTP client??? You just use the mail program from mailutils or whatever. Just from a point of view of deliverability, developing bare bones SMTP interaction over a socket is a nonstarter. You can't just connect to random mail exchange hosts directly and send mail these days. A solution has to be capable of connecting to a specific SMTP forwarding host…

There's actually a lot of things that embed a SMTP client for sending mail that should use a host MTA. The reason is that the user who actually wants to use "the thing" is often just about able to enter an SMTP server, but there's no way "the thing" can trust that a properly-configured sender MTA like sendmail is configured. Companies have huge fleets of servers that can't send system mail, and it's often not really…

Use an SMTP library then.

Re: The curious case of the missing period

#142

Post by BoppreH »

pja wrote: post #128
throw10920 wrote: post #103

Earlier quoted context omitted.

The real problem is that SMTP is a "plain-text" protocol that includes in-band signaling. It literally happened because SMTP defined "a line that only contains a single period in it" as a control sequence and not a literal line that only contains a single period in it. SMTP is an example of an unnecessarily complex design, and the implementation bugs reflect it. SMTP shouldn't be hard for someone to correctly impleme…

At some point, all protocols include in-band signalling somewhere: You have to put packets on the line, and those packets are ultimately just a stream of anonymous bytes. If it wasn’t a period, it would be something else & you’d have to handle that instead.

That's not a very useful definition of "in-band signaling". For me, the main difference is an out-of-band protocol that says:

"The first two bytes represent the string length, in big-endian, followed by that many bytes presenting the string text."

and an in-band signalling protocol:

"The string is ended by a period and a newline."

In the second one, you're indicating the end of the string from within the string. It looks simpler, but that's where accidents happen. Now you have to guarantee that the text never contains that control sequence, and you need an escaping method to represent the control sequence as part of the text.

Re: The curious case of the missing period

#143

Post by mananaysiempre »

TeMPOraL wrote: post #122
evmar wrote: post #5

> A portion of this code implemented a SMTP client. If I wanted to root cause this, the real problem is right there. Implementing protocols correctly is hard and bugs like in the post are common. A properly implemented SMTP client library, like one you would pull off the shelf, would accept text and encode it properly per the SMTP protocol, regardless of where the periods were in the input. The templating layer shoul…

The real problem isn't the protocol, but the cowboy approach to interacting with it. It's not hard to "accept text and encode it properly per the SMTP protocol", you just need to realize you need to do it in the first place. There is a multitude of classes of errors and security vulnerabilities, including "SQL injection", XSS, and similar, that are all caused by the same mistake that this case of missing period was[0…

For what it’s worth, some markup-first template systems have tried to respect the target format’s structure—Genshi[1] and TAL[2] come to mind, and of course XSLT (see also SLAX[3]). I said “markup-first” so that the whole question isn’t trivialized by JSX.

[1] https://genshi.edgewall.org/wiki/Documentation/xml-templates...

[2] https://zope.readthedocs.io/en/latest/zopebook/AppendixC.htm...

[3] https://juniper.github.io/libslax/slax-manual.html

Re: The curious case of the missing period

#144

Post by throw10920 »

TeMPOraL wrote: post #140
throw10920 wrote: post #134

Earlier quoted context omitted.

> all protocols include in-band signalling somewhere That's an incredibly reductionistic view of the world that's utterly useless for anything (including actually engineering systems) except pedantry. It's obvious that the level at which you include control information is meaningful and significantly affects the design of the protocol, as we see in the submission. Directly embedding the control information into the m…

No, it's not reductionist and pedantic. It's a reminder that there is no magic. Building an abstraction layer that separates control and data doesn't win you anything if, like the people in the article, you then forget it's a thing and write directly to the level below it.

> No, it's not reductionist and pedantic.

It's very reductionistic, because it intentionally ignores meaningful detail, and it's pedantic because it's making a meaningless distinction.

> It's a reminder that there is no magic.

This is irrelevant. Nobody is claiming that there's any magic. I'm pointing out the true fact that details about the abstraction layers matter.

In this case, the abstraction layer was poorly-designed.

Good abstraction layer: length prefix, or JSON encoding.

Bad abstraction layer: "the body of the email is mostly plain text, except when there's a line that only contains a single period".

There are very, very few problems to which the latter is a good solution. It is a bad engineering decision, and it also obfuscates the fact that there even is an abstraction layer unless you carefully read the spec.

-------------

In fact, the underlying problem goes deeper than that - the design of SMTP is intrinsically flawed because it's a text-based ad-hoc protocol that has in-band signaling.

There are very few good reasons to use a text-based data interchange format. One of them is to make the format self-documenting, such that people can easily read and write it without consulting the spec.

If the spec is complex enough that you get these ridiculous footguns, then it shouldn't be text-based in the first place. Instead, it should be binary - then you have to either read the spec or use someone else's implementation.

Failing that, use a standardized structured format like XML or JSON.

But there's no excuse for the brain-dead approach that SMTP took. They didn't even use length prefixing,

Re: The curious case of the missing period

#145

Post by talkingtab »

This reminds me of an experience debugging a network protocol implementation - specifically AppleTalk NBP for other ancient people. I had coded everything but my packets were rejected (aka silently dropped) when real (aka Apple implementation) packets were not. I had a copy of the good and bad packets on the screen of my computer and had gone over them byte by byte to find the problem. And there was none. From start to finish they were exactly the same, with correct check sums etc. It was time to go home and I decided just to print the stupid things to look at later.

As soon as I printed them, the error was clear. My version ran to two pages and the good implementation one page. I had not been careful to clear the buffer before sending the data (mbufs don't you know).

This still cracks me up.

Re: The curious case of the missing period

#146

Post by akvadrako »

pja wrote: post #128
throw10920 wrote: post #103

Earlier quoted context omitted.

The real problem is that SMTP is a "plain-text" protocol that includes in-band signaling. It literally happened because SMTP defined "a line that only contains a single period in it" as a control sequence and not a literal line that only contains a single period in it. SMTP is an example of an unnecessarily complex design, and the implementation bugs reflect it. SMTP shouldn't be hard for someone to correctly impleme…

At some point, all protocols include in-band signalling somewhere: You have to put packets on the line, and those packets are ultimately just a stream of anonymous bytes. If it wasn’t a period, it would be something else & you’d have to handle that instead.

That isn't true at all. In most binary protocols, you put the size of the message in the header. Then any sequence of bytes is allowed to follow. There are no escape sequences.

Re: The curious case of the missing period

#147

Post by easyThrowaway »

sjf wrote: post #66
onlyrealcuzzo wrote: post #53

Earlier quoted context omitted.

> Zawinski's Law captures common market pressure on software solutions, stating that “every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.”

The modern version would seem to be every platform expands until it includes instant messaging.

I believe that's outdated already, the next...uh, expansion layer would be some sort of generative AI features.

In other words, every other platform expands until it can summarize emails.

Re: The curious case of the missing period

#148

Post by the_real_tjaart »

talkingtab wrote: post #145

This reminds me of an experience debugging a network protocol implementation - specifically AppleTalk NBP for other ancient people. I had coded everything but my packets were rejected (aka silently dropped) when real (aka Apple implementation) packets were not. I had a copy of the good and bad packets on the screen of my computer and had gone over them byte by byte to find the problem. And there was none. From start…

Thanks for sharing!

Re: The curious case of the missing period

#149

Post by nottorp »

throw10920 wrote: post #144
TeMPOraL wrote: post #140

Earlier quoted context omitted.

No, it's not reductionist and pedantic. It's a reminder that there is no magic. Building an abstraction layer that separates control and data doesn't win you anything if, like the people in the article, you then forget it's a thing and write directly to the level below it.

> No, it's not reductionist and pedantic. It's very reductionistic, because it intentionally ignores meaningful detail, and it's pedantic because it's making a meaningless distinction. > It's a reminder that there is no magic. This is irrelevant. Nobody is claiming that there's any magic. I'm pointing out the true fact that details about the abstraction layers matter . In this case, the abstraction layer was poorly-d…

Why not protobufs inside protobufs then?

Re: The curious case of the missing period

#150

Post by andruby »

xg15 wrote: post #116
evmar wrote: post #5

> A portion of this code implemented a SMTP client. If I wanted to root cause this, the real problem is right there. Implementing protocols correctly is hard and bugs like in the post are common. A properly implemented SMTP client library, like one you would pull off the shelf, would accept text and encode it properly per the SMTP protocol, regardless of where the periods were in the input. The templating layer shoul…

Sounds like a great idea, until you find that your SMTP library pulls in 5 other libraries as its own dependencies and those each pull in 3 transitive dependencies of their own, one being some kitchen sink/toolbox project where only 1% of its code is actually relevant to the dependant and the rest is dead weight - but which pulls in 20 more dependencies for functions that are literally never called in your project -…

I agree with this principle in general.

But for SMTP libraries, that's often part of stdlib (Ruby, Python, PHP, ...).

Post reply on HN