A case against text protocols
unmdplyr-new.bearblog.dev
A case against text protocols
1–10 of 23 posts
Re: A case against text protocols
#2Back when HTTP and SMTP were designed, the Internet was mostly old Unix machines talking to each other. Everything was a file full of `char`, piped to an 80-column terminal. Text-based made sense. Decades later, when the computer world was bigger, faster, and more unified, other systems kind of cargo-culted off of those earlier successes. And isn't it neat that you can telnet into port 80?
I think another big reason text-based protocols are seductive is that they're an engineering path of least resistance. When you start off text-based, how to debug and analyze and interoperate with other implementations can be put off for later, or be Someone Else's Problem. Whereas if you design the same protocol but in binary, these tricky considerations are harder to ignore -- even though text-based protocols will still run into the same problems eventually, because there's no way I'm typing in a Cookie header by hand or decoding Base64 in my head.
Re: A case against text protocols
#3Another argument: text-based protocols often admit too many degrees of freedom in constructing messages, the handling of which is left underspecified and completely overlooked during implementation. (What happens if you separate lines with LF instead of CRLF in HTTP headers? What if the opening and closing HTML tags don't match? I know this should not usually happen, but how should I handle it when it does anyway?)
It's not by any means exclusive to text-based protocols, but there's this tendency to assume everything about a text-based protocols is ‘obvious’, ‘self-documenting’ and doesn't need specifying, and to think that just because the individual elements of the protocol are human-readable, this will somehow magically make the computers using the protocol follow the Gricean maxims (if it doesn't make sense, nobody will ever say that, therefore I don't need to think about it).
Re: A case against text protocols
#4Another argument: text-based protocols often admit too many degrees of freedom in constructing messages, the handling of which is left underspecified and completely overlooked during implementation. (What happens if you separate lines with LF instead of CRLF in HTTP headers? What if the opening and closing HTML tags don't match? I know this should not usually happen, but how should I handle it when it does anyway?) I…
> the handling of which is left underspecified
I used to see Postel's Law ("be conservative in what you send, be liberal in what you accept") quoted as some sort of antidote, but it seems to have fallen out of fashion -- I think enough people saw how that ideal played out in reality. Nowadays a JSON library feels justified throwing a fit if it sees a comment string instead of playing along with such shenanigans.
> It's not by any means exclusive to text-based protocols
Plus, I would argue text-based greatly increases the surface area for ambiguity, whereas, for instance, there are only a few ways a sane person would send an integer as bytes.
Re: A case against text protocols
#5Another argument: text-based protocols often admit too many degrees of freedom in constructing messages, the handling of which is left underspecified and completely overlooked during implementation. (What happens if you separate lines with LF instead of CRLF in HTTP headers? What if the opening and closing HTML tags don't match? I know this should not usually happen, but how should I handle it when it does anyway?) I…
> the handling of which is left underspecified I used to see Postel's Law ("be conservative in what you send, be liberal in what you accept") quoted as some sort of antidote, but it seems to have fallen out of fashion -- I think enough people saw how that ideal played out in reality. Nowadays a JSON library feels justified throwing a fit if it sees a comment string instead of playing along with such shenanigans. > It…
I’d say it’s played out ... ambiguously. Conservative HTTP unworkable, liberal TLS dangerous. JSON for internal APIs and data succeeded because it’s conservative, XHTML and XML+XSLT on the open web failed for the same reason. Postel’s law is less of a universal principle than it initially seemed to be, sure, but it appears to me that part of the reason for its increasing irrelevance is our moving away from open ecosystems and not deficiencies valid in its original context.
Integer encoding (as opposed to e.g. encoding of opaque binary strings) actually appears to be a bad example to me: various universal binary encoding protocols, self-describing or not, have an astounding number of unsigned and signed integer encodings among them. It’s like inventing a new one is a rite of passage or something.
Re: A case against text protocols
#6Earlier quoted context omitted.
> the handling of which is left underspecified I used to see Postel's Law ("be conservative in what you send, be liberal in what you accept") quoted as some sort of antidote, but it seems to have fallen out of fashion -- I think enough people saw how that ideal played out in reality. Nowadays a JSON library feels justified throwing a fit if it sees a comment string instead of playing along with such shenanigans. > It…
I’d say it’s played out ... ambiguously. Conservative HTTP unworkable, liberal TLS dangerous. JSON for internal APIs and data succeeded because it’s conservative, XHTML and XML+XSLT on the open web failed for the same reason. Postel’s law is less of a universal principle than it initially seemed to be, sure, but it appears to me that part of the reason for its increasing irrelevance is our moving away from open ecosy…
I see your point, but I don't agree about why XHTML failed. For starters, see: https://en.wikipedia.org/wiki/WHATWG (Basically, XHTML failed because it was a pointless boondoggle, whereas HTML5 very much wasn't.)
Regarding binary integers, having written code for a few common binary protocols and file formats I've never had to think very hard about it (just: How long? Which endian? Signed?) but maybe it's different for older or more esoteric stuff.
Re: A case against text protocols
#7Back when HTTP and SMTP were designed, the Internet was mostly old Unix machines talking to each other. Everything was a file full of `char`, piped to an 80-column terminal. Text-based made sense. Decades later, when the computer world was bigger, faster, and more unified, other systems kind of cargo-culted off of those earlier successes. And isn't it neat that you can telnet into port 80? I think another big reason…
And yet I have definitely copy&pasted cookie headers from the database or log files into the browser - something I’ve never done it could imagine doing with binary protocols
Re: A case against text protocols
#8> It is as simple as dumping struct ntp_packet on wire and reading it off it -- no parsing involved except for calling ntohX()/htonX() on all fields except li, vn and mode.
Nope, you may still need to call ntoh/hton, depending on how the compiler you use orders the bitfields inside an int. Plus you need "__attribute__((packed))" or whatever the compiler you use supports to make that C struct definition mean what it looks like it means: even then I am not sure those three bitfields are required to occupy exactly 8 bits.
Re: A case against text protocols
#9Earlier quoted context omitted.
I’d say it’s played out ... ambiguously. Conservative HTTP unworkable, liberal TLS dangerous. JSON for internal APIs and data succeeded because it’s conservative, XHTML and XML+XSLT on the open web failed for the same reason. Postel’s law is less of a universal principle than it initially seemed to be, sure, but it appears to me that part of the reason for its increasing irrelevance is our moving away from open ecosy…
I see your point, but I don't agree about why XHTML failed. For starters, see: https://en.wikipedia.org/wiki/WHATWG (Basically, XHTML failed because it was a pointless boondoggle, whereas HTML5 very much wasn't.) Regarding binary integers, having written code for a few common binary protocols and file formats I've never had to think very hard about it (just: How long? Which endian? Signed?) but maybe it's different f…
Er, no, that’s not what I was referring to. The XHTML 2 story was stupid, yes (though I think the RDF / “Linked Data” tooling could’ve been really nice had it not been a fantasy), but lots and lots of people were willing to give XHTML 1.1 a chance during the XML craze and the original web standards push; except the HTML 4.01 Strict rules which XHTML 1.1 enforced were complicated enough that nobody ended up willing to tolerate showing the user literally nothing for every fumble in a server-side script. (Part of the problem was that people were routinely generating markup from textual templates.)
Re: A case against text protocols
#10There is one thing that is often neglected in text vs binary protocols debate and that is self-terminating vs prior-length. Although it is not strictly connected, text protocols are usually self-terminating (e.g. closing tags), while binary protocols are usually prior-length (e.g. type-length-value approach). The first approach leads to escaping and all associated problems.