Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to design a wire protocol

esr.ibiblio.org

Post reply on HN
Print view View original

31–40 of 71 posts

Re: How not to design a wire protocol

#31

Post by altfredd »

The article seems heavily influenced by author's personal preferences.

He starts from presenting false dichotomy (bit stream vs self-documenting text) and proceeds to apply his personal experience with proprietary GPS trackers to well-documented NTP protocol. He describes his favorite approach without mentioning it's downsides — and that approach is JSON! JSON!

By design, JSON format lacks any capacity for extensions. It's creators figured out, that backwards and forwards compatibility is more important that anything else, so they froze the specification at version 1 and refused to introduce new features or extension support. And thus JSON can't...

1) contain comments;

2) properly encode non-Latin text (no, — hexadecimal encoding is even worse than no encoding);

3) have more than one top-level element;

4) have any data types, except ones in JSON spec.

Each of those limitations has lead to creation of at least one incompatible JSON-like format, that can't be processed by spec-compalient JSON parsers. Pick a random piece of JSON from the wild, and you may find, that it isn't actually "JSON", but one of those quasi-JSON formats. To make matters worse, JSON spec didn't mention maximum supported number size/precision, so JSON payloads from one implementation may not properly decode on another implementation.

If he wants to design JSON-based NTP protocol, he is welcome to do so. But widely adopting such thing would be unwise — we already suffer from traffic amplification attack via NTP, and bigger packet lengths would make those worse.

Re: How not to design a wire protocol

#32

Post by mhandley »

axaxs wrote: post #16
JustSomeNobody wrote: post #3

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

There is a beauty to simple bit packed protocols. That said, they seem to always end up not a great idea. Wholesale changing protocols is hard, so people tend to try to tack features onto what exists. Look at the clusterfk that DNS has become, with various degrees of support, depending on the server.

On the other hand, you could look at SIP for an example of what happens when you've an extensible ASCII-based protocol, and everyone and their dog decides to extend it. It's not pretty, that's for sure. I've come to regard it as my success disaster.

Re: How not to design a wire protocol

#33

Post by robertAngst »

bsder wrote: post #14

If one of my engineers brought me this "protocol" (um, protocols require state machines, where is it?), we would be having a series of long talks to figure out whether he needs to be educated or fired. What happens if I want to run NTP on an ARM M4 microcontroller with a lithium coin cell battery? Because, you know, perhaps I actually might want my time to be accurate on devices that even outship cell phones? Sending…

Why would someone write a protocol in 2019?

Don't we already have formats? (EDIT: Like CAN or I2C?)

Re: How not to design a wire protocol

#34

Post by basementcat »

CoolGuySteve wrote: post #5

Having written trading applications with binary, JSON, and FIX protocols: this article is fucking terrible. 1) scanf on a floating point number works differently on different platforms. If you explore the space of numbers that are expressible, you will find very different results in terms of floating point error depending on which implementation you use. Therefore transmitting floating point in ascii is just asking f…

Unfortunately the binary representation of IEEE-754 may not look the same everywhere.

https://en.wikipedia.org/wiki/Double-precision_floating-poin...

https://en.wikipedia.org/wiki/IEEE_754#Reproducibility

Re: How not to design a wire protocol

#35

Post by walshemj »

JustSomeNobody wrote: post #3

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

One of my bosses could read the raw x.400 OSI networking packets and tell which ADMD it was from their botched implementation cough sprint cough

I had to resort to a 409 dump

Re: How not to design a wire protocol

#36

Post by deathanatos »

altfredd wrote: post #31

The article seems heavily influenced by author's personal preferences. He starts from presenting false dichotomy (bit stream vs self-documenting text) and proceeds to apply his personal experience with proprietary GPS trackers to well-documented NTP protocol. He describes his favorite approach without mentioning it's downsides — and that approach is JSON! JSON! By design, JSON format lacks any capacity for extensions…

> And thus JSON can't... […] properly encode non-Latin text (no, — hexadecimal encoding is even worse than no encoding);

This is (RFC) valid JSON:

  "我喜欢吃鱼。"
Sure, JSON has some corner cases. Binary protocols can, and do, as well. While I'm sure that non-compliant JSON examples exist in the wild, I would think that overall they're exceedingly rare compared to compliant ones.

And if you don't like the limitations of JSON, extending the format for your particular use-case is a valid solution. (Though I would argue that going w/ a well-known format that already supports your needs is a more pragmatic one.)

Re: How not to design a wire protocol

#37

Post by leoc »

altfredd wrote: post #31

The article seems heavily influenced by author's personal preferences. He starts from presenting false dichotomy (bit stream vs self-documenting text) and proceeds to apply his personal experience with proprietary GPS trackers to well-documented NTP protocol. He describes his favorite approach without mentioning it's downsides — and that approach is JSON! JSON! By design, JSON format lacks any capacity for extensions…

Doesn't it also have serious security issues, partly for related reasons?

Re: How not to design a wire protocol

#38

Post by mjevans »

JustSomeNobody wrote: post #3

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

Offhand, I agree but also want to suggest.

Error replies should also include at least a short-text response (often along side a numeric one).

Initial connection strings might also have a text-string that says something useful to humans, like what the protocol is; just like the various multimedia container formats that were developed on the Internet rather than by 'media companies'.

Push complexity as far up the protocol stack as possible; but don't sacrifice useful extensiblity at lower levels if it makes sense. At the same time don't depend on that data staying the same (if it exists in another layer, it should be modifiable without breaking your actual protocol). FTP is a great example of a protocol that (because of connection multiplexing limitations) embeds data which should be low level in to a higher level.

If an RFC exists that describes the protocol then packed is probably OK. If no RFC described protocol or method exists, try to get as close as possible with off the shelf stuff and prototype with human readable things where possible on top of that until solid requirements for a new RFC are refined.

Re: How not to design a wire protocol

#39

Post by jcranmer »

tptacek wrote: post #22
akerl_ wrote: post #7

Earlier quoted context omitted.

I feel like the fact that it's ESR causes me to take him less seriously. I'm basing that on the other contents of his blog, such as http://esr.ibiblio.org/?p=7239 , http://esr.ibiblio.org/?p=26 , and http://esr.ibiblio.org/?p=6907 .

We don't even need to get into his (execrable) politics; he's substantively bad on technical issues as well. http://esr.ibiblio.org/?p=6839

Don't worry, if you look into the comments, the politics really comes out in force.

Re: How not to design a wire protocol

#40

Post by tptacek »

jcranmer wrote: post #39
tptacek wrote: post #22

Earlier quoted context omitted.

We don't even need to get into his (execrable) politics; he's substantively bad on technical issues as well. http://esr.ibiblio.org/?p=6839

Don't worry, if you look into the comments, the politics really comes out in force.

Did he manage to work white nationalism into binary versus ascii wire formats? I've seen him make some pretty impressive leaps before but that would be outdoing himself.
Post reply on HN