Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to design a wire protocol

esr.ibiblio.org

Post reply on HN
Print view View original

51–60 of 71 posts

Re: How not to design a wire protocol

#51

Post by DonHopkins »

outworlder wrote: post #42
CoolGuySteve wrote: post #5

Having written trading applications with binary, JSON, and FIX protocols: this article is fucking terrible. 1) scanf on a floating point number works differently on different platforms. If you explore the space of numbers that are expressible, you will find very different results in terms of floating point error depending on which implementation you use. Therefore transmitting floating point in ascii is just asking f…

Not terrible. I have learned some things about how the NTP protocol was designed. And it is a good summary of trade-offs involved. You may or may not agree with them, and they may or may not be important to you. However, I have learned even more things about how absolutely mind-bogglingly huge the egos of some posters here are.

Funny that you should mention mind-bogglingly huge egos in a discussion about ESR:

> “Are you” she asked “the most famous programmer in the world?”

> This was a question which I had, believe it or not, never thought about before. But it’s a reasonable one to ask...

http://esr.ibiblio.org/?p=6839

Re: How not to design a wire protocol

#52

Post by DonHopkins »

tptacek wrote: post #24
xg15 wrote: post #23

So, when can we expect his JSON serialisation of TLS?

When someone else writes it, Russell Nelson takes it over, and then gets bored enough of it to let Eric Raymond claim it for his resume.

When Russell's not too busy nominating his own wikipedia page for deletion because he wasn't allowed to whitewash the "Blacks Are Lazy" incident.

Re: How not to design a wire protocol

#53

Post by Gibbon1 »

mjevans wrote: post #38
JustSomeNobody wrote: post #3

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

Offhand, I agree but also want to suggest. Error replies should also include at least a short-text response (often along side a numeric one). Initial connection strings might also have a text-string that says something useful to humans, like what the protocol is; just like the various multimedia container formats that were developed on the Internet rather than by 'media companies'. Push complexity as far up the proto…

> Error replies should also include at least a short-text response (often along side a numeric one).

I always try and design in a debug mode. Turn it on and the destination will try to tell you exactly what you did wrong instead of stonewalling you.

Painful experience has taught me that you really want unique start and stop tokens, message type, message length, rev field and checksum/mac always. That at least allows you to mechanically validate and dispatch packets/messages.

Re: How not to design a wire protocol

#54

Post by DonHopkins »

outworlder wrote: post #41
bsder wrote: post #14

If one of my engineers brought me this "protocol" (um, protocols require state machines, where is it?), we would be having a series of long talks to figure out whether he needs to be educated or fired. What happens if I want to run NTP on an ARM M4 microcontroller with a lithium coin cell battery? Because, you know, perhaps I actually might want my time to be accurate on devices that even outship cell phones? Sending…

> Please quit giving ESR a platform when it's quite clear he really sucks as a programmer. I am not sure this post complies with Hacker News guidelines.

A better reason not to give ESR a platform is that his racism doesn't comply with Hacker News guidelines.

Re: How not to design a wire protocol

#55

Post by AgentME »

outworlder wrote: post #42
CoolGuySteve wrote: post #5

Having written trading applications with binary, JSON, and FIX protocols: this article is fucking terrible. 1) scanf on a floating point number works differently on different platforms. If you explore the space of numbers that are expressible, you will find very different results in terms of floating point error depending on which implementation you use. Therefore transmitting floating point in ascii is just asking f…

Not terrible. I have learned some things about how the NTP protocol was designed. And it is a good summary of trade-offs involved. You may or may not agree with them, and they may or may not be important to you. However, I have learned even more things about how absolutely mind-bogglingly huge the egos of some posters here are.

Yeah I'm really surprised this article is getting as much shit as it is. There's details it misses, but it seems like a good enough intro to some of the trade-offs between fixed-size messages and self-describing ones, and designing for extensibility.

Re: How not to design a wire protocol

#56

Post by syrrim »

alexeiz wrote: post #4
zinkem wrote: post #2

Not mentioned in the article, but bit packed data structures are easier to interpret in hardware and offer more guarantees. If I'm designing an FPGA/ASIC I can predict how many clock cycles a statically sized data structure will need to move through my system, and how much RAM I need while it's being processed. I have spent a lot of time working with FPGAs that process network packets and many of the performance guar…

I was a bit surprised myself that the author prefers JSON to bit-packed protocols, but given that the author is none other than Eric Raymond, I have to take him seriously. Accidentally, in my work lately I used JSON for data exchange over the network where performance is not important, and MsgPack otherwise where it is (which is essentially a packed JSON).

Even the greats make mistakes:

https://media.8ch.net/file_store/685c60d14c5b269cb9571c1f630...

Re: How not to design a wire protocol

#57

Post by zenexer »

ESR may be a horrible person, and this article may get too caught up in flawed technical examples, but there is an underlying point here that’s important.

When designing any system, be it a wire protocol or anything else, it’s tempting to optimize for metrics that are easy to measure and forget about metrics that are hard to measure. Humans are expensive. Time is expensive. It may very well be worth using a little extra bandwidth to minimize development and debugging costs. That won’t always be the case, but it’s an important question to ask while you’re still in the designing stages.

Any decent programmer can see that the technical arguments here are flawed: NTP, by nature, needs to be very predictable and use as few bytes as possible, or embedded systems are going to run into issues. That’s a hard technical requirement, not a matter of optimization. Unfortunately, that oversight, combined with the author’s poor track record, detract from an important point. Sadly, the author has chosen to present his argument as a misguided rant about a particular protocol rather than a strong theoretical debate over the pros and cons of different optimization goals.

Re: How not to design a wire protocol

#58

Post by altfredd »

deathanatos wrote: post #36
altfredd wrote: post #31

The article seems heavily influenced by author's personal preferences. He starts from presenting false dichotomy (bit stream vs self-documenting text) and proceeds to apply his personal experience with proprietary GPS trackers to well-documented NTP protocol. He describes his favorite approach without mentioning it's downsides — and that approach is JSON! JSON! By design, JSON format lacks any capacity for extensions…

> And thus JSON can't... […] properly encode non-Latin text (no, — hexadecimal encoding is even worse than no encoding); This is (RFC) valid JSON: "我喜欢吃鱼。" Sure, JSON has some corner cases. Binary protocols can, and do, as well. While I'm sure that non-compliant JSON examples exist in the wild, I would think that overall they're exceedingly rare compared to compliant ones. And if you don't like the limitations of JSO…

> This is (RFC) valid JSON:

You mean, "valid, according to the latest 2017 RFC". Such young RFC is still to raw, too immature to adopt, especially if it concerns data interchange formats. IPv6 was created in 1995, and it apparently still too young!

I fear, that a proper full-featured JSON spec, with comment support, mandatory UTF-8 and strict prohibition of hex-encoding won't be created and implemented by most JSON parsers till at least 2090. At that point the JSON format itself will likely become insufficiently hip for general use (just like XML suddenly stopped being hip enough in early 2000's).

Re: How not to design a wire protocol

#60

Post by mjevans »

Gibbon1 wrote: post #53
mjevans wrote: post #38

Earlier quoted context omitted.

Offhand, I agree but also want to suggest. Error replies should also include at least a short-text response (often along side a numeric one). Initial connection strings might also have a text-string that says something useful to humans, like what the protocol is; just like the various multimedia container formats that were developed on the Internet rather than by 'media companies'. Push complexity as far up the proto…

> Error replies should also include at least a short-text response (often along side a numeric one). I always try and design in a debug mode. Turn it on and the destination will try to tell you exactly what you did wrong instead of stonewalling you. Painful experience has taught me that you really want unique start and stop tokens, message type, message length, rev field and checksum/mac always. That at least allows…

Cannot upvote enough building it //in//; and shipping it. Conformant implementations must include it and allow admins to enable it.

You reminded me of the horror of talking to closed source things where there isn't even useful server-side debugging data. Stuff just fails or gets dropped without informing anyone why it went bad.

This also applies to things like my credit card - I would love to have the last week or two of even /failed/ attempts at using my card in my online statement. That would really, really help with figuring out if someone was trying to use my card, or if a given service that rejected use of my card even tried to hit the CC company. (This happened with a major travel site which probably did it's own processing; having a firm direction to push and solid data might have helped.)

Post reply on HN