Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to design a wire protocol

esr.ibiblio.org

Post reply on HN
Print view View original

11–20 of 71 posts

Re: How not to design a wire protocol

#11

Post by mcherm »

I wonder whether we can have the benefits of both.

Sounds like the bit-packed protocols save bits (that's clearly good), while the other protocols are self-documenting. Documenting is good.

Why can't we have both? Something like protocol buffers (yes, Raymond mentions them in the article) is a binary protocol that makes pretty efficient use of the bits on the wire. But they are also very well documented. And it's "documented" that is useful, not "documentation is included in every message that gets sent".

Is it possible to look at the on-wire protocol and tell that this message uses protocol buffers and which protocol specification it is using? (I'm not sure. I hope the answer is yes.) Is it possible to find the documentation for a particular protocol buffers specification once you know which one it is? (I think it is, if by no other means than a google search, although a more automated repository might be nice.) If both these things are true, then we can have bit-level efficiency AND have well-documented and extensible messages.

Re: How not to design a wire protocol

#12

Post by DoofusOfDeath »

IIRC, Silicon Graphics (the company) used to maintain a registry for "magic numbers" designating each file's type. Customers could contact SGI to be issued a magic number for whatever file formats they were cooking up.

I wonder if there's merit to recreating such a thing under ICANN, where the issued serial numbers are useful for file types, wire protocols, etc.

Then anyone needing to reliably interpret a packet could (1) look for the format serial number at some well-known location, and then (2) consult the well-publicized registry for whatever information has been provided regarding the format.

Re: How not to design a wire protocol

#13

Post by axaxs »

notacoward wrote: post #9

When the very first sentence is incorrect, the outlook for the rest looks grim. > A wire protocol is a way to pass data structures or aggregates over a serial channel between different computing environments. A real wire protocol involves not just the flow of data but the flow of control. Without knowing which data structures are replies to which others, issued under what circumstances and affecting which other parts…

How dare you insult the author of one quasi used json format!!

I don't like the article for the same reasons. It felt long-winded and didn't ultimately offer anything of value. What's worse is that it's written from a position of authority, common for this particular author, so some people will be duped into treating it as gospel.

Re: How not to design a wire protocol

#14

Post by bsder »

If one of my engineers brought me this "protocol" (um, protocols require state machines, where is it?), we would be having a series of long talks to figure out whether he needs to be educated or fired.

What happens if I want to run NTP on an ARM M4 microcontroller with a lithium coin cell battery? Because, you know, perhaps I actually might want my time to be accurate on devices that even outship cell phones?

Sending that message would be difficult without drift because of the huge number of bytes involved. Transmission time is far too long. I could go on and on...

If you want to see a relatively well designed protocol, go chew through the BLE (Bluetooth Low Enregy) spec. It's not perfect, but it shows you how to balance functionality vs. engineering (note the number of times you have a "length" parameter so that you can chew through your binary blobs even if you can't parse all of it).

Please quit giving ESR a platform when it's quite clear he really sucks as a programmer.

Re: How not to design a wire protocol

#15

Post by wmf »

pubby wrote: post #10

I'm ignorant on the subject, but is JSON really much better than SOAP? I thought everyone hated SOAP.

Those are different layers. JSON is a simpler alternative to XML. SOAP has its own complexity on top of the complexity of XML, so a "full" SOAP implementation is probably 100x more complex than, say, JSON-RPC.

Re: How not to design a wire protocol

#16

Post by axaxs »

JustSomeNobody wrote: post #3

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

There is a beauty to simple bit packed protocols. That said, they seem to always end up not a great idea. Wholesale changing protocols is hard, so people tend to try to tack features onto what exists. Look at the clusterfk that DNS has become, with various degrees of support, depending on the server.

Re: How not to design a wire protocol

#17

Post by wmf »

DoofusOfDeath wrote: post #12

IIRC, Silicon Graphics (the company) used to maintain a registry for "magic numbers" designating each file's type. Customers could contact SGI to be issued a magic number for whatever file formats they were cooking up. I wonder if there's merit to recreating such a thing under ICANN, where the issued serial numbers are useful for file types, wire protocols, etc. Then anyone needing to reliably interpret a packet coul…

Magic numbers are better than nothing, but "next protocol" fields accomplish the same thing without magic. Usually the steward of each protocol maintains a registry of protocol IDs. For example, IEEE maintains a list of Ethernet protocols: http://standards-oui.ieee.org/ethertype/eth.txt and IANA tracks IP protocols: https://www.iana.org/assignments/protocol-numbers/protocol-n... and TCP/UDP port numbers: https://www.iana.org/assignments/service-names-port-numbers/... and MIME types are used for HTTP and email: https://www.iana.org/assignments/media-types/media-types.xht...

In classic Mac OS and BeOS, file inodes carried file types so you didn't have to guess about file types either.

Re: How not to design a wire protocol

#18

Post by AnimalMuppet »

DoofusOfDeath wrote: post #12

IIRC, Silicon Graphics (the company) used to maintain a registry for "magic numbers" designating each file's type. Customers could contact SGI to be issued a magic number for whatever file formats they were cooking up. I wonder if there's merit to recreating such a thing under ICANN, where the issued serial numbers are useful for file types, wire protocols, etc. Then anyone needing to reliably interpret a packet coul…

IIRC, it was magic numbers and offsets. And you could run the charmingly-named magic program on a file to find out if the system knew what it was...

Re: How not to design a wire protocol

#19

Post by JustSomeNobody »

axaxs wrote: post #16
JustSomeNobody wrote: post #3

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

There is a beauty to simple bit packed protocols. That said, they seem to always end up not a great idea. Wholesale changing protocols is hard, so people tend to try to tack features onto what exists. Look at the clusterfk that DNS has become, with various degrees of support, depending on the server.

> Wholesale changing protocols is hard...

In my dozen plus years of working with them, I've not seen a large change like this. And TBH, there are way to structure them to make sure they handle changes.

I mostly deal with hardware that has to be stable in the field for years, so maybe that's the difference.

Re: How not to design a wire protocol

#20

Post by tenebrisalietum »

Consider if IPv4 source and destination addresses in IP datagrams were textual rather than 32-bit fields. IPv6 might not be necessary, NAT might not never have had to be hacked on to the Internet, and most of the centralized cloud model that is pervasive today may just not have had an opportunity to take hold.
Post reply on HN