Live data from Hacker News

How not to design a wire protocol

esr.ibiblio.org

41–50 of 71 posts

Re: How not to design a wire protocol

#41
post #14

If one of my engineers brought me this "protocol" (um, protocols require state machines, where is it?), we would be having a series of long talks to figure out whether he needs to be educated or fired. What happens if I want to run NTP on an ARM M4 microcontroller with a lithium coin cell battery? Because, you know, perhaps I actually might want my time to be accurate on devices that even outship cell phones? Sending…

> Please quit giving ESR a platform when it's quite clear he really sucks as a programmer.

I am not sure this post complies with Hacker News guidelines.

Re: How not to design a wire protocol

#42

Having written trading applications with binary, JSON, and FIX protocols: this article is fucking terrible. 1) scanf on a floating point number works differently on different platforms. If you explore the space of numbers that are expressible, you will find very different results in terms of floating point error depending on which implementation you use. Therefore transmitting floating point in ascii is just asking f…

Not terrible.

I have learned some things about how the NTP protocol was designed. And it is a good summary of trade-offs involved. You may or may not agree with them, and they may or may not be important to you.

However, I have learned even more things about how absolutely mind-bogglingly huge the egos of some posters here are.

Re: How not to design a wire protocol

#43

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

IDK if you would say that ASN.1 (DER) is 'bit packed', but I still kind of like it.

I had some young feller tell me some time back why ASN.1 was the anti-christ, but I can't remember (for the life of me why). Do you happen to know/remember why ASN.1 is 'bad'?

I can write 8-bit assembler to it - I can write 32- or 64- bit compiled C to it, and it's pretty easy to create an FPGA pre-processor (and router) for it, and it certainly is more constrained than random JSON strings. What's wrong with ASN.1? Too old?

Re: How not to design a wire protocol

#44

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

One of my bosses could read the raw x.400 OSI networking packets and tell which ADMD it was from their botched implementation cough sprint cough I had to resort to a 409 dump

Well, I’m not that good but when I see our raw data I can pick out the fields and read the values easily enough. You really do just get used to a system and how to read it. There’re patterns to it.

Re: How not to design a wire protocol

#45
Ouch! Object lesson: the Internets lets smart people write dumb things and sometimes they end up on the front page of HN.

I shouldn't pile on (Sorry Eric Raymond), but there's this one:

> A decimal digit string is a decimal digit string; there’s no real ambiguity about how to interpret it

The context is as contrasted to a 64-bit big endian value.

Of course, a decimal digit string is subject to its binary encoding (no different than anything else sent as bits).

I know the author is talking about JSON, but I've see a lot of different ways the length of decimal digit strings determined: null terminator, double-quote terminator, single-quote terminator, length is a twos-complement 16-bit, 32-bit value before the first character of the string. (I think maybe 16-bit is know as a pstring or Pascal string? My memory is not 100% here.) I'll bet someone's done a 64-bit value before the string, though I haven't seen it myself. Oh, and I've seen where the length is determined by knowledge of the data structure (that is, something like bytes 10-25 are a name, padded with spaces or null terminators, usually leaving readers to infer the encoding based on the dominant platform). And once you start terminating a string with a certain sequence of bits, there's an escaping mechanism you need to deal with. Let's look at the source code for a quality JSON parser before we call it unambiguous?

I mean, on my first paying gig of my life I made $50 writing some sample code for a BASIC tutorial. The first version of it was rejected because it didn't work right on their EBSIDIC system. I was 16 and I was thinking, "what the heck (I actually swear back then) is EBSIDIC?!?") I know we all use ASCII and Unicode now, and IIRC, the actual digits 0-9 were the same, but not the decimal point, so maybe parsing integers is OK but not floating point values. Speaking of which, using . for the decimal point is not exactly universal (even forgetting about EBSIDIC, and let's please do)...

JSON has it's rules (which is good!) but my point is: a decimal digit string not necessarily a simple thing. I realize the author doesn't know this, and I don't begrudge him -- I am certainly not happier for knowing otherwise -- I'm just trying to point out that the authors of NTPv4 were not exactly working in a era where a good programmer could possibly think a decimal digit string was anything but a hornets nest sitting on a land mine guarded by MCP (Tron reference, sorry).

So... the author complains that parsing an NTPv4 packet requires prior knowledge of things like big endianness, but parsing JSON requires plenty of prior knowledge.

I get it: big- vs. little-endian is not something people are used to dealing with these days, so it jumps up and bites you when you do. But it's just another encoding and is actually much, much simpler than ones you deal with every day.

(Back then, all the cool CPUs were big endian so I think it was pretty understandable how it ended up on the wire.)

-----

Does it matter? YES (well, not my personal anecdotes! but the other bits).

The dead truth is: you're going to have to understand and parse the messages you receive and they may or may not use conventions and idioms you already understand.

Re: How not to design a wire protocol

#46
post #43

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

IDK if you would say that ASN.1 (DER) is 'bit packed', but I still kind of like it. I had some young feller tell me some time back why ASN.1 was the anti-christ, but I can't remember (for the life of me why). Do you happen to know/remember why ASN.1 is 'bad'? I can write 8-bit assembler to it - I can write 32- or 64- bit compiled C to it, and it's pretty easy to create an FPGA pre-processor (and router) for it, and i…

ASN.1 has multiple ways to serialize the same message which opens it up to bugs in rarely-used code paths. And every ASN.1 implementation was rife with security holes.

But mostly ASN.1 comes from the same "bad neighborhood" as CORBA, X.400, X.509, OSI protocols, etc.

Re: How not to design a wire protocol

#47
post #46
post #43

Earlier quoted context omitted.

IDK if you would say that ASN.1 (DER) is 'bit packed', but I still kind of like it. I had some young feller tell me some time back why ASN.1 was the anti-christ, but I can't remember (for the life of me why). Do you happen to know/remember why ASN.1 is 'bad'? I can write 8-bit assembler to it - I can write 32- or 64- bit compiled C to it, and it's pretty easy to create an FPGA pre-processor (and router) for it, and i…

ASN.1 has multiple ways to serialize the same message which opens it up to bugs in rarely-used code paths. And every ASN.1 implementation was rife with security holes. But mostly ASN.1 comes from the same "bad neighborhood" as CORBA, X.400, X.509, OSI protocols, etc.

Fair enough - there were bugs in specs (like the X.509 letting you attach arbitrarily large image or blobs), but mostly I found that the compilers were buggy.

Whenever (way back in the day) we (our/my company) did SET (secure 3P "secure" credit card protocol) with competitors (MS/HP/RSA/IBM/Netscape/etc.), because we compiled an interpreter from the spec, we were able to put in code-path switches depending on the counter party and adapt. Since they had a buggy compiler from a 3rd party - they could not.

Was that an issue with ASN.1? Or crappy tooling that people used?

Re: How not to design a wire protocol

#48

IIRC, Silicon Graphics (the company) used to maintain a registry for "magic numbers" designating each file's type. Customers could contact SGI to be issued a magic number for whatever file formats they were cooking up. I wonder if there's merit to recreating such a thing under ICANN, where the issued serial numbers are useful for file types, wire protocols, etc. Then anyone needing to reliably interpret a packet coul…

If you want something like that, using an IEEE OUI/CID would be a good start.

https://standards.ieee.org/content/dam/ieee-standards/standa...

Re: How not to design a wire protocol

#50
post #4
post #2

Not mentioned in the article, but bit packed data structures are easier to interpret in hardware and offer more guarantees. If I'm designing an FPGA/ASIC I can predict how many clock cycles a statically sized data structure will need to move through my system, and how much RAM I need while it's being processed. I have spent a lot of time working with FPGAs that process network packets and many of the performance guar…

I was a bit surprised myself that the author prefers JSON to bit-packed protocols, but given that the author is none other than Eric Raymond, I have to take him seriously. Accidentally, in my work lately I used JSON for data exchange over the network where performance is not important, and MsgPack otherwise where it is (which is essentially a packed JSON).

Appeal to authority, huh? Do you also take all of his many racist statements seriously, "just because he's none other than Eric Raymond"?

"The average IQ of the Haitian population is 67... Haiti is, quite literally, a country full of violent idiots." -Eric S Raymond

"... The minimum level of training required to make someone effective as a self defense shooter is not very high... unfortunately, this doesn't cover the BLM crowd, which would have an average IQ of 85 if it's statistically representative of American blacks as a whole. I've never tried to train anyone that dim and wouldn't want to." -Eric S Raymond

https://twitter.com/tqbf/status/780839196231630848

(Note: this is just the tip of the shitberg. There are SO MANY MORE examples on so many other topics (like "Is the casting couch fair trade?") from so many other times over the decades.)

Post reply on HN