Live data from Hacker News

Open Sourcing the Stupid-Simple Messaging Protocol

aerofs.com

21–30 of 32 posts

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#21
post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

I used netstrings in Firestr's (https://github.com/mempko/firestr/blob/master/src/util/menco...) protocol. I picked them precisely because it is safer to parse.

Though I suspect my parsing code still have bugs and exploits.... Looks like I need to put a guard in get_util for the max size.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#22
post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

My test for this kind of thing is this: suppose you need to make a high speed implementation in Verilog for an FPGA or ASIC. By high speed, I mean that you need to process multiple characters per cycle (say a 64-bit word at a time)- if you have to make a decision on a byte-by-byte basis, it's too slow. This is a very possible scenario if the protocol catches on: for example, I made HDLC byte stuffing for PPP framing, 4 characters at a time (at least better than bit stuffing).

The CRLF encoding is not so bad in this case: read 8 characters, in parallel detect CR. If there is no CR in your word, just append the data to the buffer. When you do have a CR, it's a big pain: you need to save the last word with byte masks, then shift any remaining for the next input (and the entire next string is shifted by this left-over balance). You could try to make all strings a multiple of 8 in length to avoid this, but this adds overhead to the message so is inefficient- the hardware will just have to do it.

OK, so now in your new format the hardware has to parse a variable length decimal number and convert it to binary (ideally in parallel), very fun! You could make the conversion byte at a time, but it's slow. You need to implement overflow detection.

At the very least use hex instead of decimal. Even in software you may need overflow detection. This is easy in hex, not so much in decimal. Better is to require the number to be a multiple of four or eight digits, even though this is a waste of bandwidth.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#23

Earlier quoted context omitted.

Why wouldn't you just include a length field on strings/bytes, allowing the protocol to be "binary clean" and avoid the base64 problem entirely? This is one of the most annoying things about XMPP (even sending contact photos hits this!), so if replacing XMPP ...

A big advantage of LF-delimited over length-prefixed messages is netcat/telnet-friendliness. That was more valuable to us than being binary-clean as our use cases do not involve sending large binary messages.

I think you might want to make a distinction between a stream packet and a completed message.

If you're going for telnet compatibility then you'll want to terminate packets in CR+LF, but possibly expect to see only CR or LF from the client (ASCII mode).

Your stream could either be stateful (a message is always sent complete and in order, even if it takes multiple stream packets) or stateless* (different messages might have stream packets consecutively).

It would be more future proof if you started with a message grammar and then defined your protocol on top of that.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#24

Earlier quoted context omitted.

Why wouldn't you just include a length field on strings/bytes, allowing the protocol to be "binary clean" and avoid the base64 problem entirely? This is one of the most annoying things about XMPP (even sending contact photos hits this!), so if replacing XMPP ...

A big advantage of LF-delimited over length-prefixed messages is netcat/telnet-friendliness. That was more valuable to us than being binary-clean as our use cases do not involve sending large binary messages.

A binary-friendly client would be a hundred lines of code at most, versus inefficiency for a pretty standard protocol use-case forever.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#26
post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

After reading Swartz's post, I realize my own crush on djb is small by comparison.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#27
post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

If you use netstrings, why not take it a step further and just use bencoding?

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#29

Earlier quoted context omitted.

Why wouldn't you just include a length field on strings/bytes, allowing the protocol to be "binary clean" and avoid the base64 problem entirely? This is one of the most annoying things about XMPP (even sending contact photos hits this!), so if replacing XMPP ...

A big advantage of LF-delimited over length-prefixed messages is netcat/telnet-friendliness. That was more valuable to us than being binary-clean as our use cases do not involve sending large binary messages.

You know you could have a length-prefix and a new-line. Best of both worlds.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#30
post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

My test for this kind of thing is this: suppose you need to make a high speed implementation in Verilog for an FPGA or ASIC. By high speed, I mean that you need to process multiple characters per cycle (say a 64-bit word at a time)- if you have to make a decision on a byte-by-byte basis, it's too slow. This is a very possible scenario if the protocol catches on: for example, I made HDLC byte stuffing for PPP framing,…

I assume you're coming from a hardware perspective - HDLC/PPP parsing in hardware might make sense in niche cases; though most protocols benefit much more from the flexibility and upgrade possibilities of a software implementation.

In this case, it is a messaging protocol. The incoming message essentially must be copied somewhere, therefore space must be allocated to store it, and therefore the length must be known.

CRLF require either two passes (one to get string length, another to copy data) or continuously expanding storage, both of which are significantly more expensive than just parsing a short number in the beginning of the string.

Post reply on HN