Open Sourcing the Stupid-Simple Messaging Protocol • Hacker News Board

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#11

Post by todd8 » Thu, Sep 10, 2015, 5:08 PM UTC

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QMQP. Netstrings are easier to handle and are still easy to debug if verbs are expressed in ASCII while still handling arbitrary binary data equally well.

Netstrings are so brilliantly simple, see the wikipedia page: https://en.wikipedia.org/wiki/Netstring

This is what DJB says about the netstrings [1]:

> The famous Finger security hole may be blamed on Finger's use of the CRLF encoding. In that encoding, each string is simply terminated by CRLF. This encoding has several problems. Most importantly, it does not declare the string size in advance. This means that a correct CRLF parser must be prepared to ask for more and more memory as it is reading the string. In the case of Finger, a lazy implementor found this to be too much trouble; instead he simply declared a fixed-size buffer and used C's gets() function. The rest is history.

> In contrast, as the above sample code shows, it is very easy to handle netstrings without risking buffer overflow. Thus widespread use of netstrings may improve network security.

[1] http://cr.yp.to/proto/netstrings.txt

BTW, see Aaron Swartz's blog post on DJB, http://www.aaronsw.com/weblog/djb

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#12

Post by vonsnowman » Thu, Sep 10, 2015, 5:19 PM UTC

todd8 wrote: post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

Netstrings look neat although I find the use of a redundant comma delimiter somewhat confusing.

Buffer overflow concerns are not applicable to SSMP however, as the spec explicitly restricts message size to a maximum of 1024 bytes.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#13

Post by pbreit » Thu, Sep 10, 2015, 5:39 PM UTC

Sorry for being dense but what are the prime use cases for this? Is it for Jabber-style text messaging? Or inter-app MQ-style messaging?

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#14

Post by teacup50 » Thu, Sep 10, 2015, 5:40 PM UTC

vonsnowman wrote: post #7

thom_nic wrote: post #5

Also, since SSMP is completely text-based, does that mean the only way to send binary data is to base64-encode it? Or does it support a length header or chunking similar to HTTP?

base64 is the preferred way but any 8-bit encoding that doesn't use LF would also work. There is no support for a length header or chunking as SSMP is designed with small messages in mind.

Why wouldn't you just include a length field on strings/bytes, allowing the protocol to be "binary clean" and avoid the base64 problem entirely?

This is one of the most annoying things about XMPP (even sending contact photos hits this!), so if replacing XMPP ...

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#15

Post by mseebach » Thu, Sep 10, 2015, 5:46 PM UTC

vonsnowman wrote: post #12

todd8 wrote: post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

Netstrings look neat although I find the use of a redundant comma delimiter somewhat confusing. Buffer overflow concerns are not applicable to SSMP however, as the spec explicitly restricts message size to a maximum of 1024 bytes.

> The comma makes it slightly simpler for humans to read netstrings that are used as adjacent records, and provides weak verification of correct parsing.

Seems reasonable enough?

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#16

Post by dsr_ » Thu, Sep 10, 2015, 5:57 PM UTC

vonsnowman wrote: post #12

todd8 wrote: post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

Netstrings look neat although I find the use of a redundant comma delimiter somewhat confusing. Buffer overflow concerns are not applicable to SSMP however, as the spec explicitly restricts message size to a maximum of 1024 bytes.

Which for some reason is not in the abnf spec, and thus will be ignored up until somebody tries to implement this and fails a compatibility test.

Please put it in your abnf spec. People use them, you know.

Yes, it's hard to specify. The problem is that you have both an uncapped ID and an uncapped PAYLOAD in the same message. I recommend giving ID a max length of, say, 32, and PAYLOAD then has a max length of 951 if I'm counting right.

Or you could consider that an IPv6 path MTU is at least 1280, and use that (or 1232) as your per message bound instead of 1024. You're sending a packet, might as well get full value.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#17

Post by vonsnowman » Thu, Sep 10, 2015, 7:10 PM UTC

teacup50 wrote: post #14

vonsnowman wrote: post #7

Earlier quoted context omitted.

base64 is the preferred way but any 8-bit encoding that doesn't use LF would also work. There is no support for a length header or chunking as SSMP is designed with small messages in mind.

Why wouldn't you just include a length field on strings/bytes, allowing the protocol to be "binary clean" and avoid the base64 problem entirely? This is one of the most annoying things about XMPP (even sending contact photos hits this!), so if replacing XMPP ...

A big advantage of LF-delimited over length-prefixed messages is netcat/telnet-friendliness. That was more valuable to us than being binary-clean as our use cases do not involve sending large binary messages.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#18

Post by vonsnowman » Thu, Sep 10, 2015, 7:10 PM UTC

dsr_ wrote: post #16

vonsnowman wrote: post #12

Earlier quoted context omitted.

Netstrings look neat although I find the use of a redundant comma delimiter somewhat confusing. Buffer overflow concerns are not applicable to SSMP however, as the spec explicitly restricts message size to a maximum of 1024 bytes.

Which for some reason is not in the abnf spec, and thus will be ignored up until somebody tries to implement this and fails a compatibility test. Please put it in your abnf spec. People use them, you know. Yes, it's hard to specify. The problem is that you have both an uncapped ID and an uncapped PAYLOAD in the same message. I recommend giving ID a max length of, say, 32, and PAYLOAD then has a max length of 951 if I…

A very valid concern. We will work on this.

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#19

Post by davorb » Thu, Sep 10, 2015, 7:23 PM UTC

todd8 wrote: post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

Aren't those just P-strings with a semicolon?

https://en.wikipedia.org/wiki/String_%28computer_science%29#...

Re: Open Sourcing the Stupid-Simple Messaging Protocol

#20

Post by fazzone » Thu, Sep 10, 2015, 7:48 PM UTC

davorb wrote: post #19

todd8 wrote: post #11

The stupid-simple messaging protocol is interesting. However, I would prefer the use of something like D. J. Bernstein's netstrings rather than NL terminated strings. Netstrings have the advantage of having an explicit length field and thus allowing the receiver to allocate buffers of the appropriate size without the complications of receiving NL terminated data. Examples of protocols using netstrings are SCGI and QM…

Aren't those just P-strings with a semicolon? https://en.wikipedia.org/wiki/String_%28computer_science%29#...

Pascal-style strings have the length stored in the first byte, so a string can only be 255 characters long. Netstrings have the length in text at the front of the string instead, so you need only preallocate a buffer of 20 characters to store the text representation of string lengths up to 2^64 bytes.