It's like JSON. but fast and small. • Hacker News Board

Re: It's like JSON. but fast and small.

#41

Post by lazyjones » Sat, Dec 29, 2012, 2:34 PM UTC

The best choice always depends on your data and the quality of the implementations for the language you use. We tested many serializers recently for our backend (Perl, but we wanted something more open to other languages than Storable) and found msgpack slightly faster and 20-25% smaller than JSON at encoding, but only half as fast at decoding, so we chose JSON.

The most surprising result was that JSON was 2x (decoding) to 3x (encoding) faster than Storable. The downside is that it sets the utf-8 flag ...

Re: It's like JSON. but fast and small.

#42

Post by creationix » Sat, Dec 29, 2012, 2:36 PM UTC

pooriaazimi wrote: post #15

It looks cool, but be very careful about using it (in a browser, at least). Browsers have super-fast JSON parsers and parsing msgpack is much, much slower than parsing good old JSON. This great comment (from 201 days ago) is exactly about this issue: http://news.ycombinator.com/item?id=4091051 I quote a bit of it here: JSON size: 386kb MsgPack size: 332kb Difference: -14% Encoding with JSON.stringify 4 ms Encoding wi…

That benchmark you referenced has nothing but strings for it's data. That's not msgpack's strong point. Do another benchmark with nothing but small integers and arrays and you'll find msgpack kicks json's tail in both cpu usage and bandwidth, even when comparing the native JSON.parse with my pure javascript msgpack implementation using typed arrays.

Re: It's like JSON. but fast and small.

#43

Post by creationix » Sat, Dec 29, 2012, 2:37 PM UTC

benatkin wrote: post #37

dtf wrote: post #32

Earlier quoted context omitted.

The JavaScript msgpack codec is a couple of years old now, by the looks of the github repository. I see a custom base64 function, and no use of typed arrays. I wonder if these numbers might change a little given more modern features like typed arrays and fetching arraybuffers via XHR2. Biggest issue for me with JSON is needing to base64 any binary data, so msgpack could actually be quite useful.

I noticed that opportunity too. If anyone is looking for a fun project... The next thing would be to have practices and/or libraries that switch techniques based on browser support.

Is was a fun project actually. https://github.com/creationix/msgpack-js-browser

Re: It's like JSON. but fast and small.

#44

Post by laurent123456 » Sat, Dec 29, 2012, 2:46 PM UTC

So big news, binary encoded data is more compact than human-readable strings. I'm sure this new format is useful but bringing in the JSON comparison is completely irrelevant (and probably done only to get some page-hits). AMF is also a binary format, has been around for years, and is probably more compact than JSON too.

I think binary formats are great for online games where performance is critical, however for most applications human-readable strings are a lot easier to manage. In any case, both formats have different use cases.

Re: It's like JSON. but fast and small.

#45

Post by ville » Sat, Dec 29, 2012, 2:48 PM UTC

zurn wrote: post #33

pooriaazimi wrote: post #15

It looks cool, but be very careful about using it (in a browser, at least). Browsers have super-fast JSON parsers and parsing msgpack is much, much slower than parsing good old JSON. This great comment (from 201 days ago) is exactly about this issue: http://news.ycombinator.com/item?id=4091051 I quote a bit of it here: JSON size: 386kb MsgPack size: 332kb Difference: -14% Encoding with JSON.stringify 4 ms Encoding wi…

A lot of these bit shaving goose chases would be short circuited if browsers exposed zlib !

What would be the advantage of exposing it instead of compressing the data transparently in the background. (Like it is done already.)

Re: It's like JSON. but fast and small.

#46

Post by creationix » Sat, Dec 29, 2012, 3:02 PM UTC

laurent123456 wrote: post #44

So big news, binary encoded data is more compact than human-readable strings. I'm sure this new format is useful but bringing in the JSON comparison is completely irrelevant (and probably done only to get some page-hits). AMF is also a binary format, has been around for years, and is probably more compact than JSON too. I think binary formats are great for online games where performance is critical, however for most…

It's not a new format, I have no idea why it's popular news today. Having worked extensively with both msgpack and JSON in a javascript environment, I can tell you, its the closest to JSON of all the binary formats. The difference with JSON is msgpack's strings are binary safe (you can have a png as your value) and the format is a bit more compact, especially around integers.

JSON has: numbers, strings, booleans, null, arrays, objects. Msgpack has: numbers, raws, booleans, nil, arrays, maps.

So I guess msgpack is a superset of JSON. The raws can contain utf8 encoded strings like JSON mandates, or they can contain other things. There is no technical reason that the keys of the maps have to be strings. You could take a lua table that has another table as key and encode that in msgpack just fine.

In practice, I wanted more out of msgpack, so I extended the format using some of the reserved byte ranges to add in an "undefined" type and a distinction between utf8 encoded string and raw binary buffer.

For me this new format has been extremely useful as a general data serialization between processes (node to node, server to browser, etc..) I usually use it over binary websockets or raw tcp sockets.

Re: It's like JSON. but fast and small.

#47

Post by johnbellone » Sat, Dec 29, 2012, 3:07 PM UTC

gilgoomesh wrote: post #10

This is just an implementation of a binary encoded Property Tree. And not the best implementation out there since it lacks the redundant key/value elimination that more compact serialization formats use. Consider an array of maps where each of the maps have the same keys. By eliminating the redundant keys (have each subsequent occurrence simply reference earlier occurrences) you can halve the encoded data size relati…

Boost has a decent implementation of just a property tree which given a little magic can pretty much serialize into any key-value format. I used it pretty successfully for commandline and INI values for a simple game engine.

Re: It's like JSON. but fast and small.

#48

Post by johnbellone » Sat, Dec 29, 2012, 3:11 PM UTC

kzk_mover wrote: post #29

For those who are interested in Pinterest usecase (Memcached + MessagePack), please access to this url too. > http://engineering.pinterest.com/posts/2012/memcache-games/

This actually seems like a great use case, going to read later, but I immediately thought of MongoDB BSON format.

Re: It's like JSON. but fast and small.

#49

Post by ville » Sat, Dec 29, 2012, 3:17 PM UTC

ArchD wrote: post #11

stock_toaster wrote: post #4

json is kind of a lingua franca these days. For some kinds of things though, I do like tnetstring (has some nice qualities). msgpack (and bson too) always seemed a bit odd to me though. If you need binary packing, why not use protobufs or thrift?

Within the space of {schema-ful, schemaless} x {binary, text}, protobuf, {BSON, MessagePack}, JSON each occupy a distinct position. The position (schema-ful, text) is not very meaningful combination in practice and not covered by these, whereas the position (schema-less, binary) is a valid practical use case supported by {BSON, MessagePack}. For example, you don't know the schema before-hand but still want to minimiz…

This looks like a useful way to compare. Waiting for someone who has compared them to draw this plot :)

Re: It's like JSON. but fast and small.

#50

Post by millerm » Sat, Dec 29, 2012, 3:19 PM UTC

benatkin wrote: post #36

judofyr wrote: post #20

Earlier quoted context omitted.

sigh . Why do we always want to declare a winner? Anyone who has tried to store binary data in JSON knows that MessagePack does indeed have a use case. And anyone who is working with browsers knows that JSON/JS is the king there. > MessagePack is not smaller than gzip'd JSON But gzip'd MessagePack is smaller than gzip'd JSON, so? > MessagePack is not faster than JSON when a browser is involved Using it in the browser…

> Using it in the browser was never the use case for MessagePack. Hopefully it will be soon, with degradation for old browsers. I'd like to be able to load an avatar as binary png data with the rest of the data.

Assuming you are using a browser (other clients support this too)... For now, can't you just encode the image in your json using base 64 and use the 'data' uri scheme to decode it? Sure the encoded data can't be more than 32k in most browsers, but that is huge avatar.