Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My thoughts on MessagePack

gist.github.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: My thoughts on MessagePack

#11

Post by alexgartrell »

frsyuki wrote: post #8
ynniv wrote: post #3

cheald has an excellent comment regarding MessagePack, JSON, and Protocol Buffers in the post from 15 hours ago: http://news.ycombinator.com/item?id=4091051 MessagePack is not smaller than gzip'd JSON MessagePack is not faster than JSON when a browser is involved Other comments from that post include: MessagePack is not human readable MessagePack has issues with UTF-8 Small packets also have significant TCP/IP overhe…

Although the original blog post focuses on JavaScript and browsers, MessagePack itself doesn't mainly focus on them. A major use case of MessagePack is to store serialized objects in memcached. A blog post written by Pinterest describes this use case ( http://engineering.pinterest.com/posts/2012/memcache-games/ ). They use MessagePack with Python which is faster than one with JavaScript. They could store more objects…

> They could store more objects in a server without performance declination (e.g. gzip).

The performance declination argument is bullshit. Network's a million [0] times slower than gzip.

Truth be told, once you're on the network, you're already screwed w.r.t. most serialization. The only thing efficient compression/decompression is going to buy you is lower CPU (memcached servers run at like 2% CPU util, even under heavy load [1]).

Memcache at Facebook actually uses the ascii protocol, and the memcached implementation is a braindead strtok parser (some of our other stuff uses ragel -- you'll have a hell of a time out optimizing ragel with the right compiler flags -- I've tried and failed).

Just use whichever serialization format has the best API, because I can say with near certainty that it's not going to be a perf problem for you if you're touching disk, network, etc.

[0] Obviously a made up number, but it's way slower. Especially if you're unlucky and lose a packet or something.

[1] With the exception of weird kernel spin lock contention issues, which can happen if you're not sharding your UDP packets well and trying to reply from 8+ cores on 1 UDP socket. You probably aren't.

Re: My thoughts on MessagePack

#12

Post by SeoxyS »

I've personally been using BSON[1] as a binary alternative to JSON, and it's worked out great. I've written an Objective-C wrapper around the C lib, in case anybody's interested. Every other language has a solid implementation from 10gen (the MongoDB guys). It's a solid format with a clear spec that's extensible and fully supports UTF-8.

[1]: http://bsonspec.org/

Re: My thoughts on MessagePack

#13

Post by SeoxyS »

latch wrote: post #7
ynniv wrote: post #3

cheald has an excellent comment regarding MessagePack, JSON, and Protocol Buffers in the post from 15 hours ago: http://news.ycombinator.com/item?id=4091051 MessagePack is not smaller than gzip'd JSON MessagePack is not faster than JSON when a browser is involved Other comments from that post include: MessagePack is not human readable MessagePack has issues with UTF-8 Small packets also have significant TCP/IP overhe…

I've used it to store blobs of data in databases, specifically when space tends to directly equate to memory (like redis and mongodb). You can take this pretty far and apply different serialization or compression algorithms based on the data (and store a field that says which approach was used for when you deserialize it). Using this from the browser is not the first thing that came to my mind.

Storing a message-pack serialized object in MongoDB is silly, because Mongo is essentially storying BSON-serialized object on disk. You're double-serializing data in two competing formats.

Re: My thoughts on MessagePack

#14

Post by jamesaguilar »

frsyuki wrote: post #8
ynniv wrote: post #3

cheald has an excellent comment regarding MessagePack, JSON, and Protocol Buffers in the post from 15 hours ago: http://news.ycombinator.com/item?id=4091051 MessagePack is not smaller than gzip'd JSON MessagePack is not faster than JSON when a browser is involved Other comments from that post include: MessagePack is not human readable MessagePack has issues with UTF-8 Small packets also have significant TCP/IP overhe…

Although the original blog post focuses on JavaScript and browsers, MessagePack itself doesn't mainly focus on them. A major use case of MessagePack is to store serialized objects in memcached. A blog post written by Pinterest describes this use case ( http://engineering.pinterest.com/posts/2012/memcache-games/ ). They use MessagePack with Python which is faster than one with JavaScript. They could store more objects…

That test you linked to which claims that messagepack is 4x faster seems to rely on the serialized text staying in-process. The vrefbuffer is only zero-copy as long as you don't need to send it to any API which reads strings or char buffers (e.g. any RPC or network-oriented mechanism). Am I reading it right?

Re: My thoughts on MessagePack

#15

Post by rkalla »

frsyuki wrote: post #10

We're using MessagePack in a Rails application to log user behaviors and analyze them. Compared with other serialization libraries such as Protocol Buffers, Avro or BSON, one of the advantages of MessagePack is compatibility with JSON. (In spite of its name, BSON has special types which cause incompatibility with JSON) It means we can exchange objects sent from browsers (in JSON format) between servers written in dif…

If JSON compatibility is an issue, have you looked at UBJSON? http://ubjson.org/

May be a bit bigger than msgpack but is damn-near human readable even in its binary format and really easy to encode/decode. Also 1:1 compatibility with JSON.

Compatibility and simplicity were the core design tenantes. It may not be the right choice, just throwing it out there incase it helps.

Disclaimer: I am the author of the spec.

Re: My thoughts on MessagePack

#16

Post by samsoffes »

I think it's hilarious that 2 of the 3 comments on the gist are about how he formatted his Markdown. sigh

I think MessagePack is great. Good for him for trying to make something better. He openly admits it's not better all the time. Why not help?

Re: My thoughts on MessagePack

#17

Post by latch »

SeoxyS wrote: post #13
latch wrote: post #7

Earlier quoted context omitted.

I've used it to store blobs of data in databases, specifically when space tends to directly equate to memory (like redis and mongodb). You can take this pretty far and apply different serialization or compression algorithms based on the data (and store a field that says which approach was used for when you deserialize it). Using this from the browser is not the first thing that came to my mind.

Storing a message-pack serialized object in MongoDB is silly, because Mongo is essentially storying BSON-serialized object on disk. You're double-serializing data in two competing formats.

BSON isn't meant to be compact, it's meant to be quick and efficient to serialize and deserialize. You store it inside MongoDB as bindata, and you'll save space.

Re: My thoughts on MessagePack

#18

Post by kiyoto »

rkalla wrote: post #15
frsyuki wrote: post #10

We're using MessagePack in a Rails application to log user behaviors and analyze them. Compared with other serialization libraries such as Protocol Buffers, Avro or BSON, one of the advantages of MessagePack is compatibility with JSON. (In spite of its name, BSON has special types which cause incompatibility with JSON) It means we can exchange objects sent from browsers (in JSON format) between servers written in dif…

If JSON compatibility is an issue, have you looked at UBJSON? http://ubjson.org/ May be a bit bigger than msgpack but is damn-near human readable even in its binary format and really easy to encode/decode. Also 1:1 compatibility with JSON. Compatibility and simplicity were the core design tenantes. It may not be the right choice, just throwing it out there incase it helps. Disclaimer: I am the author of the spec.

Sounds cool. I would take a look. I think what this space (serializers) needs is objective/holistic evaluations of pros and cons of different approaches. (disclaimer: I am involved with MessagePack, although not a committer of any of its drivers).

Re: My thoughts on MessagePack

#19

Post by stock_toaster »

SeoxyS wrote: post #12

I've personally been using BSON[1] as a binary alternative to JSON, and it's worked out great. I've written an Objective-C wrapper around the C lib, in case anybody's interested. Every other language has a solid implementation from 10gen (the MongoDB guys). It's a solid format with a clear spec that's extensible and fully supports UTF-8. [1]: http://bsonspec.org/

I am kind of partial to tnetstring passed through lzf right now.

Re: My thoughts on MessagePack

#20

Post by StavrosK »

twelvechairs wrote: post #4
keithseahus wrote: post #2

The best way of serialization at present. I like the thought, performance and various implementations.

As nice as Message Pack might be, 'The best way of serialization' I'm not sure is a very helpful statement. It can't be 'the best' because 'the best' depends on the specifics of what you are doing. As noted in the OP: "...its pros and cons should be carefully considered, and there are many situations where it simply does not offer enough advantage...". I, for instance, am still using the much-less-cool yaml, because…

Isn't JSON a subset of YAML? So isn't it, quite literally, that YAML > JSON?
Post reply on HN