Live data from Hacker News

FlatBuffers: a memory efficient serialization library

google-opensource.blogspot.com

21–30 of 65 posts

Re: FlatBuffers: a memory efficient serialization library

#22
post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Incidentally, the first time I saw the Cap'n Proto homepage I thought the whole thing was a joke because I saw the "cerealization protocol" tagline and "infinitely faster" badge and stopped reading after the first paragraph...

EDIT: Oh yeah, and when I skimmed the rest of the page this "confirmed" my suspicions: "Time-traveling RPC: Cap’n Proto features an RPC system implements time travel such that call results are returned to the client before the request even arrives at the server!". I'm familiar with promise pipelining but that sentence (and diagram) made it sound like a complete joke.

Re: FlatBuffers: a memory efficient serialization library

#23
post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Incidentally, the first time I saw the Cap'n Proto homepage I thought the whole thing was a joke because I saw the "cerealization protocol" tagline and "infinitely faster" badge and stopped reading after the first paragraph... EDIT: Oh yeah, and when I skimmed the rest of the page this "confirmed" my suspicions: "Time-traveling RPC: Cap’n Proto features an RPC system implements time travel such that call results are…

FWIW it had the opposite effect on me: it sounded like this wasn't some run of the mill corporate middleware project with bland and enterprisey web pages, but made by someone who actually puts their personality into the game.

Go has a similar vibe although more subdued, with the funny mascot and opinionated culture.

Re: FlatBuffers: a memory efficient serialization library

#24
post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Incidentally, the first time I saw the Cap'n Proto homepage I thought the whole thing was a joke because I saw the "cerealization protocol" tagline and "infinitely faster" badge and stopped reading after the first paragraph... EDIT: Oh yeah, and when I skimmed the rest of the page this "confirmed" my suspicions: "Time-traveling RPC: Cap’n Proto features an RPC system implements time travel such that call results are…

I also released it on April 1st, 2013.

Re: FlatBuffers: a memory efficient serialization library

#25
post #9
post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

Would be a bad tradeoff even for trusted on-disk data, since disks are so glacially slow and serialization systems always run into bad data from time to time (due to fs data corruption, io errors, memory corrupting bugs, etc).

Re: FlatBuffers: a memory efficient serialization library

#26
post #18
post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Once you've had some time to look this over, I think we'd all appreciate a blog post that explains the differences between cap'n proto and flatbuffers

I think I might write one. I feel a bit uncomfortable since I'm obviously biased... But perhaps it would be useful even so.

Re: FlatBuffers: a memory efficient serialization library

#28
post #7
post #5

Earlier quoted context omitted.

Deeper in the site they have a benchmarks page[1] that has a short "why not Cap'n Proto": "Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does, though with a more complicated binary encoding and less flexibility (no optional fields to allow deprecating fields or serializing with missing fields for which defaults exist). It currently also isn't fully cross-platform portable (lack of VS support).…

Hmm, my personal (biased) opinion is that using variable offsets and vtables (as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed offsets. "Optional" fields (ones that don't take space on the wire at all) are not clearly much of an advantage. The usual use of optional fields in Protobufs was to emulate unions, which Protobufs itself never supported directly, but Cap'n Proto does. Also, when wire si…

I think you have maybe misunderstood the purpose of why flatbuffers exists; message packing for real time high performance games. It's not a replacement for Protocol Buffers or Cap'n Proto.

You've heard of Valve Software right? Their game engine, being a distant relative of Quake, used the old quake message packing format up until 1 or 2 years ago. Since then they have retrofitted all their popular games to use protocol buffers instead since it allows them to more easily extend their net code without breaking backwards compatibility (up to a point of course). In the context of games, optional fields are very important as are default values.

Also in this context; the author of flatbuffers is Wouter van Oortmerssen, author of the Cube and Cube2 game engines: http://sauerbraten.org

From the white paper:

> In particular, FlatBuffers focus is on mobile hardware (where memory size and memory bandwidth is even more constrained than on desktop hardware), and applications that have the highest performance needs: games.

Re: FlatBuffers: a memory efficient serialization library

#29
post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

It's always surprised me that zero-copy serialization is worth it at all. Given how much slower memory is than CPU, once you've taken the effort to stream in data from somewhere, why not transform it into a more convenient format, especially if that means your on-the-wire format might be smaller.

Is the key here that the input gets copied into memory directly without CPU intervention?

Re: FlatBuffers: a memory efficient serialization library

#30
post #3

Earlier quoted context omitted.

I dont know full details of Cap'n Proto, so please correct me if I am wrong. My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. Would that be a reasonable summary of Cap'n Proto ? Guessing from the name I expected FlatBuffers to be a unified file format for on-disk and on-RAM presence, so that you can just mmap…

> My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. No. It's a serialisation format that's specifically designed so that data structures in this format can be used as-is by programs, without unnecessarily copying data into or out of the format to use it. The RPC layer is then built on top of that, but the forma…

I always felt that the RPC should live in an extra project to not distract from the format part.
Post reply on HN