Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlatBuffers: a memory efficient serialization library

google-opensource.blogspot.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: FlatBuffers: a memory efficient serialization library

#31

Post by kentonv »

mortarion wrote: post #28
kentonv wrote: post #7

Earlier quoted context omitted.

Hmm, my personal (biased) opinion is that using variable offsets and vtables (as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed offsets. "Optional" fields (ones that don't take space on the wire at all) are not clearly much of an advantage. The usual use of optional fields in Protobufs was to emulate unions, which Protobufs itself never supported directly, but Cap'n Proto does. Also, when wire si…

I think you have maybe misunderstood the purpose of why flatbuffers exists; message packing for real time high performance games. It's not a replacement for Protocol Buffers or Cap'n Proto. You've heard of Valve Software right? Their game engine, being a distant relative of Quake, used the old quake message packing format up until 1 or 2 years ago. Since then they have retrofitted all their popular games to use proto…

I think there's some confusion here. To be clear, Cap'n Proto allows you to add new fields to a message without breaking backwards compatibility, and Cap'n Proto implements default values.

The question is how a field is represented on the wire if it's defined in the schema but you don't assign it a value before serializing. With FlatBuffers, it won't take any space on the wire (although it still takes space in the vtable, but that can be amortized with many instances of the same structure). With Cap'n Proto, all fields are located at fixed offsets from the start of the struct, therefore they take space even if they aren't set. You can truncate unused fields off the end, but not from the middle.

Also, note that Cap'n Proto uses pointers for variable-width fields. The pointer can be null if the field is not set. So the maximum wasted space for an unset field is 8 bytes, all of which are zero and therefore compress nicely.

Cap'n Proto's approach is exactly the approach used by in-memory data structures in C, C++, and all other high-performance programming languages. If your C struct contains a field you don't use, it still takes space in memory. I would think this would be preferable for games.

Re: FlatBuffers: a memory efficient serialization library

#32

Post by kentonv »

bastih wrote: post #30
vertex-four wrote: post #4

Earlier quoted context omitted.

> My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. No. It's a serialisation format that's specifically designed so that data structures in this format can be used as-is by programs, without unnecessarily copying data into or out of the format to use it. The RPC layer is then built on top of that, but the forma…

I always felt that the RPC should live in an extra project to not distract from the format part.

They are compiled as separate libraries, so you can use the serialization layer without linking in the RPC layer.

But if anything it was the serialization that distracted me from the RPC. Cap'n Proto was originally (briefly) a capability-based RPC system on top of Protobufs. :)

Re: FlatBuffers: a memory efficient serialization library

#33

Post by kentonv »

emn13 wrote: post #29
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

It's always surprised me that zero-copy serialization is worth it at all. Given how much slower memory is than CPU, once you've taken the effort to stream in data from somewhere, why not transform it into a more convenient format, especially if that means your on-the-wire format might be smaller. Is the key here that the input gets copied into memory directly without CPU intervention?

Well, first of all, there is mmap(), which does indeed make data available in memory without streaming it through the CPU first. If you have a 10GB file and really just want to read one data point out of the middle, with Cap'n Proto (and FlatBuffers) it's perfectly reasonable to mmap() the whole thing and traverse to that one item.

Meanwhile, in networking, there is RDMA, which indeed allows memory to be transferred over the network without involving the CPU on either end.

But even ignoring those things, memory bandwidth is actually a big reason why you should not want to transform things upfront. You see, if you have an upfront transformation, you're streaming the data into the CPU, then back out in the new format, and then reading it back in again later on when you actually use it. If your data is small enough, perhaps it all stays in cache and this isn't an issue. But if it's large, then you're doubling or tripling your memory bandwidth usage by having a decode step.

In any case, the fact is that there are lots of real servers out there that are CPU bound and spend double-digit percentages of their time encoding and decoding Protobufs.

Re: FlatBuffers: a memory efficient serialization library

#34

Post by swah »

haberman wrote: post #14
kentonv wrote: post #13

Earlier quoted context omitted.

> This has been implemented internally; the next open-source release should have it. Oh cool! Presumably based on the patch I wrote before I left? :)

Seems likely, though I couldn't say for sure. I am sure you will recognize a lot of the design. :)

Did you have a say on the choice of language? I ask because I remember those posts http://blog.reverberate.org/2009/12/torn-over-c-question.htm...

Re: FlatBuffers: a memory efficient serialization library

#35

Post by anon4 »

kibibu wrote: post #20
kentonv wrote: post #9

Earlier quoted context omitted.

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

> Which is an OK trade-off for certain situations (like reading your game data from disk). Although in some cases this is how your fancy-pants console security gets busted. See: http://www.wiibrew.org/wiki/Twilight_Hack (I realize you said "game data" and not "savegame data", but I think safe file reading is important pretty much everywhere)

It can be argued that arbitrary code execution capability in games on a locked platform are a feature rather than a bug :)

Re: FlatBuffers: a memory efficient serialization library

#36

Post by mantraxB »

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages).

The other problem is that Google tends to release random stuff as open source and then modify it (breaking BC) or abandon it without regard for the people who are using it.

So caveat emptor, I guess.

Re: FlatBuffers: a memory efficient serialization library

#37

Post by judk »

quotemstr wrote: post #19
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

> Cap'n Proto's solution to this was just to build unions into the language, something Protobufs should have done a long time ago. Unions? Like the ones RPC has had since time immemorial [1]? Funny how everything old is new again. [1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

Not new, so much as never bothered to be implemented by the controlling cabal.

Re: FlatBuffers: a memory efficient serialization library

#38

Post by judk »

kentonv wrote: post #31
mortarion wrote: post #28

Earlier quoted context omitted.

I think you have maybe misunderstood the purpose of why flatbuffers exists; message packing for real time high performance games. It's not a replacement for Protocol Buffers or Cap'n Proto. You've heard of Valve Software right? Their game engine, being a distant relative of Quake, used the old quake message packing format up until 1 or 2 years ago. Since then they have retrofitted all their popular games to use proto…

I think there's some confusion here. To be clear, Cap'n Proto allows you to add new fields to a message without breaking backwards compatibility, and Cap'n Proto implements default values. The question is how a field is represented on the wire if it's defined in the schema but you don't assign it a value before serializing. With FlatBuffers, it won't take any space on the wire (although it still takes space in the vt…

IME, optional fields are used for...optional fields. Like when you have a 50-field data object but only want to transmit 5 of those fields.

Re: FlatBuffers: a memory efficient serialization library

#39

Post by kentonv »

mantraxB wrote: post #36

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages…

Java has ByteBuffer, Javascript has TypedArrays, Python has the "struct" module, and other languages have other fine ways of reading raw data in this kind of format. Some languages may lack the ability to inline calls well enough for true zero-copy to be efficient, but the worst case is you fall back to parsing the message upfront like you would with any other encoding. That upfront parse is still going to be faster, because it's just easier to read fixed-width values than variable-width.

Re: FlatBuffers: a memory efficient serialization library

#40

Post by Aardappel »

kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Hi. I designed most of FlatBuffers, so let me see if I can clarify:

Clearly, FlatBuffers seeks a different design tradeoff than Cap'n Proto. We wanted to retain all the flexibility of Protobufs, while having the advantages of the "zero parsing / zero allocation" approach. That brought us to the vtable design.

In use cases where performance matters, i.e. you are churning through a large amount of data, you will be accessing the same vtable repeatedly, and the cost of the extra indirection (and the code associated with it) will be masked by the memory latency of the main data you're going through. It will be (close to) "for free". Even in cases where that isn't the case, whether or not it is able to match Cap'n Proto's accessor code is less interesting than it being endlessly faster than Protobuf and most other serializers out there, given that it is just as flexible.

vtables are de-duped on the fly during construction. It is typically not expensive.

Note this tiny overhead holds for "tables", we also have naked "structs", which do not have the indirection overhead, but have no forwards/backwards compatability. Useful for things like 2d/3d vector types, colors, and other small POD types that are used a lot.

FlatBuffers also has unions, which you should use instead of optionals when appropriate. We feel optionals have a lot of uses beyond just mere unions and forwards/backwards compatibility, however. Game objects can have a LOT of fields, many of which are often at their default value, and thus not stored on the wire. This gives significant compression. The zero-byte compression in Cap'n Proto is cool, but we prefer to not have to use additional buffers when reading. Optionals also give a lot of design freedom, i.e. you can add a field that you know is only needed for very few instances without fear of bloating your binaries, as an alternative to "subclassing", or indeed unions.

On field declaration order: to each his own, I guess. That said, the omission of explicit field id's is something that's handled at the level of the schema compiler, we could easily allow optional id declarations in the schema for people who want the flexibility of declaring things in an arbitrary order. Noted.

We should clean up the benchmark code, so it can be included, yes. It is not running a noop loop, we made sure of that. I haven't verified what makes Protobuf traversal that much slower, I am guessing it's causing a memory allocation somewhere.

Post reply on HN