Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlatBuffers: a memory efficient serialization library

google-opensource.blogspot.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: FlatBuffers: a memory efficient serialization library

#11

Post by haberman »

kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

FWIW, I had never heard of this before today, and I make a point of keeping tabs on these kinds of things.

> Cap'n Proto's solution to this was just to build unions into the language, something Protobufs should have done a long time ago.

This has been implemented internally; the next open-source release should have it.

Re: FlatBuffers: a memory efficient serialization library

#12

Post by kentonv »

jmillikin wrote: post #10
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

I haven't used FlatBuffers, but based on the description it sounds like a {competitor to,clone of} the SBE format used for low-latency transmission of financial data: http://mechanical-sympathy.blogspot.com/2014/05/simple-binar... Many of SBE's unusual properties (such as permitting new fields only at the end of a structure) are designed to maximize performance during streaming decoding.

Hmm. The vtable approach used by FlatBuffer seems harmful for streaming. I don't think the SBE people would have used such a design.

> permitting new fields only at the end of a structure

Be careful not to conflate the order of fields as written in the schema with the order of fields on the wire. Cap'n Proto (like SBE) more or less requires new fields to go on the end on the wire, but lets you put them in any order in the schema. FlatBuffer apparently requires them to go on the end in the schema but lets you put them in any order on the wire.

(I say "more or less" because Cap'n Proto can actually stick a new field into what was previously padding space between two existing fields. But usually new fields go on the end.)

Re: FlatBuffers: a memory efficient serialization library

#13

Post by kentonv »

haberman wrote: post #11
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

FWIW, I had never heard of this before today, and I make a point of keeping tabs on these kinds of things. > Cap'n Proto's solution to this was just to build unions into the language, something Protobufs should have done a long time ago. This has been implemented internally; the next open-source release should have it.

> This has been implemented internally; the next open-source release should have it.

Oh cool! Presumably based on the patch I wrote before I left? :)

Re: FlatBuffers: a memory efficient serialization library

#14

Post by haberman »

kentonv wrote: post #13
haberman wrote: post #11

Earlier quoted context omitted.

FWIW, I had never heard of this before today, and I make a point of keeping tabs on these kinds of things. > Cap'n Proto's solution to this was just to build unions into the language, something Protobufs should have done a long time ago. This has been implemented internally; the next open-source release should have it.

> This has been implemented internally; the next open-source release should have it. Oh cool! Presumably based on the patch I wrote before I left? :)

Seems likely, though I couldn't say for sure. I am sure you will recognize a lot of the design. :)

Re: FlatBuffers: a memory efficient serialization library

#15

Post by jmillikin »

kentonv wrote: post #12
jmillikin wrote: post #10

Earlier quoted context omitted.

I haven't used FlatBuffers, but based on the description it sounds like a {competitor to,clone of} the SBE format used for low-latency transmission of financial data: http://mechanical-sympathy.blogspot.com/2014/05/simple-binar... Many of SBE's unusual properties (such as permitting new fields only at the end of a structure) are designed to maximize performance during streaming decoding.

Hmm. The vtable approach used by FlatBuffer seems harmful for streaming. I don't think the SBE people would have used such a design. > permitting new fields only at the end of a structure Be careful not to conflate the order of fields as written in the schema with the order of fields on the wire. Cap'n Proto (like SBE) more or less requires new fields to go on the end on the wire , but lets you put them in any order…

  > The vtable approach used by FlatBuffer seems harmful for
  > streaming. I don't think the SBE people would have used
  > such a design.
If the vtable and fixed-size fields are placed together with variable-length fields (e.g. byte blobs) at the end, then decoding can make better use of the processor cache than if the fields are all mixed together.

Re: FlatBuffers: a memory efficient serialization library

#17

Post by kentonv »

jmillikin wrote: post #15
kentonv wrote: post #12

Earlier quoted context omitted.

Hmm. The vtable approach used by FlatBuffer seems harmful for streaming. I don't think the SBE people would have used such a design. > permitting new fields only at the end of a structure Be careful not to conflate the order of fields as written in the schema with the order of fields on the wire. Cap'n Proto (like SBE) more or less requires new fields to go on the end on the wire , but lets you put them in any order…

> The vtable approach used by FlatBuffer seems harmful for > streaming. I don't think the SBE people would have used > such a design. If the vtable and fixed-size fields are placed together with variable-length fields (e.g. byte blobs) at the end, then decoding can make better use of the processor cache than if the fields are all mixed together.

But the idea with the vtable is that many instances of the same "table" (record) would share a vtable. So the vtable would presumably never appear inside the table instance, I think.

Re: FlatBuffers: a memory efficient serialization library

#18

Post by limsup »

kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Once you've had some time to look this over, I think we'd all appreciate a blog post that explains the differences between cap'n proto and flatbuffers

Re: FlatBuffers: a memory efficient serialization library

#19

Post by quotemstr »

kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

> Cap'n Proto's solution to this was just to build unions into the language, something Protobufs should have done a long time ago.

Unions? Like the ones RPC has had since time immemorial [1]? Funny how everything old is new again.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

Re: FlatBuffers: a memory efficient serialization library

#20

Post by kibibu »

kentonv wrote: post #9
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

> Which is an OK trade-off for certain situations (like reading your game data from disk).

Although in some cases this is how your fancy-pants console security gets busted. See: http://www.wiibrew.org/wiki/Twilight_Hack

(I realize you said "game data" and not "savegame data", but I think safe file reading is important pretty much everywhere)

Post reply on HN