Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlatBuffers: a memory efficient serialization library

google-opensource.blogspot.com

Post reply on HN
Print view View original

61–65 of 65 posts

Re: FlatBuffers: a memory efficient serialization library

#61

Post by Aardappel »

kentonv wrote: post #39
mantraxB wrote: post #36

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages…

Java has ByteBuffer, Javascript has TypedArrays, Python has the "struct" module, and other languages have other fine ways of reading raw data in this kind of format. Some languages may lack the ability to inline calls well enough for true zero-copy to be efficient, but the worst case is you fall back to parsing the message upfront like you would with any other encoding. That upfront parse is still going to be faster,…

And FlatBuffers actually uses ByteBuffer as part of its Java implementation.

Re: FlatBuffers: a memory efficient serialization library

#62

Post by kentonv »

Aardappel wrote: post #46
kentonv wrote: post #9

Earlier quoted context omitted.

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

Most readers of binary file formats can be made to read memory outside the buffer by corrupting the data, and FlatBuffers is no different. That said, an option to bounds-check every offset would be possible, at a certain cost. Might be a nice optional feature to have.

> Most readers of binary file formats can be made to read memory outside the buffer by corrupting the data

I'm pretty sure that would be considered a serious security bug for any format likely to be displayed in a web browser. For instance, an image format with such a bug would allow you to implement a heartbleed-like attack on a user's browser by displaying a malicious image and then reading back the pixel values. That would be very, very bad.

But I can believe that your statement applies to formats used by games for their own assets, where those assets come directly from the game developer.

Re: FlatBuffers: a memory efficient serialization library

#63

Post by haberman »

swah wrote: post #58
haberman wrote: post #53

Earlier quoted context omitted.

I'm not involved in FlatBuffers -- I said I hadn't heard of it before yesterday. :) My comment about unions was about the Google protobuf implementation, which I am tangentially involved in.

Please apologize my misunderstandment.

No problem, I am flattered that you would have remembered a blog article I posted five years ago.

Re: FlatBuffers: a memory efficient serialization library

#64

Post by ahminus »

ssylvan wrote: post #55
ahminus wrote: post #50

Earlier quoted context omitted.

I think if you have a 50 field data object, and you have cases where you transmit only 5 of those fields, the problem is your data modelling, not the serialization library.

I think you haven't worked on games. 99% of the fields in game objects are the defaults, with some minor customization (e.g. position).

Games are all I've worked on for over a decade. See my other comments.

Re: FlatBuffers: a memory efficient serialization library

#65

Post by kentonv »

limsup wrote: post #18
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Once you've had some time to look this over, I think we'd all appreciate a blog post that explains the differences between cap'n proto and flatbuffers

Incase you didn't already see it at the top of HN, here's your blog post: https://kentonv.github.io/capnproto/news/2014-06-17-capnprot...

:)

Post reply on HN