Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlatBuffers: a memory efficient serialization library

google-opensource.blogspot.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: FlatBuffers: a memory efficient serialization library

#51

Post by dragonwriter »

kentonv wrote: post #9
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

> OK, AFAICT there is no bounds checking. When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess.

I think the key use case for FlatBuffers is mostly for very-high-performance communication between a set of processes that you control to scale out high-performance systems into distributed systems while keeping the communication overhead minimal, not for, e.g., communicating between untrusted machines over a public network. So, I don't see that as a huge problem in the key use case.

Re: FlatBuffers: a memory efficient serialization library

#52

Post by mantraxB »

kentonv wrote: post #39
mantraxB wrote: post #36

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages…

Java has ByteBuffer, Javascript has TypedArrays, Python has the "struct" module, and other languages have other fine ways of reading raw data in this kind of format. Some languages may lack the ability to inline calls well enough for true zero-copy to be efficient, but the worst case is you fall back to parsing the message upfront like you would with any other encoding. That upfront parse is still going to be faster,…

I don't get it, you can't read anything out of a ByteBuffer, except a bunch of bytes.

You can't map 1:1 any more complicated in-memory structure to that buffer, or am I wrong?

Re: FlatBuffers: a memory efficient serialization library

#53

Post by haberman »

swah wrote: post #44
haberman wrote: post #43

Earlier quoted context omitted.

I don't understand your question -- the choice of language for what? The blog article you linked is about my own protobuf implementation "upb", which is separate from anything else we've been talking about in this thread.

For this new project, FlatBuffers, since it appears you were involved in it.

I'm not involved in FlatBuffers -- I said I hadn't heard of it before yesterday. :)

My comment about unions was about the Google protobuf implementation, which I am tangentially involved in.

Re: FlatBuffers: a memory efficient serialization library

#54

Post by [deleted] »

Aardappel wrote: post #46
kentonv wrote: post #9

Earlier quoted context omitted.

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

Most readers of binary file formats can be made to read memory outside the buffer by corrupting the data, and FlatBuffers is no different. That said, an option to bounds-check every offset would be possible, at a certain cost. Might be a nice optional feature to have.

[deleted]

Re: FlatBuffers: a memory efficient serialization library

#55

Post by ssylvan »

ahminus wrote: post #50
judk wrote: post #38

Earlier quoted context omitted.

IME, optional fields are used for...optional fields. Like when you have a 50-field data object but only want to transmit 5 of those fields.

I think if you have a 50 field data object, and you have cases where you transmit only 5 of those fields, the problem is your data modelling, not the serialization library.

I think you haven't worked on games. 99% of the fields in game objects are the defaults, with some minor customization (e.g. position).

Re: FlatBuffers: a memory efficient serialization library

#56

Post by ssylvan »

kentonv wrote: post #7
cakoose wrote: post #5

Earlier quoted context omitted.

Deeper in the site they have a benchmarks page[1] that has a short "why not Cap'n Proto": "Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does, though with a more complicated binary encoding and less flexibility (no optional fields to allow deprecating fields or serializing with missing fields for which defaults exist). It currently also isn't fully cross-platform portable (lack of VS support).…

Hmm, my personal (biased) opinion is that using variable offsets and vtables (as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed offsets. "Optional" fields (ones that don't take space on the wire at all) are not clearly much of an advantage. The usual use of optional fields in Protobufs was to emulate unions, which Protobufs itself never supported directly, but Cap'n Proto does. Also, when wire si…

I think looking up values based on a simple offset is far simpler to all the clever rules and "parsing" that happens in capn' proto (some of which no doubt caused by the desire to support streaming "automatically", an anti-feature IMO, streaming should happen at a higher level to take advantage of domain knowledge).

I really don't see how you can even pretend that cap'n proto is less complicated. Yes there's an extra indirection at runtime, but the benefit is great simplicity, and quite possibly better performance when the vtable is hot in the cache.

Re: FlatBuffers: a memory efficient serialization library

#57

Post by kentonv »

mantraxB wrote: post #52
kentonv wrote: post #39

Earlier quoted context omitted.

Java has ByteBuffer, Javascript has TypedArrays, Python has the "struct" module, and other languages have other fine ways of reading raw data in this kind of format. Some languages may lack the ability to inline calls well enough for true zero-copy to be efficient, but the worst case is you fall back to parsing the message upfront like you would with any other encoding. That upfront parse is still going to be faster,…

I don't get it, you can't read anything out of a ByteBuffer, except a bunch of bytes. You can't map 1:1 any more complicated in-memory structure to that buffer, or am I wrong?

ByteBuffer has methods for reading primitive values of all sizes -- ints, floats, etc. -- from arbitrary offsets within the buffer. So you just need a wrapper object with nice generated methods for each field which in turn call the ByteBuffer getters.

None that the C++ code works this way too. We don't cleverly generate a struct that just happens to have the right layout; we generate a class that wraps a pointer and provides inline accessors that read/write values from the correct offset relative to that pointer. After inlining, it's just as fast.

Re: FlatBuffers: a memory efficient serialization library

#58

Post by swah »

haberman wrote: post #53
swah wrote: post #44

Earlier quoted context omitted.

For this new project, FlatBuffers, since it appears you were involved in it.

I'm not involved in FlatBuffers -- I said I hadn't heard of it before yesterday. :) My comment about unions was about the Google protobuf implementation, which I am tangentially involved in.

Please apologize my misunderstandment.

Re: FlatBuffers: a memory efficient serialization library

#60

Post by sitkack »

tlrobinson wrote: post #22
kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Incidentally, the first time I saw the Cap'n Proto homepage I thought the whole thing was a joke because I saw the "cerealization protocol" tagline and "infinitely faster" badge and stopped reading after the first paragraph... EDIT: Oh yeah, and when I skimmed the rest of the page this "confirmed" my suspicions: "Time-traveling RPC: Cap’n Proto features an RPC system implements time travel such that call results are…

If everyone always asks the same question, can you not respond correctly before they speak? If question B is always followed by question A, wouldn't it be prudent to answer both at the same time?
Post reply on HN