FlatBuffers: a memory efficient serialization library • Hacker News Board

Re: FlatBuffers: a memory efficient serialization library

#41

Post by gliptic » Tue, Jun 17, 2014, 2:52 PM UTC

mantraxB wrote: post #36

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages…

Rust does have unsafe direct memory access and a work-in-progress port of Cap'n Proto (https://github.com/dwrensha/capnproto-rust).

Re: FlatBuffers: a memory efficient serialization library

#42

Post by blt » Tue, Jun 17, 2014, 3:39 PM UTC

Aardappel wrote: post #40

kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

Hi. I designed most of FlatBuffers, so let me see if I can clarify: Clearly, FlatBuffers seeks a different design tradeoff than Cap'n Proto. We wanted to retain all the flexibility of Protobufs, while having the advantages of the "zero parsing / zero allocation" approach. That brought us to the vtable design. In use cases where performance matters, i.e. you are churning through a large amount of data, you will be acc…

Structs are a welcome improvement. Using Protobuf, I was forced to move from a 3d vector type to an array of doubles to get packed=true performance. Small loss of expressiveness, no big deal, but I can imagine it getting a lot worse for heterogeneous and/or more complex structs. Was a shock to see a dynamic allocation for every vector in my 1000-element array. (I understand why this is necessary given the spec of Protobuf though.)

Re: FlatBuffers: a memory efficient serialization library

#43

Post by haberman » Tue, Jun 17, 2014, 4:01 PM UTC

swah wrote: post #34

haberman wrote: post #14

Earlier quoted context omitted.

Seems likely, though I couldn't say for sure. I am sure you will recognize a lot of the design. :)

Did you have a say on the choice of language? I ask because I remember those posts http://blog.reverberate.org/2009/12/torn-over-c-question.htm...

I don't understand your question -- the choice of language for what?

The blog article you linked is about my own protobuf implementation "upb", which is separate from anything else we've been talking about in this thread.

Re: FlatBuffers: a memory efficient serialization library

#44

Post by swah » Tue, Jun 17, 2014, 4:16 PM UTC

haberman wrote: post #43

swah wrote: post #34

Earlier quoted context omitted.

Did you have a say on the choice of language? I ask because I remember those posts http://blog.reverberate.org/2009/12/torn-over-c-question.htm...

I don't understand your question -- the choice of language for what? The blog article you linked is about my own protobuf implementation "upb", which is separate from anything else we've been talking about in this thread.

For this new project, FlatBuffers, since it appears you were involved in it.

Re: FlatBuffers: a memory efficient serialization library

#45

Post by Aardappel » Tue, Jun 17, 2014, 4:27 PM UTC

blt wrote: post #42

Aardappel wrote: post #40

Earlier quoted context omitted.

Hi. I designed most of FlatBuffers, so let me see if I can clarify: Clearly, FlatBuffers seeks a different design tradeoff than Cap'n Proto. We wanted to retain all the flexibility of Protobufs, while having the advantages of the "zero parsing / zero allocation" approach. That brought us to the vtable design. In use cases where performance matters, i.e. you are churning through a large amount of data, you will be acc…

Structs are a welcome improvement. Using Protobuf, I was forced to move from a 3d vector type to an array of doubles to get packed=true performance. Small loss of expressiveness, no big deal, but I can imagine it getting a lot worse for heterogeneous and/or more complex structs. Was a shock to see a dynamic allocation for every vector in my 1000-element array. (I understand why this is necessary given the spec of Pro…

Yup. An array of Vec3 (3 floats) will just take N*12 bytes, as you expect.

Re: FlatBuffers: a memory efficient serialization library

#46

Post by Aardappel » Tue, Jun 17, 2014, 4:37 PM UTC

kentonv wrote: post #9

kentonv wrote: post #6

Huh. So Google is releasing a competitor to Cap'n Proto. As the former maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google), I'm pretty surprised that I hadn't heard about this. I also don't recognize any of the names, so this is not from the people who were working on Protobufs at the time I left. I'm the main competitor, so take me with a grain of salt here. The docs don't look very detailed…

OK, AFAICT there is no bounds checking . When you want to read a message, you give FlatBuffers a bare pointer to the start of the message -- no size. So you can't use this to read data you don't trust I guess. Which is an OK trade-off for certain situations (like reading your game data from disk). But... not for any kind of secure network protocol. Maybe I'm missing something, though. I've only been looking at this f…

Most readers of binary file formats can be made to read memory outside the buffer by corrupting the data, and FlatBuffers is no different.

That said, an option to bounds-check every offset would be possible, at a certain cost. Might be a nice optional feature to have.

Re: FlatBuffers: a memory efficient serialization library

#47

Post by kibwen » Tue, Jun 17, 2014, 4:49 PM UTC

gliptic wrote: post #41

mantraxB wrote: post #36

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages…

Rust does have unsafe direct memory access and a work-in-progress port of Cap'n Proto ( https://github.com/dwrensha/capnproto-rust ).

If any Pittsburgh HNers are interested, dwrensha is likely to be giving a presentation regarding his Rust-based Cap'n Proto library sometime in the next few months.

http://www.meetup.com/Pittsburgh-Code-Supply/

Re: FlatBuffers: a memory efficient serialization library

#48

Post by digitalboss » Tue, Jun 17, 2014, 5:49 PM UTC

emmelaich wrote: post #21

Just because I'm a fan of his, I'd like to point out that the author, Wouter van Oortmerssen aka Aardappel (strlen.com) is creator of the Cube 3d engine and lots of other interesting stuff. Oh I'm also a fan of Kenton's in case it matters :-)

And lobster http://strlen.com/lobster (game programming language open source). He's a personal friend as well, one of the wisest dorks I know :)

"Wouter van Oortmerssen is a humorous guy preferring strange names for his programming languages and often also programming examples."

Re: FlatBuffers: a memory efficient serialization library

#49

Post by ahminus » Tue, Jun 17, 2014, 5:51 PM UTC

kentonv wrote: post #33

emn13 wrote: post #29

Earlier quoted context omitted.

It's always surprised me that zero-copy serialization is worth it at all. Given how much slower memory is than CPU, once you've taken the effort to stream in data from somewhere, why not transform it into a more convenient format, especially if that means your on-the-wire format might be smaller. Is the key here that the input gets copied into memory directly without CPU intervention?

Well, first of all, there is mmap(), which does indeed make data available in memory without streaming it through the CPU first. If you have a 10GB file and really just want to read one data point out of the middle, with Cap'n Proto (and FlatBuffers) it's perfectly reasonable to mmap() the whole thing and traverse to that one item. Meanwhile, in networking, there is RDMA, which indeed allows memory to be transferred…

"In any case, the fact is that there are lots of real servers out there that are CPU bound and spend double-digit percentages of their time encoding and decoding Protobufs."

Having written two such servers (C++, fully async, single thread), this is true. At the time I chose Protobufs, it was the best option, but avoiding the encode/decode is a big win on an async. server that is handling 50,000 connections. Saving a few bytes here and there on the wire is much less important than blasting messages out and consuming them as fast as possible. This can also mean queueing as much of the work to be done by another process on the server side after the consumption of a message (i.e., work queues).

Re: FlatBuffers: a memory efficient serialization library

#50

Post by ahminus » Tue, Jun 17, 2014, 5:59 PM UTC

judk wrote: post #38

kentonv wrote: post #31

Earlier quoted context omitted.

I think there's some confusion here. To be clear, Cap'n Proto allows you to add new fields to a message without breaking backwards compatibility, and Cap'n Proto implements default values. The question is how a field is represented on the wire if it's defined in the schema but you don't assign it a value before serializing. With FlatBuffers, it won't take any space on the wire (although it still takes space in the vt…

IME, optional fields are used for...optional fields. Like when you have a 50-field data object but only want to transmit 5 of those fields.

I think if you have a 50 field data object, and you have cases where you transmit only 5 of those fields, the problem is your data modelling, not the serialization library.