Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

151–160 of 166 posts

Re: FlexBuffers

#151

Post by senderista »

kentonv wrote: post #35
rurban wrote: post #26

Earlier quoted context omitted.

Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

You can also combine flatbuffers and flexbuffers: https://users.soe.ucsc.edu/~jlefevre/skyhookdb/fb_structure_...

Re: FlexBuffers

#152

Post by nitrogen »

sophiebits wrote: post #5
uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

Sounds like some implementations of MessagePack may be in the same category.

MessagePack/msgpack is great for that middle ground where JSON ser/des is too slow, but you don't have enough engineers to justify the maintenance burden of a heavier-weight schemaful protocol.

I did end up writing a simple schema verifier for Ruby (ClassyHash, on GitHub) in one of the jobs where I used msgpack, but I no longer have access to maintain it. My benchmarks showed msgpack+classyhash was faster than native JSON (didn't test oj I think) and other serialization formats, and faster than all the other popular Ruby schema validators at the time.

Tldr: msgpack rocks, use it instead of JSON for internal services

Re: FlexBuffers

#153

Post by Matthias247 »

cmrdporcupine wrote: post #117
jeffbee wrote: post #72

Earlier quoted context omitted.

Nothing of your comment was false but there's no intrinsic value to your stronger definition of zero-copy. A direct mapping to memory is not always optimal for performance. Indeed, there are high-performance computation packages that compress data structures in L1-cached-sized blocks, to save main memory bandwidth. So, you've used the word "achieve" to decorate an outcome that might not be optimal. By the way I worke…

Not-Faster on what platform? In Borg, in Google3 code, deployed on a fast machine with a nice fast wide memory bus and a large cache? What about in embedded code, or in a game? A place where memory bandwidth is scarce, or where we're trying desperately to reduce the number of syscalls and jumps back and forth between kernel and user space? Having the entire payload memory mapped, and copies avoided, makes an absolute…

> Some people in my team had gRPC foisted on them to run on an ARM Cortex class device running a real time OS. It boggled my mind they were even able to get it to ship.

I shipped an embedded project using an RTOS and Overall using Protocol Buffers was a great success in that project, since we could share schemas between IoT devices and the backend, and were able to generate a lot of code which would otherwise have been hand-written.

> Using something like flatbuffers would have made a lot more sense.

It might be able to solve the same problem. But it also needs to answer the questions: Is a suitable library available for all consumers of the data-structure? I don't think this was the case for our use-case, so it wouldn't have made more sense to use it back then.

Re: FlexBuffers

#154

Post by kentonv »

senderista wrote: post #150
kentonv wrote: post #135

Earlier quoted context omitted.

For a DAG, maybe, but it requires a lot more bookkeeping. Now you have to remember all the pointers you've seen before in order to detect dupes. To do that you probably need a hash map and some dynamic memory allocation, ugh. And what happens if you copy two different branches of one message into another, and they happen to share some children? Do you have to keep your seen-pointer map around across multiple copies?…

This problem (persistent graph structures) has been solved since the 90s: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.478...

This isn't a question of finding a magical solution, it's a question of trade-offs in performance, complexity, and usability that any solution necessarily imposes, and whether those trade-offs are worth it to support a feature that 99% of use cases don't need.

"Skyscrapers have been solved since the 20's" doesn't answer whether I should use steel beams when building a house.

Re: FlexBuffers

#156

Post by Aardappel »

bsaul wrote: post #87
Aardappel wrote: post #44

I made this thing! AMA :)

there is a comment up thread from people at google claiming flatbuffers aren't faster than protobuf in practice. that surprises me a lot, as i was interested in flat buffers for performance gains. do you have an opinion ?

I've worked with people internally, and it is indeed a challenge to speed up services with FlatBuffers. This is because 99.9% of internal communications flow over Protobuf, and many such services are simply: receive Protobuf, modify a few things, send it on the next service. So to deploy FlatBuffers there, you usually have to translate from Protobuf to FlatBuffers, do some high intensity stretch of communication in FlatBuffers only, then convert back. Given the cost of conversions, it would require replacing fairly large amounts of infrastructure for it to be a clear win, and given that those conversions have to be written entirely by hand, that is often not feasible. So translation of what the other comment said: "we gave up".

FlatBuffers is used to great effect, but often with new teams building new things, that are not yet shackled in Protobuf everywhere, like many of the AI related teams.

Re: FlexBuffers

#157

Post by genocidicbunny »

Aardappel wrote: post #142
XCSme wrote: post #116

Earlier quoted context omitted.

What do you think is the best buffer protocol to use for multiplayer games? We used Protobuf for a fast-paced .io game, but encoding-decoding turned out to be pretty slow and generated a lot of garbage in JS. We were in the process to switch to FlatBuffers (before the company went bankrupt), but the syntax made it feel harder to use compared to Protobuf, not sure about the performance though (we expected it to be fas…

I originally designed FlatBuffers for games (though admittedly more for things like level data or save game data than network packets), so I'd think it is pretty suitable. I had actually used Protobuf on a game project just before, and its performance problems led directly to the no-unpacking no-allocation design that FlatBuffers has. So FlatBuffers will make an incoming packet waaay faster to work with than Protobuf…

> Frankly, for the absolute highest performance (and lowest bandwidth) game networking you still need a custom encoding

This is really the truth here. The efficiency of a networking protocol for a multiplayer game is somewhat sensitive to the context; Are you trying to do a fast-paced game with lots of client prediction? Do you need to have guaranteed delivery? Do you expect your game to be used with highly-lossy networks, or mostly from stable connections?

It's not uncommon to find something like flatbuffers or flexbuffers available in a multiplayer game engine, but the high-performance systems like movement or ai will probably utilize a custom protocol better suited to their task.

Re: FlexBuffers

#158

Post by cbsmith »

uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

Except in truth Protobuf, Thrift, Avro, etc. all support doing the same "schemaless" decoding as FlexBuffers.

Re: FlexBuffers

#159

Post by Dylan16807 »

jeffbee wrote: post #138
fluffything wrote: post #137

Earlier quoted context omitted.

> Yes, you discard the ability to just jump to any random field, but that is not always important. I don't think this is the criticism being raised. The main criticism being raised against non-zero-copy serialization is that this often requires maintaining different memory representations for the same value - the copies are the consequence of transforming from one representation to another one.

We do that all the time in high-performance computing. You keep a packed representation in memory and unpack it in small pieces to operate on it. Sparse matrices, compressed columns, etc. This is not evil, it’s an adaptation to the way the machine works. Saying that Kenton’s definition of zero-copy is unconditionally better is an aesthetic argument and I don’t buy it.

> Saying that Kenton’s definition of zero-copy is unconditionally better

It feels like you're conflating things here.

It's not always a better algorithm, but it's a better definition.

Re: FlexBuffers

#160

Post by bsaul »

Aardappel wrote: post #156
bsaul wrote: post #87

Earlier quoted context omitted.

there is a comment up thread from people at google claiming flatbuffers aren't faster than protobuf in practice. that surprises me a lot, as i was interested in flat buffers for performance gains. do you have an opinion ?

I've worked with people internally, and it is indeed a challenge to speed up services with FlatBuffers. This is because 99.9% of internal communications flow over Protobuf, and many such services are simply: receive Protobuf, modify a few things, send it on the next service. So to deploy FlatBuffers there, you usually have to translate from Protobuf to FlatBuffers, do some high intensity stretch of communication in F…

Ok, now that gives a very different picture from what the original comment explained. Of course, protobuf to flatbuffer is slower than just protobuf...

Thanks a lot for your time (and work!). FYI, i'm planning on using flatbuffer for communication between a mobile app native code and a cross-platform library, hoping to save on data decoding time. A bit similar to what Xi ide is doing. If you have any advice, you're welcome ! :D

Post reply on HN