Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

21–30 of 166 posts

Re: FlexBuffers

#21

Post by leakybucket »

uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

We're still early in development, but the ZNG format is one we're working on for schema-less, semi-structured, zero-copy operations:

https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1...

We currently use this to store security log data, but think it's an interesting midpoint between having no schema at all vs requiring schema registries to do useful work.

Re: FlexBuffers

#22

Post by ChristianBundy »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Thanks for this explanation!

Off-topic: I see a few links in your HN profile, is that the best place to keep up with the projects you're working on?

Re: FlexBuffers

#23

Post by top_coder »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Could you point to documentation on how does Cap'n Proto achieve this? Does it keep a header with offsets of bye positions for individual fields? What happens when a variable sized field is edited?

Re: FlexBuffers

#24

Post by taeric »

typon wrote: post #19
8organicbits wrote: post #13

> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about wha…

If you use cpp17, you can use the built-in std::optional to solve this problem.

Strictly, not the same. One returns a value and an error. One returns a value or nothing.

Having the error can signal both if a value could not be retrieved, or if it could be retrieved, but was coerced, as an example.

Re: FlexBuffers

#25

Post by tkzed49 »

ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

FlatBuffers libraries are implemented in a big list of languages, as you can see on the website. FlexBuffers is not implemented in all the same languages yet, at least in the official Google repo. Otherwise, it is very relevant in any language--it's a binary serialization format.

Re: FlexBuffers

#26

Post by rurban »

kentonv wrote: post #16
cortesoft wrote: post #6

Earlier quoted context omitted.

Aren't the newer versions of protobuf zero-copy?

No, not in the way that Cap'n Proto and FlatBuffers are. Protobuf can support zero-copy of strings and byte arrays embedded in the message. Cap'n Proto and FlatBuffers support zero-copy of the entire message structure. (Disclosure: I'm the author of Cap'n Proto and Protobuf v2.)

Kenton, so what are the main difference to your Cap'n Proto?

My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.

Re: FlexBuffers

#29

Post by iso8859-1 »

8organicbits wrote: post #13

> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about wha…

But look at the encoding, they compact the int down to the amount of bytes it needs for storage. There is no space to encode the type, and there is no way to distinguish encoded uint from an int. You must layer your own schema on top.
Post reply on HN