Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

71–80 of 166 posts

Re: FlexBuffers

#71

Post by m0zg »

vore wrote: post #52
m0zg wrote: post #45

I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.

It looks like https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")

Far too complicated for what it's supposed to do. Google's RecordIO basically just concatenates messages together, with a few additional provisions for compression and handling small messages. In contrast, Riegeli depends on three different compressors and Abseil, and its file format specification is quite elaborate: https://github.com/google/riegeli/blob/master/doc/riegeli_re.... This could explain its lack of market penetration (only 221 stars at the time of this writing). The authors apparently don't care that it's not popular, since they don't even provide a code example on how to use it. It also apparently uses HighwayHash for integrity checking, which is overkill - such things should use crc32c, for which most modern CPUs have a dedicated, fast instruction.

Re: FlexBuffers

#72

Post by jeffbee »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Nothing of your comment was false but there's no intrinsic value to your stronger definition of zero-copy. A direct mapping to memory is not always optimal for performance. Indeed, there are high-performance computation packages that compress data structures in L1-cached-sized blocks, to save main memory bandwidth. So, you've used the word "achieve" to decorate an outcome that might not be optimal.

By the way I worked on protobuf performance at Google for years and we could never get flatbuffers to go any faster.

Re: FlexBuffers

#73

Post by liuliu »

ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

We use Flexbuffer to serialize aribitary Objective-C objects through reflection (rather than NSCoding). It is faster and stable for us in a few years actually (we implemented that in early 2019).

Re: FlexBuffers

#74

Post by Aardappel »

excerionsforte wrote: post #69
Aardappel wrote: post #61

Earlier quoted context omitted.

"market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.

Marketing is an umbrella term for spreading awareness. Less people that know about it then the less likely your product gets experimented with and then chosen. I don't think any recent serialization benchmarks include flexbuffers. There's different ways to make people aware of alternatives. I chose to spread word of this product on HN :)

Even that kind of "marketing" takes time :) But thanks for helping out.

Re: FlexBuffers

#75

Post by deadmutex »

uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

One thing I wish people knew more about is SBE, it seems to be super fast by design:

https://speice.io/2019/09/binary-format-shootout.html

Re: FlexBuffers

#76

Post by greggman3 »

jeffbee wrote: post #72
kentonv wrote: post #15

Earlier quoted context omitted.

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Nothing of your comment was false but there's no intrinsic value to your stronger definition of zero-copy. A direct mapping to memory is not always optimal for performance. Indeed, there are high-performance computation packages that compress data structures in L1-cached-sized blocks, to save main memory bandwidth. So, you've used the word "achieve" to decorate an outcome that might not be optimal. By the way I worke…

There is a big difference to me. I haven't used any of those systems but I have written plenty of console games where the artists and designers want to fill memory. That means I don't have memory for 2 representations, unparsed and parsed. I also want fast loading so loading say 4k at a time into some temp buffer and parsing into memory is also out. I load the file directly into memory, fix up the pointers, and use it in place.

Re: FlexBuffers

#77

Post by Aardappel »

mavam wrote: post #66
Aardappel wrote: post #44

I made this thing! AMA :)

Unlike MsgPack, FlexBuffers doesn’t support custom/extension types. Is this something that is planned for the future?

I searched for "msgpack extension types" and "msgpack custom types" and nothing conclusive came up, so you'll have to explain what you mean.

FlexBuffers is a schemaless format much like JSON, so is naturally extensible, and typeless, so I don't see how it would need any such thing.

Re: FlexBuffers

#78

Post by ludocode »

iso8859-1 wrote: post #30
armitron wrote: post #27

I'd rather use MessagePack. Clean, simple, small.

It has worse performance though. Not zero-copy.

It depends what you mean by zero-copy. It can't be traversed without parsing the structure, but strings and binary blobs can be used in-place without copying. (This is also possible with JSON even, for example with RapidJSON's in-situ parser.)

This has worse performance for decoding, but far better performance for encoding. Encoding any traversible format is guaranteed to be expensive because you have to encode it inside-out, calculating the nested sizes of everything as you go. For messages that are only encoded once and read once, for example network messages, a format that is traversible in-place is a poor choice.

Re: FlexBuffers

#79

Post by lultimouomo »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Worth noting that by the weak definition of zero-copy JSON can also have zero-copy deserialization; many JSON embedded libraries either write a null terminator in the original buffer or return a pointer and a length.

Re: FlexBuffers

#80

Post by boardwaalk »

JoshTriplett wrote: post #47
ChristianBundy wrote: post #20

Earlier quoted context omitted.

Yeah, protobuf requires that both parties agree on a schema whereas CBOR is self-describing like XML or JSON.

CBOR also has a schema format, but yes, it's typically used schemaless.

I hadn't heard of this, so I had to look it up: https://tools.ietf.org/html/rfc8610
Post reply on HN