Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

81–90 of 166 posts

Re: FlexBuffers

#81

Post by mzaks »

kentonv wrote: post #68
mzaks wrote: post #58

Earlier quoted context omitted.

One use case where schema-less is the way to go, when you provide the infrastructure, but have no „ownership“ of data it will be used for. E.g. you build a logging or analytics tool where customers can send arbitrary data. Or a document database as a matter of fact. There schema-less / self described data is a must.

Not necessarily. For logging/analytics, you could have customers upload their schema when configuring the service. I would think that doing so would allow for some powerful optimization opportunities, enabling your service to save quite a bit of CPU and maybe some bandwidth, too. It would probably also allow you to provide a better user experience, like making it easier to construct dashboards and such because you ac…

True, I guess it all boils down to ease of use (convenience). You can build a system which accepts schema + data and build dash boards and data relevant relevant processing + optimisations, but that results in a much more complex system with higher entry burden. Sadly convenient systems always get broader adoption.

Re: FlexBuffers

#82

Post by [deleted] »

mzaks wrote: post #58
kentonv wrote: post #35

Earlier quoted context omitted.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

One use case where schema-less is the way to go, when you provide the infrastructure, but have no „ownership“ of data it will be used for. E.g. you build a logging or analytics tool where customers can send arbitrary data. Or a document database as a matter of fact. There schema-less / self described data is a must.

[deleted]

Re: FlexBuffers

#83

Post by lrem »

mzaks wrote: post #58
kentonv wrote: post #35

Earlier quoted context omitted.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

One use case where schema-less is the way to go, when you provide the infrastructure, but have no „ownership“ of data it will be used for. E.g. you build a logging or analytics tool where customers can send arbitrary data. Or a document database as a matter of fact. There schema-less / self described data is a must.

I own a service where users simply upload their schemas along their data. Doesn't seem to be a major issue.

Re: FlexBuffers

#84

Post by kccqzy »

jchook wrote: post #9
ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

AFAIK some languages can use C++ fairly easily, including D and Rust

These languages can use C fairly easily. To use C++, you must wrap the C++ interface in `extern "C"` interfaces. It's very hard to use the full spectrum of C++ features.

Consider that you're writing a custom struct/data type called MyStruct in your language. Now, can you use std::vector? This requires your other language to read the entire vector header and instantiate a new type based on both your definition of your custom type and the vector template. The vector instantiation will ask questions like whether or not your type is copy constructible and move constructible, and if so, use these implementations. Designing this into your other language requires a lot of compromises.

And that's just templates. ADL poses an even bigger difficulty. Again when you write your custom type in your language and define a swap function for it, will C++ standard algorithms that need swapping find your implementation via ADL? Recall that swapping is invoked via having `using std::swap;` and then invoking the bare function `swap` which triggers ADL. Again, designing this into your other language requires a lot of compromises.

The only language that I know of that comes close to having full interoperability with C++ is Objective-C, and it does so by creating a new Frankenstein language called Objective-C++. It has two different kinds of classes, two different exception mechanisms, etc.

Re: FlexBuffers

#85

Post by pjmlp »

kccqzy wrote: post #84
jchook wrote: post #9

Earlier quoted context omitted.

AFAIK some languages can use C++ fairly easily, including D and Rust

These languages can use C fairly easily. To use C++, you must wrap the C++ interface in `extern "C"` interfaces. It's very hard to use the full spectrum of C++ features. Consider that you're writing a custom struct/data type called MyStruct in your language. Now, can you use std::vector ? This requires your other language to read the entire vector header and instantiate a new type based on both your definition of you…

Wrong, D has support for C++ FFI.

https://dlang.org/spec/cpp_interface.html

Re: FlexBuffers

#86

Post by kccqzy »

pjmlp wrote: post #85
kccqzy wrote: post #84

Earlier quoted context omitted.

These languages can use C fairly easily. To use C++, you must wrap the C++ interface in `extern "C"` interfaces. It's very hard to use the full spectrum of C++ features. Consider that you're writing a custom struct/data type called MyStruct in your language. Now, can you use std::vector ? This requires your other language to read the entire vector header and instantiate a new type based on both your definition of you…

Wrong, D has support for C++ FFI. https://dlang.org/spec/cpp_interface.html

It supports neither templates nor ADL, the two of my examples.

In fact the doc says,

> Being 100% compatible with C++ means more or less adding a fully functional C++ compiler front end to D. Anecdotal evidence suggests that writing such is a minimum of a 10 man-year project, essentially making a D compiler with such capability unimplementable.

It then goes on to describe the pragmatic approach that doesn't allow full compatibility:

> D takes a pragmatic approach that assumes a couple modest accommodations can solve a significant chunk of the problem:

> matching C++ name mangling conventions

> matching C++ function calling conventions

> matching C++ virtual function table layout for single inheritance

Can you see how limited this subset of interoperable API is?

Re: FlexBuffers

#87

Post by bsaul »

Aardappel wrote: post #44

I made this thing! AMA :)

there is a comment up thread from people at google claiming flatbuffers aren't faster than protobuf in practice. that surprises me a lot, as i was interested in flat buffers for performance gains. do you have an opinion ?

Re: FlexBuffers

#89

Post by occamrazor »

uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

Apache Arrow is schema-ful, zero-copy (it is both an in-memory and serialization format)

Re: FlexBuffers

#90

Post by gorset »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

> Aside from the mmap() call, you can do all this without even allocating any memory at all.

I think this is the important point when it comes to discussing zero-copy. I've written a custom protobuf implementation for java which can do exactly that.

It's a bit tricky since protobuf supports recursive messages and java's Unsafe is not as powerful as what you can have in C++. My trade-off was to require the caller to pre-allocate messages needed before parsing the data. This works great when working with multi-gigabyte files where you want to process a large number of (possibly nested) messages, but is not as ergonomic as normal protobuf code.

It obviously doesn't come for free, as you need to do a linear scan to find those tag-values, but there are ways to speed that up too, so it becomes very fast in practice.

I'm sure Cap'n Proto and FlatBuffers are faster for some use-cases (I haven't tested), but a very important point for me is to be wire-compatible with protobuf3 and its ecosystem... and still be zero-copy/zero-alloc.

Post reply on HN