Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

61–70 of 166 posts

Re: FlexBuffers

#61

Post by Aardappel »

excerionsforte wrote: post #50
Aardappel wrote: post #44

I made this thing! AMA :)

When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ , which seems to be a recent addition. I was looking for an alternative to messagepack.

"market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.

Re: FlexBuffers

#62

Post by mzaks »

tkzed49 wrote: post #25
ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

FlatBuffers libraries are implemented in a big list of languages, as you can see on the website. FlexBuffers is not implemented in all the same languages yet, at least in the official Google repo. Otherwise, it is very relevant in any language--it's a binary serialization format.

If you need a language support for FlexBuffers please create an issue on GitHub repo. I ported it already to Swift, C#, Dart, Python, JS (is in review). If I am familiar with the language you need, it will take me just a couple of days to port it.

Re: FlexBuffers

#63

Post by kentonv »

Aardappel wrote: post #55
kentonv wrote: post #51

Earlier quoted context omitted.

I suppose you could layer something on top. But I think it would be tricky to come up with something with satisfying performance properties. A naive encoding of JSON into Cap'n Proto would result in messages that are much larger than JSON messages, because of all the pointers and padding and text field names. I haven't looked into exactly how FlexBuffers work but off the top of my head I suspect that leveraging FlatB…

FlexBuffers are actually not built on top of the FlatBuffers encoding, they have their own special purpose encoding, which tries to be as compact as possible while still allowing in-place access (details, search for FlexBuffers here: https://google.github.io/flatbuffers/flatbuffers_internals.h... ). Funny you should say vtables may not be worth it.. I was of a similar opinion (why would you have many fields that are…

> some of the Protobuf schemas in use at Google, with hundreds of fields, most unused

Yeah, I was quite familiar with those back when I was at Google. But, IIRC, a lot of them were semantically unions, but weren't actually marked as such mostly because "oneof" was introduced relatively late in Protobuf's lifetime.

Re: FlexBuffers

#64

Post by hackcasual »

kentonv wrote: post #35
rurban wrote: post #26

Earlier quoted context omitted.

Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

Flatbuffers can support some writeable updates and it's language support is pretty comprehensive.

> Personally I'm a strong believer that schemas are highly desirable

I'm currently working on a project that's very executable size sensitive, and having a fixed schema is crucial to that, as can optimize a lot more aggressively. We can even use dead code feedback to identify schema parts that aren't consumed on the client

Re: FlexBuffers

#65

Post by Aardappel »

kentonv wrote: post #63
Aardappel wrote: post #55

Earlier quoted context omitted.

FlexBuffers are actually not built on top of the FlatBuffers encoding, they have their own special purpose encoding, which tries to be as compact as possible while still allowing in-place access (details, search for FlexBuffers here: https://google.github.io/flatbuffers/flatbuffers_internals.h... ). Funny you should say vtables may not be worth it.. I was of a similar opinion (why would you have many fields that are…

> some of the Protobuf schemas in use at Google, with hundreds of fields, most unused Yeah, I was quite familiar with those back when I was at Google. But, IIRC, a lot of them were semantically unions, but weren't actually marked as such mostly because "oneof" was introduced relatively late in Protobuf's lifetime.

Agreed, most use cases are not this extreme. But I saw them as an "upper bound" on how people would stretch a serialization system. I didn't want to be the guy saying "640k should be enough for everyone".

No joke, I originally was arguing for 8-bit vtable entries because surely no-one ever needs more than 256 bytes worth of fields. Good thing my co-workers were smarter than me.

And yes, FlatBuffers has built-in unions from day 1, which was probably helpful.

Re: FlexBuffers

#67

Post by webmaven »

Aardappel wrote: post #61
excerionsforte wrote: post #50

Earlier quoted context omitted.

When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ , which seems to be a recent addition. I was looking for an alternative to messagepack.

"market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.

Rather than "market", you could "promote" or "evangelize" instead...

Re: FlexBuffers

#68

Post by kentonv »

mzaks wrote: post #58
kentonv wrote: post #35

Earlier quoted context omitted.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

One use case where schema-less is the way to go, when you provide the infrastructure, but have no „ownership“ of data it will be used for. E.g. you build a logging or analytics tool where customers can send arbitrary data. Or a document database as a matter of fact. There schema-less / self described data is a must.

Not necessarily. For logging/analytics, you could have customers upload their schema when configuring the service. I would think that doing so would allow for some powerful optimization opportunities, enabling your service to save quite a bit of CPU and maybe some bandwidth, too. It would probably also allow you to provide a better user experience, like making it easier to construct dashboards and such because you actually know how the data is structured.

For a document database, I don't agree at all. Some time back I spent more time than I'd like developing on Mongo, and boy did I wish I could actually tell it the schema of documents in each collection and have it enforce that (not to mention optimize based on it). A lot of developers actually use libraries on top of Mongo to define and enforce schemas.

Re: FlexBuffers

#69

Post by excerionsforte »

Aardappel wrote: post #61
excerionsforte wrote: post #50

Earlier quoted context omitted.

When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ , which seems to be a recent addition. I was looking for an alternative to messagepack.

"market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.

Marketing is an umbrella term for spreading awareness. Less people that know about it then the less likely your product gets experimented with and then chosen. I don't think any recent serialization benchmarks include flexbuffers.

There's different ways to make people aware of alternatives. I chose to spread word of this product on HN :)

Re: FlexBuffers

#70

Post by kentonv »

alexeldeib wrote: post #56
kentonv wrote: post #35

Earlier quoted context omitted.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

Thanks for linking the article. Despite the original article being about FlexBuffers, I happen to have been looking at FlatBuffers vs Cap'n Proto today. That article is a bit old; is there anything that stands out to you in the last ~5 years where things have diverged?

I haven't kept track of FlatBuffers so I don't know what might have changed there, except that I imagine they support a lot more languages now (probably more than Cap'n Proto honestly). Cap'n Proto's serialization layer hasn't changed very much in those 5 years; development focus has been more on the RPC system.
Post reply on HN