Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

51–60 of 166 posts

Re: FlexBuffers

#51

Post by kentonv »

vlovich123 wrote: post #39
kentonv wrote: post #35

Earlier quoted context omitted.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

In theory, you could build something like flexbuffers for cap’n’proto if someone was really motivated, no?

I suppose you could layer something on top. But I think it would be tricky to come up with something with satisfying performance properties. A naive encoding of JSON into Cap'n Proto would result in messages that are much larger than JSON messages, because of all the pointers and padding and text field names.

I haven't looked into exactly how FlexBuffers work but off the top of my head I suspect that leveraging FlatBuffer's "virtual table" technique probably helps here. In (normal, schema-ful) Cap'n Proto, fields within a struct have fixed offset, meaning that unused fields still take space. As I understand it, FlatBuffers tries to avoid this by adding an extra layer of indirection -- each struct has a sort of "virtual table" which stores the offsets of each field, where some fields might not be present at all. If multiple structs in a message happen to end up with the same virtual table, then the virtual table is only written once.

Totally speculating here since, again, I haven't actually looked at FlexBuffers, but if I were building something isomorphic to JSON on top of FlatBuffers, I'd probably look into extending the virtual tables to index fields by name rather than number. So if you have two structures with the same set of field names, they can share a virtual table, and those field names only have to appear once. That'd be a pretty great way to compress JSON.

Back in Cap'n Proto, we don't have these vtables. For data with fixed schemas, my opinion is that these vtables seem like they require more bookkeeping than they are worth. But for dynamic schemas they seem like a much bigger win. So if you wanted to encode dynamic schemas layered on top of Cap'n Proto, you'd probably have to come up with some similar vtable thing yourself.

Re: FlexBuffers

#52

Post by vore »

m0zg wrote: post #45

I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.

It looks like https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")

Re: FlexBuffers

#53

Post by 8organicbits »

iso8859-1 wrote: post #29
8organicbits wrote: post #13

> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about wha…

But look at the encoding, they compact the int down to the amount of bytes it needs for storage. There is no space to encode the type, and there is no way to distinguish encoded uint from an int. You must layer your own schema on top.

I believe this is the docs that explain their type byte. That information is stored:

https://google.github.io/flatbuffers/flatbuffers_internals.h...

> A type byte is made up of 2 components (see flexbuffers.h for exact values):

> * 2 lower bits representing the bit-width of the child (8, 16, 32, 64). This is only used if the child is accessed over an offset, such as a child vector. It is ignored for inline types.

> * 6 bits representing the actual type (see flexbuffers.h).

> Thus, in this example 4 means 8 bit child (value 0, unused, since the value is in-line), type SL_INT (value 1).

Type aware methods like `.isInt()` wouldn't make sense if they couldn't determine the underlying type.

Re: FlexBuffers

#54

Post by mzaks »

Aardappel wrote: post #40
ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

FlexBuffers currently is implemented in C++, Java, Rust, Dart, Swift, (and JS under review).

You forgot Python and C# :). Plus a Unity3D specific version, which is battle tested in a game with more than half a million downloads ;).

Re: FlexBuffers

#55

Post by Aardappel »

kentonv wrote: post #51
vlovich123 wrote: post #39

Earlier quoted context omitted.

In theory, you could build something like flexbuffers for cap’n’proto if someone was really motivated, no?

I suppose you could layer something on top. But I think it would be tricky to come up with something with satisfying performance properties. A naive encoding of JSON into Cap'n Proto would result in messages that are much larger than JSON messages, because of all the pointers and padding and text field names. I haven't looked into exactly how FlexBuffers work but off the top of my head I suspect that leveraging FlatB…

FlexBuffers are actually not built on top of the FlatBuffers encoding, they have their own special purpose encoding, which tries to be as compact as possible while still allowing in-place access (details, search for FlexBuffers here: https://google.github.io/flatbuffers/flatbuffers_internals.h...).

Funny you should say vtables may not be worth it.. I was of a similar opinion (why would you have many fields that are not in use??) until people showed me some of the Protobuf schemas in use at Google, with hundreds of fields, most unused. This is what pushed me in the direction of the vtable design.

Data always starts out neatly.. but the longer things live, the more this kind of flexibility pays off.

Re: FlexBuffers

#56

Post by alexeldeib »

kentonv wrote: post #35
rurban wrote: post #26

Earlier quoted context omitted.

Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

Thanks for linking the article. Despite the original article being about FlexBuffers, I happen to have been looking at FlatBuffers vs Cap'n Proto today.

That article is a bit old; is there anything that stands out to you in the last ~5 years where things have diverged?

Re: FlexBuffers

#57

Post by kentonv »

dreamcompiler wrote: post #41
kentonv wrote: post #15

Earlier quoted context omitted.

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Sounds like there's a point to be made for referential integrity too. That is, if a struct contains string A twice, when you read it back in you'd want both those pointers to be identical. You'd get this for free with Cap'n Proto, but it would require extra care with "one-copy" or looser definitions of "zero-copy."

Heh, well, you could get it for free in Cap'n Proto if Cap'n Proto allowed pointer aliasing. It doesn't, though, because if it did, then messages would not be trees, they'd be graphs, which ruins a lot of stuff. For example, a very common thing to do with a message is copy one branch of the tree into a different message. Deep-copying a branch of a tree is easy. Deep-copying a branch of a graph, though -- what does that even mean?

Re: FlexBuffers

#58

Post by mzaks »

kentonv wrote: post #35
rurban wrote: post #26

Earlier quoted context omitted.

Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.

FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special…

One use case where schema-less is the way to go, when you provide the infrastructure, but have no „ownership“ of data it will be used for. E.g. you build a logging or analytics tool where customers can send arbitrary data. Or a document database as a matter of fact. There schema-less / self described data is a must.

Re: FlexBuffers

#59

Post by Aardappel »

bowmessage wrote: post #48
Aardappel wrote: post #42

Earlier quoted context omitted.

As you can see from the documentation, `AsInt64` is a convenience method that either says, I know this is an int, or make it so. There are also ways to check the type before you access, if you prefer. You can even check if its unsigned, if you want that level of type-safety.

Right, but GP is suggesting that the requirement to ignore an 'err' value that would make the API user think twice about possible failure cases that could otherwise go ignored with the convenience API.

You are choosing to use a schema-less, dynamically typed representation (where a strongly typed alternative is directly available). I'd say, the convenience of being able to just say I am going to assume this is an int is fitting. If you wanted to be forced to error check (which needs an if-then in most languages), you might as well use the schema based, strongly typed version of the system instead, which would guarantee correct types for you, and requires no error checking.

Re: FlexBuffers

#60

Post by tele_ski »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Maybe it should be called zero parse and zero copy? Seems like two different terms...
Post reply on HN