Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

161–166 of 166 posts

Re: FlexBuffers

#161

Post by Dylan16807 »

ludocode wrote: post #78
iso8859-1 wrote: post #30

Earlier quoted context omitted.

It has worse performance though. Not zero-copy.

It depends what you mean by zero-copy. It can't be traversed without parsing the structure, but strings and binary blobs can be used in-place without copying. (This is also possible with JSON even, for example with RapidJSON's in-situ parser.) This has worse performance for decoding, but far better performance for encoding. Encoding any traversible format is guaranteed to be expensive because you have to encode it in…

That's too general. Sometimes there's [almost] nothing to calculate, so no slowdown. Sometimes the traversible format can be used as your working data structure and then encoding is a no-op.

Re: FlexBuffers

#162

Post by mavam »

Aardappel wrote: post #77
mavam wrote: post #66

Earlier quoted context omitted.

Unlike MsgPack, FlexBuffers doesn’t support custom/extension types. Is this something that is planned for the future?

I searched for "msgpack extension types" and "msgpack custom types" and nothing conclusive came up, so you'll have to explain what you mean. FlexBuffers is a schemaless format much like JSON, so is naturally extensible, and typeless, so I don't see how it would need any such thing.

Here's the link to the spec about extension types: https://github.com/msgpack/msgpack/blob/master/spec.md#exten...

From there:

MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data. Applications can assign 0 to 127 to store application-specific type information. An example usage is that application defines type = 0 as the application's unique type system, and stores name of a type and values of the type at the payload.

Re: FlexBuffers

#163

Post by Aardappel »

mavam wrote: post #162
Aardappel wrote: post #77

Earlier quoted context omitted.

I searched for "msgpack extension types" and "msgpack custom types" and nothing conclusive came up, so you'll have to explain what you mean. FlexBuffers is a schemaless format much like JSON, so is naturally extensible, and typeless, so I don't see how it would need any such thing.

Here's the link to the spec about extension types: https://github.com/msgpack/msgpack/blob/master/spec.md#exten... From there: MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data. Applications can assign 0 to 127 to store application-specific…

Ah, thanks! I presume that is done for compactness, because I otherwise see no reason to pack an application type in with the built-in types.

FlexBuffers could support a similar scheme if we wanted to, though putting custom data in the format is already easy enough using the "blob" type, which is arbitrary bytes much like the MessagePack feature.

Re: FlexBuffers

#164

Post by mavam »

Aardappel wrote: post #163
mavam wrote: post #162

Earlier quoted context omitted.

Here's the link to the spec about extension types: https://github.com/msgpack/msgpack/blob/master/spec.md#exten... From there: MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data. Applications can assign 0 to 127 to store application-specific…

Ah, thanks! I presume that is done for compactness, because I otherwise see no reason to pack an application type in with the built-in types. FlexBuffers could support a similar scheme if we wanted to, though putting custom data in the format is already easy enough using the "blob" type, which is arbitrary bytes much like the MessagePack feature.

The nice thing with custom types is that you only need one type dispatching logic.

Sure, it works via stuffing the custom things in blobs, but then you have (1) an extra layer of indirection during dispatch, and (2) always waste a few bits or bytes to store the custom type information, even when you don't need it for actual blobs.

In principle, it's not a lot of effort to reserve a fraction of the type ID space for custom types. From a user's perspective, it's also much nicer to hear that a library cleanly uses existing extensions to the type system, rather than hijacking a general-purpose blob type.

If there's nothing that prevents it, feel free to take this as a feature request :-)

Re: FlexBuffers

#165

Post by Aardappel »

bsaul wrote: post #160
Aardappel wrote: post #156

Earlier quoted context omitted.

I've worked with people internally, and it is indeed a challenge to speed up services with FlatBuffers. This is because 99.9% of internal communications flow over Protobuf, and many such services are simply: receive Protobuf, modify a few things, send it on the next service. So to deploy FlatBuffers there, you usually have to translate from Protobuf to FlatBuffers, do some high intensity stretch of communication in F…

Ok, now that gives a very different picture from what the original comment explained. Of course, protobuf to flatbuffer is slower than just protobuf... Thanks a lot for your time (and work!). FYI, i'm planning on using flatbuffer for communication between a mobile app native code and a cross-platform library, hoping to save on data decoding time. A bit similar to what Xi ide is doing. If you have any advice, you're w…

If you use it for networked communications, by default use the verifier on the C++ side, would be my tip.

Re: FlexBuffers

#166

Post by Aardappel »

mavam wrote: post #164
Aardappel wrote: post #163

Earlier quoted context omitted.

Ah, thanks! I presume that is done for compactness, because I otherwise see no reason to pack an application type in with the built-in types. FlexBuffers could support a similar scheme if we wanted to, though putting custom data in the format is already easy enough using the "blob" type, which is arbitrary bytes much like the MessagePack feature.

The nice thing with custom types is that you only need one type dispatching logic. Sure, it works via stuffing the custom things in blobs, but then you have (1) an extra layer of indirection during dispatch, and (2) always waste a few bits or bytes to store the custom type information, even when you don't need it for actual blobs. In principle, it's not a lot of effort to reserve a fraction of the type ID space for c…

You'd also need a way to patch the custom (de-)serializer into your "dispatch", which currently there isn't.. as unlike MsgPack, FlexBuffers doesn't actually unpack anything.

So this feature would look like a new function IsCustom1() or AsCustom1() etc, where the latter would give you a pointer to the bytes to do your own thing.. so not much difference with Blobs.

Also note that because FlexBuffers is O(1) access to elements, inline data (most scalars) is all the same size, whereas variable sized data is stored over an offset. So again, would not be very different from blobs.

Post reply on HN