Java example is very weird.
FlexBuffers
111–120 of 166 posts
Re: FlexBuffers
#112So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)
Some formats optionally support schemas, or bundle the schema with the file. I think Avro does that.
Re: FlexBuffers
#113So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)
XDR (schema-ful, zero-copy (if done correctly)), but it's older and even less cool.
[EDIT] and I forgot to mention re-entrant as well (again if done correctly).
Re: FlexBuffers
#114Earlier quoted context omitted.
Sounds like there's a point to be made for referential integrity too. That is, if a struct contains string A twice, when you read it back in you'd want both those pointers to be identical. You'd get this for free with Cap'n Proto, but it would require extra care with "one-copy" or looser definitions of "zero-copy."
Heh, well, you could get it for free in Cap'n Proto if Cap'n Proto allowed pointer aliasing. It doesn't, though, because if it did, then messages would not be trees, they'd be graphs, which ruins a lot of stuff. For example, a very common thing to do with a message is copy one branch of the tree into a different message. Deep-copying a branch of a tree is easy. Deep-copying a branch of a graph, though -- what does th…
Copying a branch of a DAG has about the same meaning as copying a branch of a tree, right?
Re: FlexBuffers
#115Earlier quoted context omitted.
One use case where schema-less is the way to go, when you provide the infrastructure, but have no „ownership“ of data it will be used for. E.g. you build a logging or analytics tool where customers can send arbitrary data. Or a document database as a matter of fact. There schema-less / self described data is a must.
Not necessarily. For logging/analytics, you could have customers upload their schema when configuring the service. I would think that doing so would allow for some powerful optimization opportunities, enabling your service to save quite a bit of CPU and maybe some bandwidth, too. It would probably also allow you to provide a better user experience, like making it easier to construct dashboards and such because you ac…
You can apply a Schema to your collection and have MongoDB enforce it . This has been in MongoDB since version 3.6. We support the JSON schema standard. https://docs.mongodb.com/manual/core/schema-validation/
Re: FlexBuffers
#116I made this thing! AMA :)
What do you think is the best buffer protocol to use for multiplayer games? We used Protobuf for a fast-paced .io game, but encoding-decoding turned out to be pretty slow and generated a lot of garbage in JS. We were in the process to switch to FlatBuffers (before the company went bankrupt), but the syntax made it feel harder to use compared to Protobuf, not sure about the performance though (we expected it to be faster and less garbage created because of the zero-copy).
So, would you recommend Protobuf, Cap'n'Proto, FlatBuffers or FlexBuffers for multiplayer games? The usual packets are game states or user input sent at high frequency.
Re: FlexBuffers
#117Earlier quoted context omitted.
This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…
Nothing of your comment was false but there's no intrinsic value to your stronger definition of zero-copy. A direct mapping to memory is not always optimal for performance. Indeed, there are high-performance computation packages that compress data structures in L1-cached-sized blocks, to save main memory bandwidth. So, you've used the word "achieve" to decorate an outcome that might not be optimal. By the way I worke…
Not-Faster on what platform? In Borg, in Google3 code, deployed on a fast machine with a nice fast wide memory bus and a large cache?
What about in embedded code, or in a game? A place where memory bandwidth is scarce, or where we're trying desperately to reduce the number of syscalls and jumps back and forth between kernel and user space?
Having the entire payload memory mapped, and copies avoided, makes an absolutely huge difference once these kinds of concerns are real. Having something mmap'd in theory means it's nice and hot and fresh in the kernel's mind, and potentially kept in cache.
Some people in my team had gRPC foisted on them to run on an ARM Cortex class device running a real time OS. It boggled my mind they were even able to get it to ship. Using something like flatbuffers would have made a lot more sense.
At Google protobuf is the veritable "once I have a hammer everything looks like a nail", to the point where I've seen protobufs in JSON payloads, or vice versa, or even deeper... protobof in JSON inside a Chromium Mojo payload... because, well, how good could it be without protobuf?
Re: FlexBuffers
#118Earlier quoted context omitted.
While it's true that XML is a generic serialization of angle-bracket markup that doesn't need a schema for serialization (which was exactly the main motivation for its "invention" as a SGML subset), the reason it's being used in inter-party or long-term/loosely-coupled service payload serialization is because it has fairly powerful and well-established schema languages (XML DTDs and XML Schema) for validation. This i…
Schema-less and schema-ful are better be split into schema-for-parsing, schema-for-validation, no-schema-invented-yet.
Or even schema-for-type, schema-for-value-validation
Just as an aside. The new version of OpenAPI (v3.1 RC) for REST interfaces now fully supports the json-schema validation mechanism so we may see an uptake in use of json schema.
Its not perfect and has some holes.
Re: FlexBuffers
#119Earlier quoted context omitted.
While it's true that XML is a generic serialization of angle-bracket markup that doesn't need a schema for serialization (which was exactly the main motivation for its "invention" as a SGML subset), the reason it's being used in inter-party or long-term/loosely-coupled service payload serialization is because it has fairly powerful and well-established schema languages (XML DTDs and XML Schema) for validation. This i…
Jsonschema exists. https://json-schema.org/
This is a later creation, not yet even finalized.
Re: FlexBuffers
#120So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)
The msgpack parser handles strings zero copy as well. Some tiny JSON parsers for embedded stuff do that as well.