Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

121–130 of 166 posts

Re: FlexBuffers

#121

Post by mintyc »

XCSme wrote: post #116
Aardappel wrote: post #44

I made this thing! AMA :)

What do you think is the best buffer protocol to use for multiplayer games? We used Protobuf for a fast-paced .io game, but encoding-decoding turned out to be pretty slow and generated a lot of garbage in JS. We were in the process to switch to FlatBuffers (before the company went bankrupt), but the syntax made it feel harder to use compared to Protobuf, not sure about the performance though (we expected it to be fas…

My experience with capn proto was a mixed bag.

Quite apart from its neglect for years as Kenton was working on sandstorm.io, I found the low level access interfaces to extract best performance to be both fiddly and often rely on an intimate knowledge of the internals.

As is often the case, maximum performance for your use case requires more intimate detail.

I can't remember, but some of these zero copy protocols may still require byte order swapping on mixed endian systems. (But cache access times probably dwarf the byte swapping)

Re: FlexBuffers

#122

Post by fsaintjacques »

t0nyh0 wrote: post #43
uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

Apache Avro is also another data serialization format. Schema-Ful, unsure about zero copying or copying. https://avro.apache.org/

Not zero copy, requires deserialization.

Re: FlexBuffers

#123

Post by jeffbee »

cmrdporcupine wrote: post #117
jeffbee wrote: post #72

Earlier quoted context omitted.

Nothing of your comment was false but there's no intrinsic value to your stronger definition of zero-copy. A direct mapping to memory is not always optimal for performance. Indeed, there are high-performance computation packages that compress data structures in L1-cached-sized blocks, to save main memory bandwidth. So, you've used the word "achieve" to decorate an outcome that might not be optimal. By the way I worke…

Not-Faster on what platform? In Borg, in Google3 code, deployed on a fast machine with a nice fast wide memory bus and a large cache? What about in embedded code, or in a game? A place where memory bandwidth is scarce, or where we're trying desperately to reduce the number of syscalls and jumps back and forth between kernel and user space? Having the entire payload memory mapped, and copies avoided, makes an absolute…

> Not-Faster on what platform? In Borg, in Google3 code, deployed on a fast machine with a nice fast wide memory bus and a large cache?

Weird rebuttal, considering that my argument is that the copying nature of protobuf can save memory bandwidth.

Re: FlexBuffers

#124

Post by jeffbee »

m0zg wrote: post #45

I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.

TensorFlow's TFRecord is RecordIO.

Re: FlexBuffers

#126

Post by secondcoming »

I'm looking to replace our current Boost.Serialization code with something else. I have the following requirements:

- serialisation cannot be intrusive (i.e. I don't want a class definition to be generated from a schema file)

- zero copy

- some sort of versioning so any accidental message version mismatch between sender and receiver doesn't cause a crash / undefined behviour.

- many language support

Protobufs are intrusive, so that rules them out. JSON is just too verbose and slow. MessagePack looked like the most promising path to pursue at the moment. Flatbuffers look quite similar.

Re: FlexBuffers

#127

Post by [deleted] »

uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

[deleted]

Re: FlexBuffers

#128

Post by kentonv »

jd_mongodb wrote: post #115
kentonv wrote: post #68

Earlier quoted context omitted.

Not necessarily. For logging/analytics, you could have customers upload their schema when configuring the service. I would think that doing so would allow for some powerful optimization opportunities, enabling your service to save quite a bit of CPU and maybe some bandwidth, too. It would probably also allow you to provide a better user experience, like making it easier to construct dashboards and such because you ac…

You can apply a Schema to your collection and have MongoDB enforce it . This has been in MongoDB since version 3.6. We support the JSON schema standard. https://docs.mongodb.com/manual/core/schema-validation/

Cool! My time using Mongo was pre-3.6 so I didn't know about this.

Re: FlexBuffers

#129

Post by kentonv »

mzaks wrote: post #81
kentonv wrote: post #68

Earlier quoted context omitted.

Not necessarily. For logging/analytics, you could have customers upload their schema when configuring the service. I would think that doing so would allow for some powerful optimization opportunities, enabling your service to save quite a bit of CPU and maybe some bandwidth, too. It would probably also allow you to provide a better user experience, like making it easier to construct dashboards and such because you ac…

True, I guess it all boils down to ease of use (convenience). You can build a system which accepts schema + data and build dash boards and data relevant relevant processing + optimisations, but that results in a much more complex system with higher entry burden. Sadly convenient systems always get broader adoption.

Yeah, I see this as pretty similar to the debate over type-safe vs. dynamic languages. It used to be that people argued that dynamic languages were just way easier to use, and type-safe languages were just an optimization. But I think the real issue was that the tooling enabled by type-safe languages didn't exist yet. These days, TypeScript is no faster than JavaScript but is very popular, because of the tooling it enables, like editors with auto-complete and jump-to-definition.

I think protocols still don't have the level of tooling that makes it really obvious why schemas are better.

Re: FlexBuffers

#130

Post by nojvek »

Negitivefrags wrote: post #98
randombytes6869 wrote: post #94

Earlier quoted context omitted.

Very true. IMO many people that hate XML config files just haven't used an IDE that validates schema. Its super nice to have auto-complete and property validation on config files, something not offered by JSON or YAML. A good reason to stick with XML for complicated configs. Its one of reasons I don't mind maven. Yeah there's 1000+ line XML config file, but maven DTD is so tight nearly any syntax issue will be flagge…

Visual studio does schema validation for JSON and gives errors inline. There is a big list of supported schemas built in and you can define your own. Never actually looked at the format personally.

Json schema is what vscode uses. It’s a standard nowadays. Just as effective as XMLSchema.

https://jsonschema.net/

http://www.schemastore.org/json/

Post reply on HN