Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlexBuffers

google.github.io

Post reply on HN
Print view View original

41–50 of 166 posts

Re: FlexBuffers

#41

Post by dreamcompiler »

kentonv wrote: post #15
jeffbee wrote: post #12

Earlier quoted context omitted.

Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any refer…

This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message…

Sounds like there's a point to be made for referential integrity too. That is, if a struct contains string A twice, when you read it back in you'd want both those pointers to be identical. You'd get this for free with Cap'n Proto, but it would require extra care with "one-copy" or looser definitions of "zero-copy."

Re: FlexBuffers

#42

Post by Aardappel »

8organicbits wrote: post #13

> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about wha…

As you can see from the documentation, `AsInt64` is a convenience method that either says, I know this is an int, or make it so. There are also ways to check the type before you access, if you prefer. You can even check if its unsigned, if you want that level of type-safety.

Re: FlexBuffers

#43

Post by t0nyh0 »

uluyol wrote: post #3

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

Apache Avro is also another data serialization format. Schema-Ful, unsure about zero copying or copying.

https://avro.apache.org/

Re: FlexBuffers

#45

Post by m0zg »

I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.

Re: FlexBuffers

#46

Post by _nhynes »

ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

The codegen for flatbuffers is fairly extensive, but C++ is still the blessed language for gRPC, however [0].

[0] https://google.github.io/flatbuffers/flatbuffers_grpc_guide_...

Re: FlexBuffers

#47

Post by JoshTriplett »

ChristianBundy wrote: post #20
throw0101a wrote: post #18

Earlier quoted context omitted.

So in this/your view, Protobuf and CBOR would be used in different scenarios?

Yeah, protobuf requires that both parties agree on a schema whereas CBOR is self-describing like XML or JSON.

CBOR also has a schema format, but yes, it's typically used schemaless.

Re: FlexBuffers

#48

Post by bowmessage »

Aardappel wrote: post #42
8organicbits wrote: post #13

> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about wha…

As you can see from the documentation, `AsInt64` is a convenience method that either says, I know this is an int, or make it so. There are also ways to check the type before you access, if you prefer. You can even check if its unsigned, if you want that level of type-safety.

Right, but GP is suggesting that the requirement to ignore an 'err' value that would make the API user think twice about possible failure cases that could otherwise go ignored with the convenience API.

Re: FlexBuffers

#49

Post by Aardappel »

_nhynes wrote: post #46
ccktlmazeltov wrote: post #4

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

The codegen for flatbuffers is fairly extensive, but C++ is still the blessed language for gRPC, however [0]. [0] https://google.github.io/flatbuffers/flatbuffers_grpc_guide_...

gRPC FlatBuffers is supported in quite a few languages besides C++, Java, Go, Python, Swift, off the top of my head..

Re: FlexBuffers

#50

Post by excerionsforte »

Aardappel wrote: post #44

I made this thing! AMA :)

When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future.

I'm glad to have seen this, which was on https://serde.rs/, which seems to be a recent addition. I was looking for an alternative to messagepack.

Post reply on HN