Bebop: An Efficient, Schema-Based Binary Serialization Format • Hacker News Board

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#51

Post by otabdeveloper4 » Tue, Mar 30, 2021, 4:28 PM UTC

pdimitar wrote: post #45

otabdeveloper4 wrote: post #36

Earlier quoted context omitted.

FlatBuffers creates its own demented and completely incompatible data structures when deserializing into C++. Which means you then need to copy the FlatBuffers structures into normal ones, defeating the entire point of "zero-copy" in the first place. This thing it looks like uses normal C++ structures under the hood, and if so that's a huge plus.

I don't see why C++ considerations get priority over everything else, or are you making another point?

They do get priority when you're primarily coding in C++, obviously. Not everything is Javascript-first. Different requirements for different folks.

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#52

Post by pdimitar » Tue, Mar 30, 2021, 5:14 PM UTC

otabdeveloper4 wrote: post #51

pdimitar wrote: post #45

Earlier quoted context omitted.

I don't see why C++ considerations get priority over everything else, or are you making another point?

They do get priority when you're primarily coding in C++, obviously. Not everything is Javascript-first. Different requirements for different folks.

I am not a JS dev, don't project your frustration on me. ;)

So okay, FlatBuffers doesn't map its zero-copy philosophy perfectly everywhere -- fact of life. What would you offer then? Which other format and/or library?

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#53

Post by otabdeveloper4 » Tue, Mar 30, 2021, 5:43 PM UTC

pdimitar wrote: post #52

otabdeveloper4 wrote: post #51

Earlier quoted context omitted.

They do get priority when you're primarily coding in C++, obviously. Not everything is Javascript-first. Different requirements for different folks.

I am not a JS dev, don't project your frustration on me. ;) So okay, FlatBuffers doesn't map its zero-copy philosophy perfectly everywhere -- fact of life. What would you offer then? Which other format and/or library?

Never used this Bebop thing, but I'd have definitely preferred it to FlatBuffers back when I was shopping for serialization libraries.

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#54

Post by The_rationalist » Tue, Mar 30, 2021, 9:40 PM UTC

Noob question: How does performance of state of the art Binary formats compare versus state of the art Json serializers such as SIMDjson? (both in throughput and file size)

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#55

Post by fyrn- » Wed, Mar 31, 2021, 12:16 AM UTC

andrewmd5 wrote: post #47

the_duke wrote: post #19

The article doesn't mention at all how those speedups have been achieved, what tradeoff the format makes and what it is optimized for - which would be the actually interesting part. A link to the benchmark code and description of the data would be nice. It also doesn't show data for FlatBuffer, which is often a lot faster and leaner than ProtoBuf, or for Capt'n Proto. ProtoBuf is not exactly known for amazing perform…

One of the authors of Bebop here: - The benchmark code is present in the laboratory directory of the repository. - We don’t compare to Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare. - Flatbuffers are fast but have a notoriously awful API to work with while also creating their own non-standar…

Even with that I think flatbuffer, is still the main competition for something like this, not including it might make this look more impressive, but including it and mentioning that bebop has some advantages over flatbuffers (although with less language support) would be more fair.

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#56

Post by dikei » Wed, Mar 31, 2021, 12:55 AM UTC

The_rationalist wrote: post #54

Noob question: How does performance of state of the art Binary formats compare versus state of the art Json serializers such as SIMDjson? (both in throughput and file size)

Good binary formats with separate schema are always gonna be smaller and faster than JSON, because it doesn't have to encode field labels with every messages. JSON field names wastes so much space if your values are mostly numbers.

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#57

Post by lttlrck » Wed, Mar 31, 2021, 1:13 AM UTC

vlovich123 wrote: post #25

jnwatson wrote: post #23

Earlier quoted context omitted.

Speaking of which, I’ve never seen a marshaling protocol with as sophisticated a message versioning scheme as ASN.1. The amount of thought that went into specifying how to allow “implementations from the future” to interoperate with those of the past is impressive.

And yet outclassed on any meaningful metric by generic serialization languages? ASN.1 is a programming language in addition to a data transport format opening up many opportunities for security vulnerabilities. Performance is shit because it’s using text to encode everything so that it’s human readable. Similarly parsing is extra complicated and slow for that reason. Serialization/deserialization requires a dedicated…

Are you confusing ASN.1 "the language" with the encoding rules that are driven by it?

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#58

Post by andrewmd5 » Wed, Mar 31, 2021, 5:05 AM UTC

fyrn- wrote: post #55

andrewmd5 wrote: post #47

Earlier quoted context omitted.

One of the authors of Bebop here: - The benchmark code is present in the laboratory directory of the repository. - We don’t compare to Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare. - Flatbuffers are fast but have a notoriously awful API to work with while also creating their own non-standar…

Even with that I think flatbuffer, is still the main competition for something like this, not including it might make this look more impressive, but including it and mentioning that bebop has some advantages over flatbuffers (although with less language support) would be more fair.

They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. but this means Bebop and FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers does makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet.

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#59

Post by user-the-name » Wed, Mar 31, 2021, 8:06 AM UTC

vanderZwan wrote: post #40

w23j wrote: post #39

Earlier quoted context omitted.

Wow, that CV is ... impressive.

I guess he's a bit like the programmer equivalent of the underground musician that's not that well known by the mainstream but who inspired a lot of other musicians. For example, he wrote the language that inspired Brainfuck[0]. [0] http://strlen.com/false-language/

Not just Brainfuck, basically the entire field of esoteric programming languages.

Re: Bebop: An Efficient, Schema-Based Binary Serialization Format

#60

Post by eska » Thu, Apr 01, 2021, 10:01 AM UTC

I "audited" the generated c++ code (looked at it for 10 seconds) and immediately noticed that the decode functions all create a copy of the input buffer..