Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

41–50 of 195 posts

Re: Don't Use Protobuf for Telemetry

#41

Post by googlyengineer »

Cloudef wrote: post #14

Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completely unrelated person to google. 1: https://jpa.kapsi.fi/nanopb/

As a google engineer, I would appreciate it if you did not spurt nonsense. Unless you have some clear evidence of your claims.

Re: Don't Use Protobuf for Telemetry

#42

Post by kjgkjhfkjf »

Luker88 wrote: post #33

Does anyone have an up-to date comparison protobuf vs captnproto instead? ( https://capnproto.org/ ) I am starting a new project and it looked really interesting, but I still have to read comments from people who used both in detail

CP was written by Kenton Varda, who spent many years working on protobuf at Google.

One massive advantage of protobuf for mainstream Google languages (C++, Java) is that Google has used them extremely heavily for many years, and you can trust that they've been extensively battle tested in Google's enormous high-traffic services, and scrutinized by Google's vast army of engineers. Kenton's experience notwithstanding, the same cannot be said of CP.

This argument doesn't apply for languages other than the ones that Google uses for production services.

Re: Don't Use Protobuf for Telemetry

#43

Post by notretarded »

thisiscorrect wrote: post #34
tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...

Re: Don't Use Protobuf for Telemetry

#44

Post by haberman »

tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/134406813297822105...

Re: Don't Use Protobuf for Telemetry

#45

Post by inetknght »

notretarded wrote: post #43
thisiscorrect wrote: post #34

Earlier quoted context omitted.

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...

Oh look sarcasm! On Hacker News!

Re: Don't Use Protobuf for Telemetry

#46

Post by Animats »

haberman wrote: post #44
tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/134406813297822105...

"possibly multi-MB messages"

Megabytes?

By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?

Re: Don't Use Protobuf for Telemetry

#47

Post by stkdump »

I created a binary serialization format and library that is comparable to json but binary. It uses 1-byte tags to give type information, which often already contains the value (in case of 0,1,true,false,null,empty string, empty list, empty object, single letter string or one of the last 32 interned strings) very similarly to msgpack. But it has the advantage, that strings are interned, which does about the same for performance as a runtime schema-based thing like Avro, but is more generic. Also, like bencode, it uses the convention that keys in objects are sorted, so that linear deserialization directly into the target objects is much simpler. And instead of prepending the size or length of objects and lists, there is a separate end-tag, so that you can start streaming data, before you know how much it will be. Sure, you can't skip without reading the data you skip over, but you (the receiver) can just do an initial pass and store the skip information in a cache, if you need. That seems better than requiring the sender to do multipass without knowing if anyone will ever use the skip info.

Re: Don't Use Protobuf for Telemetry

#48

Post by alisonkisk »

kjgkjhfkjf wrote: post #42
Luker88 wrote: post #33

Does anyone have an up-to date comparison protobuf vs captnproto instead? ( https://capnproto.org/ ) I am starting a new project and it looked really interesting, but I still have to read comments from people who used both in detail

CP was written by Kenton Varda, who spent many years working on protobuf at Google. One massive advantage of protobuf for mainstream Google languages (C++, Java) is that Google has used them extremely heavily for many years, and you can trust that they've been extensively battle tested in Google's enormous high-traffic services, and scrutinized by Google's vast army of engineers. Kenton's experience notwithstanding,…

Part of the reason that Kenton left Google and created CapnProto, was because he was the only person working on proto at Google trying to fix the problems Google found.

Re: Don't Use Protobuf for Telemetry

#49

Post by TickleSteve »

tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

pb is also used in low-resource/embedded systems (to send the data to the servers). In these cases, the processors typically have little to no cache and very limited RAM,

Performing multiple encoding passes is the only option and each pass is as expensive as the first.

Re: Don't Use Protobuf for Telemetry

#50

Post by [deleted] »

Animats wrote: post #46
haberman wrote: post #44

Earlier quoted context omitted.

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/134406813297822105...

"possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?

[deleted]
Post reply on HN