Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

281–290 of 307 posts

Re: Arguing against using protobuffers

#281

Post by killerstorm »

the_other_guy wrote: post #55

Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin

It definitely is not.

ASN.1 PER encodes data on a bit-level (that is, it represents integers using minimal number of bits they can fit in) and doesn't waste space on tags if not necessary. It is used in all kinds of telecom protocol, particularly mobile protocols such as LTE. Obviously, it needs to be both space-efficient and efficient in processing.

You can find more about ASN.1 uses here: https://www.marben-products.com/

Re: Arguing against using protobuffers

#282

Post by hardwaresofton »

taeric wrote: post #253
hardwaresofton wrote: post #143

Earlier quoted context omitted.

> My complaint on the items being less than compelling is that they almost instantly devolve into bitter fights about what I'm supposed to care about. I fully grant that XML was too heavy in much of its schema attempts. However, it is much easier to reason about the extremity of XML than it is where in the middle road I want to be in. Could you give a more concrete example? I'm really not following -- JSON + jsonsche…

XML/XSD did devolve into many fights. I will not argue against that. The main outcome was the wide adoption of JSON. What XML/XSD did give was ridiculously easy to implement autocomplete in any xml editor that was decently featured. And it gave people a ton of rope to try and over specify things. Jsonschema seems poised to repeat the rope mistake. That is, I'm of the mind that the anti-fragile practices of schemaless…

> Jsonschema seems poised to repeat the rope mistake. That is, I'm of the mind that the anti-fragile practices of schemaless JSON have actually been a boon to its success. Trying to solidify things back to a schema gives me little reason to think it will be better this time around.

Yeah I think I agree -- while considering your point I found myself wondering what made jsonschema any different this time around...

> I definitely agree it would be good to have bad decisions over turned. However, I will also not adopt a technology that is a constant grind to retread the same ground over and over. At least, not if I can avoid it. This puts me in the position of not trusting the likes of committees that seem to think throwing out decisions of a scant few years ago are a good idea.

Not like you need me to verify but that's an absolutely reasonable stance, the timeline of a standard and associated versions is important in deciding whether to use it or not.

> My assertion for "pretty easy to read and works pretty well for validation" was not broadly applied to everyone. In particular, it failed at cross vendor efforts. However, the original claim was merely the vague "people", and I stand by that being just as true for XML as it was for JSON. With similar levels of complexity regarding different bugs in parsers and whatnot.

I don't think I agree, but we can definitely agree to disagree here. XML was certainly easy to read (if you didn't use any advanced features) and as good for validation, but I think the simplest feature-complete JSON parser is more complex than the simplest feature-complete XML parser one could write -- but then again, maybe all the features I'm thinking of that XML had weren't in the core spec (I haven't read it).

Re: Arguing against using protobuffers

#283

Post by earthboundkid »

enneff wrote: post #65
atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the ne…

That’s what you should do with all serialization, but everyone always thinks it’s a good idea to tie the model object to the serializer object at the beginning of a project, and by the time you are ready to build a time machine to kill the person who made that mistake, it’s much too late to do anything but rewrite the whole thing.

Re: Arguing against using protobuffers

#284

Post by earthboundkid »

hardwaresofton wrote: post #103
atombender wrote: post #94

Earlier quoted context omitted.

I don't think GraphQL is over-hyped at all. Maybe it's flawed, but the design is absolutely on the right traack. GraphQL completely changes how you work with APIs in a front end. I work on React apps, and by using GraphQL, a component's data requirements can now be entirely declarative. For example, a component can do this (simplified): {({data, loading, error}) => { return {data.posts.map((title, {creator}) => {titl…

Nothing about what you just posted couldn't be done with a normal RESTful endpoint with sufficient support for column-level filtering and embedded item filtering. Your post is the perfect example of GraphQL is being over hyped. I absolutely get that there's a benefit to filtering at both these levels, and that the DSL cuts down on noise and gives you a way to "query" without thinking of web requests, but it's not a l…

The other problem with GraphQL is letting external clients make arbitrary queries is a denial of service attack in waiting, and even with internal clients, how much can you trust another department? Having ugly, non proper REST endpoints adds a negligible amount of implementation time to the client but lets the server ensure it can actually meet its performance goals.

Re: Arguing against using protobuffers

#285

Post by traviscline »

atombender wrote: post #75
enneff wrote: post #65

Earlier quoted context omitted.

An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the ne…

Yes, we use the same technique for our larger apps. We treat the Protobuf layer as a distinct layer related to the API; we have two packages, "protofy" and "unprotofy", so in the server implementation, you do something like: func (srv *Server) GetFoo( ctx context.Context, req *proto.GetFooRequest) (*proto.GetFooResponse, error) { foo, err := srv.db.getFoo(foo.ID) if err != nil { ... } pFoo, err := protofy.Foo(foo) if…

You might consider https://godoc.org/github.com/tmc/grpcutil/protoc-gen-tstypes

Re: Arguing against using protobuffers

#286

Post by taeric »

hardwaresofton wrote: post #282
taeric wrote: post #253

Earlier quoted context omitted.

XML/XSD did devolve into many fights. I will not argue against that. The main outcome was the wide adoption of JSON. What XML/XSD did give was ridiculously easy to implement autocomplete in any xml editor that was decently featured. And it gave people a ton of rope to try and over specify things. Jsonschema seems poised to repeat the rope mistake. That is, I'm of the mind that the anti-fragile practices of schemaless…

> Jsonschema seems poised to repeat the rope mistake. That is, I'm of the mind that the anti-fragile practices of schemaless JSON have actually been a boon to its success. Trying to solidify things back to a schema gives me little reason to think it will be better this time around. Yeah I think I agree -- while considering your point I found myself wondering what made jsonschema any different this time around... > I…

To be fair, I fully acknowledge I'm being cynical. And, to that, I'm not proud.

Re: Arguing against using protobuffers

#287

Post by amluto »

tathougies wrote: post #200
amluto wrote: post #196

Earlier quoted context omitted.

No, you should not. By the time you have a wrong-endian uint64_t or whatever, you’ve already done it wrong.

After re-reading your comment above, I'm actually confused. You think you should never store a big-endian int? That is ridiculous. Some architectures are big-endian. You should not be using custom bitswapping as part of application code, because you cannot know the endianness of your architecture. The ntoh* functions are the right approach, and your claim is not only strong, it's wrong. The ntoh* functions exist to t…

Let me try saying it differently. The following code is poorly written:

    char *buf = ...;
    uint32_t word = *(int32_t *)buf;
    uint32_t host_word = ntohl(word);
Because you just type-punned the read from buf. (In fact, this code is UB.) You could write it a little better like:

    char *buf = ...;
    uint32_t word;
    memcpy(&word, buf, 4);
    uint32_t host_word = ntohl(word);
Although IIRC there is or at least was still some disagreement as to whether this might be UB. You could use a union to make it definitely not UB.

But none of these variants are sensible, and, in fact, they don't even translate to most safer languages than C. The correct way to write this code is:

    char *buf = ...;
    uint32_t host_word = ((uint32_t)buf[0]
On any recent compiler, this will generate as good or better code, and it doesn't make pointless assumptions about the representation of uint32_t on the platform you're using.

So I stand by my claim: well-written modern C code should not contain any "network-order" values. They should contain bytes, vectors of bytes, and numbers.

Re: Arguing against using protobuffers

#288

Post by DougBTX »

joshuamorton wrote: post #276
docker_up wrote: post #270

Earlier quoted context omitted.

Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because…

>Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. This depends on the update. If indeed the field was optional, you won't. A common example would be a field that is necessary for a new feature, but without which everything functions just fine, or functions with a minor degredation in experience. But more imp…

> each server/client update dance has to be mostly atomic

There’s a nice write up showing how that’s a risk but not an absolute restriction of using messages with required fields here: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-...

The basic idea is to acknowledge that all systems with forward and backward compatibility will have a translation layer, the question is just how is that defined and implemented?

If all fields are optional, it means that all readers need to handle any field being missing, in other words all readers must be able to process empty messages. A user update message might be missing a user id, and the reader will have to handle that. A couple of options come to mind: do nothing if there is no user id, or return an invalid message error. The key is that this is a translation layer that can noop or error before the message reaches the service business logic.

Then another thought is that message schemes needn’t be defined with version ids, trying to define a strict ordering between message versions is hard as you say, especially when handling non-linear updates, eg rollbacks or readers and writers skipping versions.

Instead, let’s define message schema compatibility. The user message processor could be defined to say it will only process messages with user ids - which practically speaking will be the case regardless of the message definition format - then a message without a user id can be rejected by common message parsing code, without per-service per-field translation code.

With a clear set of compatibility rules, it is even possible to write sensible reusable schema compatibility checking, eg: https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/...

Re: Arguing against using protobuffers

#289

Post by hardwaresofton »

taeric wrote: post #286
hardwaresofton wrote: post #282

Earlier quoted context omitted.

> Jsonschema seems poised to repeat the rope mistake. That is, I'm of the mind that the anti-fragile practices of schemaless JSON have actually been a boon to its success. Trying to solidify things back to a schema gives me little reason to think it will be better this time around. Yeah I think I agree -- while considering your point I found myself wondering what made jsonschema any different this time around... > I…

To be fair, I fully acknowledge I'm being cynical. And, to that, I'm not proud.

No worrieds I'm pretty cynical myself which is why I'm pretty surprised when I see technology that actually makes me believe things could be less-shit in the future (hyperschema + json LD gave me that feeling).

Re: Arguing against using protobuffers

#290

Post by mping »

earthboundkid wrote: post #284
hardwaresofton wrote: post #103

Earlier quoted context omitted.

Nothing about what you just posted couldn't be done with a normal RESTful endpoint with sufficient support for column-level filtering and embedded item filtering. Your post is the perfect example of GraphQL is being over hyped. I absolutely get that there's a benefit to filtering at both these levels, and that the DSL cuts down on noise and gives you a way to "query" without thinking of web requests, but it's not a l…

The other problem with GraphQL is letting external clients make arbitrary queries is a denial of service attack in waiting, and even with internal clients, how much can you trust another department? Having ugly, non proper REST endpoints adds a negligible amount of implementation time to the client but lets the server ensure it can actually meet its performance goals.

I have seen research in this problem area, where one is capable of knowing ahead if the query is legit or not. Anyway for internal purposes it's fine
Post reply on HN