Arguing against using protobuffers • Hacker News Board

Re: Arguing against using protobuffers

#181

Post by com2kid » Thu, Oct 11, 2018, 2:22 AM UTC

candiodari wrote: post #119

lucio wrote: post #113

Earlier quoted context omitted.

I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.

Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beat…

Protobufs also run anywhere, there are libraries of it ported to almost any machine.

When my team was deciding on a binary serialization and RPC communication format, we needed something that ran on a Cortext m3 with no malloc, and every major phone platform (including WP at the time) and desktop OS.

It came down to cap'n proto, and Protobufs.

Protobufs runs everywhere. It won.

Re: Arguing against using protobuffers

#182

Post by rdtsc » Thu, Oct 11, 2018, 2:38 AM UTC

> Give the ability to parameterize product and coproduct types by other types.

This is "worse is better" at work. protobufs are inconsistent and types map to Java. But it's a feature because Java is a language designed for for your average programmer. It avoids words like "coproduct types". Because most people's eyes will glaze over that and their mind will think "this is too complicated, there must be something simpler, like a map or something..."

> "Google is filled with the world's best engineers," and that "anything they build is, by definition, not built by amateurs.

That's true too though. It's a major factor when picking tools. So protobufs are used because of Google. gRPC is used because of Google. Kubernetes because of Google. Angular because of Google. Go became popular because of Google etc. Of course there are technical advantages to those things and they are great tools, but I think quite often the reason is simply ... because it's Google.

Re: Arguing against using protobuffers

#183

Post by stouset » Thu, Oct 11, 2018, 2:42 AM UTC

asdfasgasdgasdg wrote: post #158

mnsc wrote: post #157

Earlier quoted context omitted.

Would you recommend someone, not Google, to standardize on protobuffers now?

I'd probably pick Cap'n Proto or Flat Buffers if speed were paramount, in a grass is greener sort of way. I haven't used either of those technologies, though, just read about them. I'm also cool with plain JSON, which is beautiful from an ease-of-getting-started and universality perspective. I also think GraphQL is super compelling, and there's something to be said for records-as-in-SQL. Mainly, I just think that int…

Absolutely no to JSON, IMO.

The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code.

Figuring out these as hoc formats is nigh impossible, so you end up just looking at the data over the wire and writing code that parses what you see. Except then you have broken code that doesn’t parse the less-common variants that have extra fields, don’t have extra fields, sometimes have their fields as stringified ints instead of just binary ints, etc.

Virtually every time I’ve seen JSON used as an internal interchange format, reverse engineering that format a few years later.has become a massive, error-prone, tedious, and time-consuming task. Save yourself from this ahead of time and pick formats that require a predefined structure, like protobufs.

Re: Arguing against using protobuffers

#184

Post by weberc2 » Thu, Oct 11, 2018, 2:53 AM UTC

nostrademons wrote: post #78

enneff wrote: post #69

Earlier quoted context omitted.

Minor clarification: the author of Capnproto worked on protobufs at Google but was not their original creator.

Minor clarification of the clarification: the author of Cap'n'proto was the original creator of Proto2, which was a ground-up (but binary compatible) rewrite of Proto1. The original authors of Proto1 were Jeff Dean & Sanjay Ghemawat. The current version of protobufs is Proto3, which AIUI is maintained by a team at Google (it was released after I left), and is an evolution of the Proto2 codebase with many new features…

Ah, my mistake. Thanks guys. :+1:

Re: Arguing against using protobuffers

#185

Post by packetslave » Thu, Oct 11, 2018, 3:17 AM UTC

romed wrote: post #139

packetslave wrote: post #71

Earlier quoted context omitted.

The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.

Proto doesn't really have a schema. You can encode a proto of type X and tell your program to decode it as type Y, and it might work. Might not, but might. You can easily imagine messages that are isomorphic on the wire. E.g. message X { optional string foo = 1; } message Y { optional X foo = 1; } Encoded buffers of X will decode as Y just fine.

Sure, but it has more of a schema than JSON does :-) That's why I said "you can be REASONABLY sure you've got a valid message" (if it decodes successfully). It might not be the actual message you THINK it is, but at least all fields are going to be there.

Then again, if you're accepting arbitrary bytes off the wire and blindly assuming they're going to be the correct proto as long as they decode, you're... most stubby services I worked on. :-)

Re: Arguing against using protobuffers

#186

Post by atombender » Thu, Oct 11, 2018, 3:21 AM UTC

_dmurph wrote: post #31

This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like "Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses." According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be.…

While you're right, I'd like to point out that an employee earning 120k in wages costs an employer a lot more than 120k. Social security tax, Medicare tax, Federal Unemployment Tax Act tax, possibly state unemployment tax, health insurance, and some states have something called workers' compensation insurance. At Google, I'm sure there's also a lot of benefits included that would count towards the cost of each employee.

Re: Arguing against using protobuffers

#187

Post by asdfasgasdgasdg » Thu, Oct 11, 2018, 3:27 AM UTC

stouset wrote: post #183

asdfasgasdgasdg wrote: post #158

Earlier quoted context omitted.

I'd probably pick Cap'n Proto or Flat Buffers if speed were paramount, in a grass is greener sort of way. I haven't used either of those technologies, though, just read about them. I'm also cool with plain JSON, which is beautiful from an ease-of-getting-started and universality perspective. I also think GraphQL is super compelling, and there's something to be said for records-as-in-SQL. Mainly, I just think that int…

Absolutely no to JSON, IMO. The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code. Figuring out these as hoc formats is nigh impossible, so you en…

This is the classic static vs dynamic typing question. Suffice to say, I don't think there is sufficient science or anecdata on the issue to really give a solid answer.

Re: Arguing against using protobuffers

#188

Post by camgunz » Thu, Oct 11, 2018, 3:33 AM UTC

asdfasgasdgasdg wrote: post #162

hardwaresofton wrote: post #126

Earlier quoted context omitted.

GraphQL certainly has a cleaner/less involved interface, but it's also less interoperable. My problem with it is that it doesn't offer much value for the amount of effort , and it isn't the paradigm shift it's claiming to be in the first place, it's just being marketed well. HateOAS[0]+Swagger[1]/jsonschema hyperschema[2] is enough to do what GraphQL does. > This is being obtuse. You know perfectly well what I meant:…

> GraphQL has offered you a way to avoid writing: > axios.get("/posts?limit=10&filter[0]=prop&filter[1]=otherProp&fetchEmbeddedEntities[0]=thing") Is that right? I've never used GraphQL, but I thought the main advantage was not that the client didn't have to write that, but that the server didn't have to write the handling code for the joins to the embedded entity (to pick one example), or any of the dozens of other…

GraphQL is more or less a data description language. It describes queries and mutations. There are JS libraries that take a GraphQL query/mutation, convert it into the spec'd JSON payload, send it to the backend (single endpoint), and map the requested fields back into a JS object to be used. This is super nice when paired with React -- it's hard to overstate how nice it is.

On the backend, there are libraries that process that JSON payload into whatever your framework wants (dicts in Python, for example). From that decoded payload, you have to figure out what to do. If you're using Django and you get something like

    {'name': 'Ruth', 'age': 85}

to your 'People' query, you need to ask your ORM for People named Ruth who are 85 years old, and return them. It is essentially exactly like REST API design, with a few important differences:

- You don't have to worry about irrelevant HTTP stuff. What status code is appropriate? What HTTP action? Isn't this URL a little wrong? All irrelevant.

- There is no standard for pagination or ordering. In fairness, REST's answer (do it in query params... just like everything else!) is a little unsatisfying. But REST never made grandiose claims about data description. You go through the whole GraphQL site and it's more or less just an exposition in how great it is to just _describe_ the fields you want to retrieve/change, but nothing about "what if I just want the first 10,000 results?" This is, as you might imagine, an important and common issue.

- There is no (meaningful) standard for errors.

I think it's a step in the right direction from REST. URL-based APIs really just never earn their keep, HTTP actions are too limiting, versioning weirds everything, query params naturally grow to contain/become their own DSLs. But as far as backend engineering goes, GraphQL really only addresses the problems we didn't really mind (URLs/HTTP actions) anyway. To answer your question directly: the benefit of GraphQL is... 98% realized only by API clients. The backend is still doing everything it used to.

Re: Arguing against using protobuffers

#189

Post by camgunz » Thu, Oct 11, 2018, 3:35 AM UTC

Fellshard wrote: post #79

hardwaresofton wrote: post #67

Earlier quoted context omitted.

I'd like to humbly suggest that we use JSON please, in particular: JSON + JSONSchema[0] +/- JSON Hyperschema[1] +/- JSON LD[2] It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guarantee…

I'd love to see something with the approximate structure of JSON, but less JavaScript-bound syntax. EDN has caught my eye, but then again I've always been fonder of Lisp-y syntax as a whole.

Yeah EDN is a great format. It should be a lot more popular than it is.

Re: Arguing against using protobuffers

#190

Post by hardwaresofton » Thu, Oct 11, 2018, 3:56 AM UTC

camgunz wrote: post #188

asdfasgasdgasdg wrote: post #162

Earlier quoted context omitted.

> GraphQL has offered you a way to avoid writing: > axios.get("/posts?limit=10&filter[0]=prop&filter[1]=otherProp&fetchEmbeddedEntities[0]=thing") Is that right? I've never used GraphQL, but I thought the main advantage was not that the client didn't have to write that, but that the server didn't have to write the handling code for the joins to the embedded entity (to pick one example), or any of the dozens of other…

GraphQL is more or less a data description language. It describes queries and mutations. There are JS libraries that take a GraphQL query/mutation, convert it into the spec'd JSON payload, send it to the backend (single endpoint), and map the requested fields back into a JS object to be used. This is super nice when paired with React -- it's hard to overstate how nice it is. On the backend, there are libraries that p…

This is an excellent response and is 100% correct.

At this point all I can say about my stance is that I would have preferred a standard for HTTP+JSON instead of how GraphQL is completely different.

I also completely forgot about HAL[0].

To be clear, I'm not against GraphQL, I understand it and I understand what it brings to the table but I just thought the alternative was better if only because it relied on a web of standards that were always aimed at doing more (semantic web).

[0]: http://stateless.co/hal_specification.html