Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

111–120 of 307 posts

Re: Arguing against using protobuffers

#111

Post by derefr »

dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

The author isn’t trying to say either of those things, really.

I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have.

(Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)

Re: Arguing against using protobuffers

#112

Post by justicezyx »

why_only_15 wrote: post #52
_dmurph wrote: post #31

This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like "Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses." According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be.…

Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense. [1]: https://www.androidauthority.com/alphabet-q4-2017-earnings-8...

Anytime Pull out the revenue gun. It sounds like the revenue is actually produced by the machines though. I wonder why there isn't another search company competing with Google...

Btw, if you use glass door to deduce Google's cost on employee, you probably have no idea how company works...

Re: Arguing against using protobuffers

#113

Post by lucio »

dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.

Re: Arguing against using protobuffers

#114

Post by _dmurph »

why_only_15 wrote: post #52
_dmurph wrote: post #31

This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like "Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses." According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be.…

Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense. [1]: https://www.androidauthority.com/alphabet-q4-2017-earnings-8...

True. I'm also guessing this number changes a lot with stock grants (that number was just base pay), and that yearly number seems pretty low (NYT has the entry level engineer at 124,000 https://www.nytimes.com/2017/09/08/technology/google-salarie...) so glassdoor might not have good data.

I wouldn't be surprised if it was much more than 6%.

Re: Arguing against using protobuffers

#115

Post by _dmurph »

justicezyx wrote: post #112
why_only_15 wrote: post #52

Earlier quoted context omitted.

Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense. [1]: https://www.androidauthority.com/alphabet-q4-2017-earnings-8...

Anytime Pull out the revenue gun. It sounds like the revenue is actually produced by the machines though. I wonder why there isn't another search company competing with Google... Btw, if you use glass door to deduce Google's cost on employee, you probably have no idea how company works...

lol yeah I know, but it's at least the low bar. I would guess employees are one of the largest expenses for Google.

Re: Arguing against using protobuffers

#116

Post by atombender »

hardwaresofton wrote: post #103
atombender wrote: post #94

Earlier quoted context omitted.

I don't think GraphQL is over-hyped at all. Maybe it's flawed, but the design is absolutely on the right traack. GraphQL completely changes how you work with APIs in a front end. I work on React apps, and by using GraphQL, a component's data requirements can now be entirely declarative. For example, a component can do this (simplified): {({data, loading, error}) => { return {data.posts.map((title, {creator}) => {titl…

Nothing about what you just posted couldn't be done with a normal RESTful endpoint with sufficient support for column-level filtering and embedded item filtering. Your post is the perfect example of GraphQL is being over hyped. I absolutely get that there's a benefit to filtering at both these levels, and that the DSL cuts down on noise and gives you a way to "query" without thinking of web requests, but it's not a l…

Could one invent an ad-hoc REST API to perform joins, apply parameters and so forth? Of course. That's what people have been doing. That means every solution is ad-hoc. People get tired of this.

Have you tried any of the GraphQL client tools? You can literally point a client at an arbitrary server and not only see its schema, but also interact with the API. REST doesn't give you that, because the authors of REST neglected to actually specify anything beyond some fuzzy principles.

> ??? The component doesn't know anything, components don't think.

This is being obtuse. You know perfectly well what I meant: That the component localizes the information needed to make an informed request. This is domain knowledge encoded in the component.

This is contrary to how most developers design clients, where they might call an endpoint such as /posts?limit=10, which happens to fetch all attributes, even if the component doesn't actually need all the attributes.

> You've lost me here. The excerpt you've posted looks even worse than SQL.

Before a summary (and, honestly, tone deaf) dismissal such as this, I recommend reading up on it a bit [1], and maybe trying it out. SQL is great, but it does not handle nested documents/relationships, so you end up with a lot of messy structural mapping between flat relational data and structured data, which is why projects like Rails/ActiveRecord and Hibernate are so popular.

For example, a query such as this:

  *{
     name,
     photos->{ url },
     friends->{ name }
  }
will return something like:

  [
    {
      "name": "Bob",
      "photos": [
        {"url": "http://..."},
        {"url": "http://..."}
      ],
      "friends": [
        {"name": "Jane"}
      ]
    },
    ...
  ]
The point isn't that you cannot, with a sufficiently complex ORM and enough elbow grease, do that with SQL. It's that it should be unnecessary, because developers shouldn't need to write an entire data layer every time they want to bring an app up. This is part of what GraphQL brings to the table, too.

[1] https://www.sanity.io/docs/data-store/how-queries-work

Re: Arguing against using protobuffers

#117

Post by batmansmk »

dweis wrote: post #60

Hi there, I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises. - D P.S. I also don't beli…

Dear D,

I m very interested in Protocol Buffers. Could you explain the tradeoffs you made while designing protobuff and what would you change if you were to design it now?

Cheers!

Re: Arguing against using protobuffers

#118

Post by schizoidboy »

dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

If we suppose that his conclusion is using boolean logic, then what you're saying is a strawman because of his last claim; namely, protobufs are bad if "[...] && !Google":

> They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has.

This dovetails with other arguments that I've seen recently that are becoming more frequent:

Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?

Re: Arguing against using protobuffers

#119

Post by candiodari »

lucio wrote: post #113
dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.

Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front.

* cap'n proto https://capnproto.org/

* flatbuffers https://google.github.io/flatbuffers/

* hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/

You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in.

Protobufs focus on:

* standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from)

* streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory)

* extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand)

* language agnostic

* composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable)

* an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?)

And in this protobuf will beat the above formats (except cap'n proto).

None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.

Re: Arguing against using protobuffers

#120

Post by saalweachter »

vl wrote: post #37
_dmurph wrote: post #21

Earlier quoted context omitted.

Right - this is the same as having the possibility of both required and optional. I'm saying there should not be any possibility of a 'required' field

In context of current protobuf design required was a mistake and optional is clearly better, but author argues about grand type system that is based on different principles, including stronger validation. Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.

From a practical standpoint the problem is that "required" handles a trivial subset of message validation.

I mean, I'm not going to claim that it never happens that your only constraint on a valid value of a message field is "present", but you quite often want to be able to require that one of three fields is set, or a number be between 0 and 1048576, or that a field be equal to an existing user ID, it that a string contain at least one printable non-punctuation, non-space letter.

So no matter your RPC message parsing code, you're going to need a custom bit of code in each of your handlers to say "This isn't a valid message, fuck off". Enforcing "this field should be of this primitive type" saves a lot of time, but it turns out that "this field should exist" doesn't save that much because you still have to write "... and have a sensible value" in your own code.

So you have a language feature which causes problems sometimes and doesn't really help much.

Post reply on HN