Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

191–200 of 307 posts

Re: Arguing against using protobuffers

#191

Post by whateveracct »

dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

He did give a way to solve the problem though -.- the author isn’t spouting dogma and in fact uses a lot of factual arguments against their design.

Re: Arguing against using protobuffers

#192

Post by whateveracct »

ender7 wrote: post #136
derefr wrote: post #111

Earlier quoted context omitted.

The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, th…

No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot. It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original. As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any…

How is the author showing off exactly? Because he said “coproduct” and “Prism” (both of which have more well-defined meaning than profobufs btw lmao)

Re: Arguing against using protobuffers

#193

Post by docker_up »

taeric wrote: post #161
docker_up wrote: post #147

Earlier quoted context omitted.

> every application has to be updated when a field is added, even if they do not use that field No, you maintain the older versions of the API. V1 of the API uses the V1 struct. V2 of the API uses the V2 struct, etc. Older applications maintain compatibility because it calls the older APIs, and you can convert between V1 to V2 and only keep one version of the API. Or, if you want, you can maintain both versions of th…

You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4. I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.

I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.

Re: Arguing against using protobuffers

#194

Post by camgunz »

hardwaresofton wrote: post #190
camgunz wrote: post #188

Earlier quoted context omitted.

GraphQL is more or less a data description language. It describes queries and mutations. There are JS libraries that take a GraphQL query/mutation, convert it into the spec'd JSON payload, send it to the backend (single endpoint), and map the requested fields back into a JS object to be used. This is super nice when paired with React -- it's hard to overstate how nice it is. On the backend, there are libraries that p…

This is an excellent response and is 100% correct. At this point all I can say about my stance is that I would have preferred a standard for HTTP+JSON instead of how GraphQL is completely different. I also completely forgot about HAL[0]. To be clear, I'm not against GraphQL, I understand it and I understand what it brings to the table but I just thought the alternative was better if only because it relied on a web of…

Ha thanks.

I have fundamental misgivings about "APIs as bespoke ORMs for browser clients" -- at least how they're currently commonly implemented ([db] -> [orm] -> [controller magic] -> [request parser & deserializer / response serializer]). It just has this sort of... ad-hoc lack of coupling. Things like Datasette or PostgREST feel more "right", but I'm still a little incredulous that they're sufficient for even the majority of needs.

Designing a system that can handle concurrent reads/writes at production scale is hard even when you control the entire stack. When you consider all the constraints that modern web dev labors under, it's a miracle any of it works at all. But to be honest, my suspicion is that we get it to work by trading 90% of our performance potential for all this extra machinery between the client and the DB. We needed to do this 20 years ago because DBs didn't have the functionality (ex: row-level security) they do now, but at this point, maybe it's time to revisit. Either way, I'm doubtful that choosing a schema and protocol for client/server communication would meaningfully move the needle. I think a more fundamental rethink is necessary. Maybe after 3-4 years of wasm.

Re: Arguing against using protobuffers

#195

Post by whateveracct »

anothergoogler wrote: post #150
finnthehuman wrote: post #122

Earlier quoted context omitted.

>experienced engineers don’t want to work with people who think like the author of this article - has opinions on interface design - isn't afraid to be wrong publicly - is brash on a personal blog I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job.…

Brash on a personal blog is tricky. In this post's case, I'd worry that the blogger has a hard time separating technical deficiencies from professional incompetence. "Designed by amateurs" is an over-the-top and dubious claim.

The Googlers who designed Protobuffers’ type system were undeniably amateurs in that realm without knowledge of the state-of-the-art.

Re: Arguing against using protobuffers

#196

Post by amluto »

tathougies wrote: post #108
amluto wrote: post #106

Earlier quoted context omitted.

htons, etc are very 1980s, and I’ll make a fairly strong claim: they should never be used in new code, with a single exception. The reason is that an int with network endianness simply should not exist. In other words, when someone sends you a four byte big-endian integer, they sent four bytes, not an int. You can turn it into an int by shifting each byte by the relevant amount and oring them together. And a modern c…

You could also use the functions with explicit bit widths, like bswap64 and bswap32.

No, you should not. By the time you have a wrong-endian uint64_t or whatever, you’ve already done it wrong.

Re: Arguing against using protobuffers

#197

Post by hermanradtke »

mlthoughts2018 wrote: post #70
mixmastamyk wrote: post #39

Earlier quoted context omitted.

Apache Thrift is another one.

Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.

This is part of the original lambda architecture. What would you recommend the schema be represented in instead? Or are you preferring something like NewSQL?

Re: Arguing against using protobuffers

#198

Post by why_only_15 »

_dmurph wrote: post #114
why_only_15 wrote: post #52

Earlier quoted context omitted.

Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense. [1]: https://www.androidauthority.com/alphabet-q4-2017-earnings-8...

True. I'm also guessing this number changes a lot with stock grants (that number was just base pay), and that yearly number seems pretty low (NYT has the entry level engineer at 124,000 https://www.nytimes.com/2017/09/08/technology/google-salarie... ) so glassdoor might not have good data. I wouldn't be surprised if it was much more than 6%.

I also wouldn't be surprised if it was more than 6% - but I very much doubt it could be over 12-15%. That's still not a very high percentage of revenue. It's most definitely a significant cost, but unlikely to be their biggest cost.

Re: Arguing against using protobuffers

#199

Post by ruslan_talpa »

camgunz wrote: post #194
hardwaresofton wrote: post #190

Earlier quoted context omitted.

This is an excellent response and is 100% correct. At this point all I can say about my stance is that I would have preferred a standard for HTTP+JSON instead of how GraphQL is completely different. I also completely forgot about HAL[0]. To be clear, I'm not against GraphQL, I understand it and I understand what it brings to the table but I just thought the alternative was better if only because it relied on a web of…

Ha thanks. I have fundamental misgivings about "APIs as bespoke ORMs for browser clients" -- at least how they're currently commonly implemented ([db] -> [orm] -> [controller magic] -> [request parser & deserializer / response serializer]). It just has this sort of... ad-hoc lack of coupling. Things like Datasette or PostgREST feel more "right", but I'm still a little incredulous that they're sufficient for even the…

in relation to your concerns about PostgREST check out these links http://docs.subzero.cloud/postgrest-starter-kit/ http://docs.subzero.cloud/architecture-and-project-structure...

Re: Arguing against using protobuffers

#200

Post by tathougies »

amluto wrote: post #196
tathougies wrote: post #108

Earlier quoted context omitted.

You could also use the functions with explicit bit widths, like bswap64 and bswap32.

No, you should not. By the time you have a wrong-endian uint64_t or whatever, you’ve already done it wrong.

After re-reading your comment above, I'm actually confused. You think you should never store a big-endian int? That is ridiculous. Some architectures are big-endian. You should not be using custom bitswapping as part of application code, because you cannot know the endianness of your architecture.

The ntoh* functions are the right approach, and your claim is not only strong, it's wrong. The ntoh* functions exist to transform network byte-order to host byte-order. Depending on your architecture endianness, their functionality will change.

Post reply on HN