61–70 of 91 posts
https://reasonablypolymorphic.com/blog/protos-are-wrong/inde... For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
My previous commentary: https://news.ycombinator.com/item?id=18190005
https://reasonablypolymorphic.com/blog/protos-are-wrong/inde... For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
Earlier quoted context omitted.
TBH not really. I've talked to the founder a few times. He doesn't want to put his name on things, as he's kind of a private person and really wants to direct the spotlight at his employees. But he's just an engineer who has spent a lot of time working with Protobufs, not a sales person at all. (Disclosure: I was the maintainer of Protobuf who put together the first open source release at Google, and I made a small i…
Ok now I'm pretty interested in this company: 1. Very technical founder at the bleeding edge of the field. 2. Many people against this idea Usually a sign there's something really good here... or at least something that is super non-obvious to most people. I'm curious to learn what I'm missing... mind if I reach out?
Happy to answer emails but can't guarantee I have anything interesting to say.
https://reasonablypolymorphic.com/blog/protos-are-wrong/inde... For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
That’s a good idea. I bet the creator of cap’n proto would tell us what a bad idea this is. What does kentonv have to say?
Wow, “Oh look, we’re going to re-invent the wheel for what is at least the third time.” When I worked at Google I sat down one day across from one of the gRPC engineering leads who was talking about the things they were doing for the then current generation of gRPC. I asked if I could ask some questions about it and they agreed and then dissected their design in half a dozen ways that would fail in both non-important…
Note that Buf is building tooling around Protobuf/gRPC, not replacing them. Maybe I'm just another clueless millennial developer who doesn't understand the history of the 80's or whatever, but I've never been able to understand this claim that RPC is broken. There's a lot of assertions that everyone knows RPC was broken because smart people in the 80's said so but... no one has ever been able to give me a concrete re…
The basic premise being that you specify the interface and you can use tooling to build some skeleton code that makes the code the user writes look like any other code they write, and yet it might magically be running on half a dozen machines.
Of course the actual difference between invoking a “procedure call” which is simply a program counter change and the same stack you had before, and one where the parameters provided are marshaled into a canonical form so that at the destination you can reliably unmarshal them and correctly interpret them, where the step that had been done by the linker resolving one symbol in your binary is now an active agent that is using yet another protocol at the start of execution to resolve the symbols and plumb the necessary networking code. And the execution itself which may happen exactly as expected, or happen multiple times without you knowing it has done so, or might not happen at all.
The minimalist camp, of which I consider myself a member, says “No, you can’t make these seamless, they really are just syntactic sugar that lets you specify a network protocol.” In that simple world you acknowledge that, and plan for, any part of the process to fail. Your code had failure checks and exceptions that deal with “at most once” or “at least once” semantics, you write functions rather than procedures to be idempotent when you can to minimize the penalty of trying to maintain the illusion of procedure call semantics in what is in fact a network protocol implementation.
But there is another camp, and from the material Buf has put out they seem to be in that camp, which is “networking is hard and complicated, but we can make it so that developers don’t need to even know they are going over a network. Just use these tools to describe what you want to do and we’ll do all the rest.”
My experience is that obfuscating what is going on under the hood to lower the cognitive load on developers breaks down when trying to distribute systems. That is especially true for languages that don’t explicitly allow for it. The number of projects/ideas/companies that have crashed on that reef are numerous.
And there is this part : “All gRPC and Protobuf are doing that is different from HTTP is they're using a binary protocol based on explicitly-defined schemas. The protobuf compiler will take your schemas and generate convenient framework code for you, so you don't have to waste your time on boilerplate HTTP and parsing. And the binary encoding is faster and more compact than text-based encoding like JSON or XML. But this is all convenience and optimization, not fundamentally different on a conceptual level.”
I agree 100% with that statement, and that is exactly what ONC RPC does, and that is exactly what ASN.1 does, and that is exactly what DCS does. That same wheel, again and again. So what I was suggesting originally is that Buf should try to explain what they are doing that these other systems failed to do, and in that explanation acknowledge the reasons this wheel has been re-invented so many times before, and then explain how they think they are going to make a more durable solution that lasts for more than a few years.
https://reasonablypolymorphic.com/blog/protos-are-wrong/inde... For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
Sorry, I'm the author of Cap'n Proto and I think that article is full of shit. My previous commentary: https://news.ycombinator.com/item?id=18190005
> his article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering.
I'm not the author, but they mention their prior industrial experience with protobufs at Google, among other unnamed places.
I'm not a PL theorist either, and I see that you don't fully understand the problems of composability, compatibility, and versioning and are too eager to dismiss them based on your prior experience with inferior type systems. And here's why I think it is the case:
> > This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse.
You are conflating your experience with particular conventional tooling with a general availability of superior type systems and toolings out there. There's a high demand in utilising their properties in protocol designs today, where most of the currently popular protocols are hampering type systems for no good reason (no productivity gain, no performance gain, no resource utilisation gain).
Version negotiation is not the only option available to a protocol designer. It is possible to use implicit-for-client and explicit-for-developer strategies to schema migration. It is also possible to semi-automate inference of those strategies. Example [1]
> This seems to miss the point of optional fields. Optional fields are not primarily about nullability but about compatibility. Protobuf's single most important feature is the ability to add new fields over time while maintaining compatibility.
There are at least two ways to achieve compatibility, and the optional fields that expand a domain type to the least common denominator of all encompassing possibilities is the wrong solution to this. Schema evolution via unions, versioning, and migrations is the proper approach that allows for strict resolution of compatibility issues with a level of granularity (distinct code paths) you like.
> Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change.
This is false. In practice I want a schema versioning and deprecation policies, and not ever-growing domain expansion to the blob of all-optional data.
> It's that way because the "oneof" pattern long-predates the "oneof" language construct. A "oneof" is actually syntax sugar for a bunch of "optional" fields where exactly one is expected to be filled in.
this is not true either, and it doesn't matter what pattern predates which other pattern. Tagged unions are neither a language construct nor a syntax sugar, it's a property of Type Algebra where you have union- and product-compositions. Languages that implement Type Algebra don't do it to just add another fancy construct, they do it to benefit from mathematical foundations of these concepts.
> How do you make this change without breaking compatibility?
you version it, and migrate over time at your own pace without bothering your clients too often [1]
Wow, “Oh look, we’re going to re-invent the wheel for what is at least the third time.” When I worked at Google I sat down one day across from one of the gRPC engineering leads who was talking about the things they were doing for the then current generation of gRPC. I asked if I could ask some questions about it and they agreed and then dissected their design in half a dozen ways that would fail in both non-important…
Note that Buf is building tooling around Protobuf/gRPC, not replacing them. Maybe I'm just another clueless millennial developer who doesn't understand the history of the 80's or whatever, but I've never been able to understand this claim that RPC is broken. There's a lot of assertions that everyone knows RPC was broken because smart people in the 80's said so but... no one has ever been able to give me a concrete re…
https://reasonablypolymorphic.com/blog/protos-are-wrong/inde... For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
That’s a good idea. I bet the creator of cap’n proto would tell us what a bad idea this is. What does kentonv have to say?
You can draw your own conclusion based on the provided arguments and some additional exploratory work. Someone else's opinion is good but optional and is not always as insightful as your own discoveries.
Wow, “Oh look, we’re going to re-invent the wheel for what is at least the third time.” When I worked at Google I sat down one day across from one of the gRPC engineering leads who was talking about the things they were doing for the then current generation of gRPC. I asked if I could ask some questions about it and they agreed and then dissected their design in half a dozen ways that would fail in both non-important…
Note that Buf is building tooling around Protobuf/gRPC, not replacing them. Maybe I'm just another clueless millennial developer who doesn't understand the history of the 80's or whatever, but I've never been able to understand this claim that RPC is broken. There's a lot of assertions that everyone knows RPC was broken because smart people in the 80's said so but... no one has ever been able to give me a concrete re…
Earlier quoted context omitted.
Their whole thing is tooling around protocol buffers, not the data format itself. The format is completely open source and comes from Google.
Protobuf is not even all that good as a format, flatbuffers has generally better properties.