181–190 of 195 posts
There's a marshalling analog to Greenspun's Tenth Rule. Every sufficiently complicated serialization implementation contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of ASN.1. I'd add that this is true of most ASN.1 implementations as well. The length encoding is a solved problem in ASN.1 (just use CER).
Are there any decent CER encoders for C that are free?
I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…
You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing enginee…
Earlier quoted context omitted.
If you are arguing that there exist use cases for which Protobuf is ill-suited, then fine, I obviously agree. But you seem to be arguing that Protobuf is bad because it is not well suited to certain use cases, dismissing all other use cases as "non-serious". That is offensive.
I asked you not to argue semantics, but you're doing it anyway. I didn't respond here to argue protobufs actually is bad, since all software is suited to something for somebody somewhere. In fact, I can just do that now: Protobufs is bad. You seem to agree, because you haven't even advocated for it -- just flatbuffers and Cap'n Proto and the phonebook of alternatives. Listen, I don't care who you've worked for or wha…
Earlier quoted context omitted.
If you're designing a critical component that's going to impact your business indefinitely, you can take some time for due diligence. This is basic Shift Left mentality and it has lots of benefits.
This was 2001-ish. No one at the time had any idea how widely the thing would end up being used, nor how big the company would grow. And spending too much time dwelling on each part of the tech stack then could easily have given a competitor the opportunity to pull ahead, in which case Google wouldn't be what it is today.
Earlier quoted context omitted.
Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post…
Once upon a time, we were concerned with handling network traffic at wire speed. I can remember when the AIX folks were happy they could implement the TCP fast path entirely in the interrupt handler.
I want to be a fan of Apache Avro ( https://avro.apache.org/ ) so much for situations such as this. And while I like Avro as a standard, most of the implementations I have found (specifically in C/C++) are... lacking. I feel Avro would be a great fit for something like this close to zero overhead (assuming a pre-shared schema) but there is little to no support for pre-shared schema and the RPC part of the standard is…
I think AVRO is fairly well designed overall, but JSON schemas? If you need deeply nested objects just look away, it becomes unintelligible when trying to understand the schema with any real nesting. The company I was at used it fairly well as long as you keep things 1 or 2 levels nested at most, but we had a legacy schema that was 5 or 6 levels deep at some points and it was just a disaster. My anecdote anyways.
Earlier quoted context omitted.
> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out…
> This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. I think this is a really important point. I don't think I've ever told you this before, but I am really impressed by how quickly you turned out proto2. While there are some things here and there that we wish we could change, a lot o…
Though in retrospect, I don't know if proto2 was a net win, given all the migration pain it caused. If I were doing it again I would take a more incremental improvement approach on proto1. It would have taken a lot longer but with less pain, I think. That said... who knows if that would have resulted in a better or worse outcome. Open sourcing would have taken a lot longer.
Had I realized at the time that I was taking on a project with no good solutions... heh.
Earlier quoted context omitted.
Thanks for the insight, that makes sense! Perhaps this is too unhip, but I wonder - have you ever dealt with the OMG's Data Distribution Service (DDS)? I used to think it was too fuddy-duddy to even glance at, but the recent adoption of DDS by ROS2 made me take another look. DDS seems to have a lot of interesting properties, particularly the QoS and discovery mechanisms. What made me think of it is that DDS is really…
Eh, I'm not a big fan of the "provide paid support for open tech" model... it doesn't scale well, and it seems like it creates a perverse incentive to make the tech hard to use, to generate contracts.
Earlier quoted context omitted.
Once upon a time, we were concerned with handling network traffic at wire speed. I can remember when the AIX folks were happy they could implement the TCP fast path entirely in the interrupt handler.
Certainly you wouldn't use protobuf to encode TCP packets. It's meant for encoding application-layer messages, typically ones that aren't especially large but contain a lot of complex structure.
Earlier quoted context omitted.
> This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. I think this is a really important point. I don't think I've ever told you this before, but I am really impressed by how quickly you turned out proto2. While there are some things here and there that we wish we could change, a lot o…
Thanks! Though in retrospect, I don't know if proto2 was a net win, given all the migration pain it caused. If I were doing it again I would take a more incremental improvement approach on proto1. It would have taken a lot longer but with less pain, I think. That said... who knows if that would have resulted in a better or worse outcome. Open sourcing would have taken a lot longer. Had I realized at the time that I w…
There are lot of improvements we can make without touching API, but whenever the API itself is a barrier to further improvements, there are just few good options for managing that.
Your point about incremental changes reminds me of the Linus rant about "bundling", where he argues that incremental changes lead to a better result than a big bang rewrite: https://yarchive.net/comp/linux/bundling.html I very much agree with that approach when possible, but Linux has the benefit of a much narrower API offered to its users. The protobuf API is not only the API of the core library, but of every generated class. The massive surface area just makes any kind of change to generated APIs an enormous challenge (for example, returning string_view from accessors instead of std::string).