Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is.Required and optional is just an encoding of nullability in the type system. This is a common feature in most modern languages (Go excepted). Clearly Google got a very long way with proto1, whose designers felt strongly enough this was important to put it into what is otherwise a very feature-lite system.
The crux of your argument is not fundamental to serialisation or messaging systems but rather, is specific to Google's environment, business and early design choices. It is wrong for virtually all users of serialisation mechanisms:
Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto
There are many, many examples where a field does start required and stay required for the lifetime of a system. But even if over time a required field stops being used and you'd like to clean up the data structures or protocols by removing it, what proto3 does is completely wrong for any normal business.
Obviously, you can always remove a required field by changing it to optional and updating the software that uses that field to handle the case where it's missing (perhaps by doing nothing). The "required considered harmful" movement that took hold at Google whilst I was there directly contradicts all modern statically typed language design, except for Go, which also originated at Google and to put it bluntly, almost revels in its reputation for having ignored any PL thinking from the past 20 years. But if you look at - say - Kotlin, which the Android team have adopted as their new practical working language for practical working programmers, it has required/optional as a deeply integrated feature of the language. And this is not controversial. I do not see long running flamewars saying that Maybe types or Kotlin-style integrated null checking is a bad idea. Basically everyone who tries it says, this is great!
So why did Google have this feature and remove it?
The stated rationale of the "optional everywhere" movement was something like this: if we make a required field optional and stop setting it, some server, somewhere, might barf and cause a production outage. And nothing is worse than an outage, therefore, it is better to process data that's wrong (some default zero value casted to whatever that field is meant to mean), than crash.
This set of priorities is absurd for basically any company that isn't Google or in a closely related business, i.e. consumer services in which huge sums of money can be made by serving data even if it's "wrong" (or in which correctness is unknowable, like a search result page). But for most firms correctness does matter. They cannot afford to corrupt data by interpreting a version skew bug as whatever zero might have meant.
Arguably even Google can't afford to do this, which is why Jeff Dean made "required" a feature of the proto1 language.
There's a simple and very bad reason for why protocol buffers were changed to make all fields optional - the protocol buffer wire format was set very, very early on in the companies lifetime when there were only a few servers and Google was very much in startup mode, writing everything from scratch in C++ ... and without using any other frameworks. If memory serves, they're called "protocol buffers" because they started as a utility class on top of the indexserver protocol. The class provided basic PushTaggedInt() type methods which wrote a buffer representing a request. Over time the IDL and compiler was added on top. But ultimately the wire format has never changed, except for re-interpreting some fields from being "byte array of unknown encoding" to being UTF-8 encoded strings (and even that subtlety caused data corruption in prod).
Look at this wire format! https://developers.google.com/protocol-buffers/docs/encoding
There is no metadata anywhere. A protobuf with a single tagged int32 field encodes to just three bytes. There's no version header. There's no embedded schema. There's nothing that can be used to even mark a message as using an upgraded version of the format. There is however a rather complex varint encoding.
This makes a ton of sense given it was refactored out of a protocol for inter-server communication in a web search engine (Google search being largely a machine that spends its time decoding varints). But having come into existence this way, protobufs quickly proliferated everywhere including things like log files which may have to be read years later. So almost immediately the format became permanently fixed.
And what can't you do if the format is fixed? You can't add any sort of schema metadata that might help you analyse where data is going or what versions of a data structure are deployed in production.
As a consequence, when I was at Google, nobody had any way to know whether there was something running in production still producing or consuming a protobuf that they were trying to remove a field from. With lack of visibility comes fear of change. Combined with a business that makes bazillions of dollars a minute and in which end users can't complain if the served results are wrong, you get protocol buffers.
This is not fundamental. Instead it reflects an understandable but unfortunate lack of forward planning when protobufs were first created. Other companies are not compelled to repeat Google's mistake.