You have misunderstood the "required considered harmful" argument. It's not fundamentally about the abstract concept of required vs. optional but about the specific implementation in Protocol Buffers, which turns out to have had unintended consequences.
Specifically: As implemented, required field checking occurred every time a message was serialized, not just when it was produced or consumed. Many systems involve middlemen who receive a message and then send it on to another system without looking at the content -- except that it would check that all required fields were sent, because that's baked into the protobuf implementation.
What happened over and over again is that some obscure project would redefine a "required" field to "optional", update both the producer and the consumer of the message, and then push it to production. But, once in production, the middlemen would start rejecting any message where these fields were missing. Often, the middleman servers were operating on "omnibus" messages containing bits of data originating from many different servers and projects -- for example, a set of search results might contain annotations from dozens of Search Quality algorithms. Normally, those annotations are considered non-essential, and Google's systems are carefully architected so that the failure of one back-end doesn't lead to overall failure of the system. However, when an optional backned sent a message missing required fields, the entire omnibus message would be rejected, leading to a total production outage. This problem repeatedly affected the search engine, gmail, and many other projects.
The fundamental lesson here is: A piece of data should be validated by the consumer, but should not be validated by pass-through middlemen. However, because required fields are baked into the protobuf implementation, it was unclear how they could follow this principle. Hence, the solution: Don't use required fields. Validate your data in application code, at consumption time, where you can handle errors gracefully.
Could you design some other version of "required" that doesn't have this particular problem? Probably. But would it actually be valuable? People who don't have a lot of experience here -- including Jeff and Sanjay when they first designed protobufs -- think that the idea of declaring a field "required" is obvious. But the surprising result that could only come from real-world experience is that this kind of validation is an application concern which does not belong in the serialization layer.
> There is no metadata anywhere.
Specifically, you mean there is no header / container around a protobuf. This is one of the best properties of protobufs, because it makes them compose nicely with other systems that already have their own metadata, or where metadata is irrelevant. Adding required metadata wastes bytes and creates ugly redundancy. For example, if you're sending a protobuf in an HTTP response, the obvious place to put metadata is in the headers -- encoding metadata in the protobuf body would be redundant and wasteful.
From what you wrote it sounds like you think that if Protobufs had metadata, it would have been somehow easier to migrate to a new encoding later, and Google would have done it. This is naive. If we wanted to add any kind of metadata, we could have done so at any time by using reserved field numbers. For example, the field number zero has always been reserved. So at any time, we could have said: Protobuf serialization version B is indicated by prefixing the message with 0x00 0x01 -- an otherwise invalid byte sequence to start a protobuf.
The reason no one ever did this is not because the format was impossible to change, but because the benefits of introducing a whole new encoding were never shown to be worth the inevitable cost involved: implementation in many languages and tools, code bloat of supporting two encodings at once (code bloat is a HUGE problem for protobuf!), etc.