Regarding 1), again that is how things should be, and to people's credit, they make fair attempts towards it. But the real world is messy. As an example, very frequently there are a number of ways to achieve similar results. Then often the argument becomes "why's your way better than mine"? Each party prefers a certain approach because it's better for them
for whatever reason. Hence a compromise must be reached, and this is where the wheeling and dealing begins. This is just one of many reasons why, as dragonwriter said, this becomes a political process.
While the solution you propose for 2) makes perfect technical sense, I'd say it steps too much into "inconveniencing millions of users" territory. Providing tools for migration and checking fidelity are big pain points because it does not fit into their usual workflow, which alone makes it highly unlikely to be adopted. If you wanted people to have minimum disruption, the new format would have to be a completely compliant reimplementation of the old one, and I'd guess it's inevitable that the new standard would look exactly like the old one simply because it's the path of least resistance.