Earlier quoted context omitted.
The point is that HTML is almost 30 years old, and based on SGML which is much older (even though ISO 8879 is officially "only" from 1986), where SGML is just a formalization of typesetting practices established in the 1960's and 1970's. Given the depth of usage of HTML in everyday life (laws, contracts in ecommerce, medical records, personal communication, education, etc., etc.), I think HTML deserves better than be…
The markup parts of HTML (e.g. the parsing) are pretty frozen and have been for a while. This is the part that can sort of argue it used to be based on SGML (though now it's not, for various reasons). The "HTML spec" includes a lot of APIs and processing model details that need tweaking as new constraints come up. A good example is that a lot of APIs that involve cross-window access need changes to their specificatio…
HTML doesn't cease to be based on SGML by mere declaration, or even by brittling it to the point it can't be parsed by any known formal standard. That's more a political stance, as in American isn't English or some such.
WHATWG's formulation of HTML has deliberately distanced itself from SGML out of ignorance and a desire to not being accountable or formally verifiable (aka move fast and break things) against established, rich theoretical foundations of markup languages. And it shows: already in a paper I published two years ago [1], I show flaws in HTML as described by WHATWG, some of which have since been fixed ([3]). Not only is the concept of "transparent markup" flawed and underspecified, it also has since be used in the definition of the content model for the dl element, and as an unintended consequence also the div element (cf [2]), flaws that could be easily avoided by just using SGML for the grammar WHATWG is attempting to describe when SGML has been around for ages.
It's also not entirely true that WHATWG HTML can parse all legacy docs. For example, the keygen element has been removed, and while not a terrible loss as such ;), since keygen is a void element (an element with declared content EMPTY in SGML parlance), its presence in a legacy document (eg without an end-element tag) will make HTML5 parsers fail hard ([4]). It's also completely unclear which version of HTML is being validated by eg. W3C's nu or another validator. Heck, even the spec text for WHATWG HTML itself reads
> This file is NOT HTML, it's a proprietary language that is then post-processed into HTML
when a large portion of the spec text portrays HTML in the role of an authoring language.
So tell me why, as an author, I should follow WHATWG's vision for HTML? As you say yourself, WHATWG hasn't advanced HTML the markup language at all, and has rather prevented the evolution of declarative UI features, to the effect of making JavaScript essential for all but the most basic documents.
[1]: http://archive.xmlprague.cz/2017/files/xmlprague-2017-procee...
[2]: https://github.com/w3c/html/issues/1116
[3]: https://github.com/whatwg/html/commit/6e305c457e42276bf275b8...
[4]: Edit: this affects differences introduced in HTML 5.2 vs HTML 5.1, not some distant archaic HTML 4 version