I don't like that SVG uses things like CSS and JS and requires pulling in the whole browser to display. Instead of being a simple vector image format, it became just an extension of HTML. Maybe we need a new format, and if someone decides to do it, please add ability to embed fonts, wrap text and decent animations.
But for the most part, I 100% agree, and I've been considering making a format for my own use-cases. I think the biggest issue is in agreeing as to what subset is necessary; plus, of course, getting any level of adoption (though the latter isn't a factor for my own use ... except in the sense that there are no tools to help).
For example, do we need animations? Gradients? If so on the latter, what kind?