Well... sort of.
For specific apps, or parts of apps, yes. Doing less is how you make things faster, highly agreed. And sometimes canvas allows you to do that, and then your app is much faster.
The problem is that in many cases, moving to canvas eventually turns into having a UI framework that renders to canvas, which turns into a layer of abstractions that handle keyboard and mouse events for you, including stuff like hover, which means suddenly you're tracking element position on your raster surface and thinking about z-indexes and event bubbling to parent elements...
I think this is part of the reason why individual apps that start using canvas and that can genuinely cut down on complexity by doing so tend to be able to get real speed improvements, but app frameworks like Flutter tend to perform so poorly. Eventually your cross-platform GUI toolkit like Flutter ends up being just another browser engine written in WASM. And in that scenario your approach becomes strict downside.
One good example: the browser doesn't expose an accessibility engine other than the DOM. So what I see apps end up eventually doing is either writing their own accessibility engine that doesn't work with programs like JAWS, or rendering out to a hidden DOM. For something like a game, you can get away that, maybe you don't even provide an accessibility layer at all. For a web component or a chart, a lot of your rendering might be unrelated to accessibility at all. But you get away with those kind of shortcuts because it's a targeted, specific use. For a big UI toolkit, it's harder to do that, and then surprise, suddenly you have all the overhead of updating a DOM tree and the overhead of updating a canvas.
When people talk about getting raw access to the graphics layer, I think it's important to understand there's a difference between apps that are genuinely reducing complexity vs the theoretical canvas-backed "universal web framework" that people sometimes talk about as just around the corner.