And what do you do for more complex, stateful apps? I don't think it's fair to dismiss this problem as "just use HTML and CSS and a sprinkling of vanilla JS". What happens when you need to build anything more complex than a basic form? A dashboard, or web map, or Figma, or Slack, or Gmail, or Gsheets... everything from state to AJAX (and other async) to persistence to URL routing, etc. becomes insanely complex.
I feel like "don't use tooling/frameworks" is the web dev version of "you don't need Java or .NET or WPF or WinForms, just use assembly and gfx drivers and draw pixels on the screen". HTML and ECMAscript evolve very slowly relative to the needs of software businesses, and it would take 30x (if not 300x) as long to write complex apps in those alone. Having a "build and bundle" step is just the webdev version of compiling developer-friendly code (React and frameworks) into the "lowest common denominator" code of browser-parseable HTML + JS.
It's not about "cleverness", but being able to do your job in a reasonable amount of time and effort, vs going back to a 90s-style internet where every page is basically a dumb terminal and all the actual interactivity has to happen on the backend.
I think is a better option is to know your target audience, their expected device and connection speed (metro desktop? rural mobile?) and not to worry about over-optimization. Nobody cares if your website loads in a second if it's so hard to use that it takes THEM more time to do what they need. Better to take 5 seconds to load, and then be actually usable as an app.
FWIW I don't think web apps are getting slower because developers are getting worse, but because the newer frameworks are enabling new use cases that weren't before possible. Photo editing? You used to have to download and install Photoshop, which can take minutes or days depending on your network and connection. Now you can run Photopea in a browser window in a few seconds and do most of the same things.
As for core web vitals? It's not a bad thing to use a performance arms race to get websites to be faster. It forces frameworks to evolve their delivery, caching, loading, hydration mechanisms. But that doesn't have to mean "give up on frameworks and complex apps". It can also just mean keep up with their latest performance improvements, and/or hope that ECMAscript evolves faster, that things like Web Components or PWAs etc. become more commonplace.