I think the point of AMP is it gives Google a simple and reliable way to detect fast rendering pages that render well on mobile in a way that can't be easily gamed. When you allow unrestricted usage of JavaScript that's changing page content, loading styles etc. for example, detecting pages like this is more difficult. You can write performant pages without AMP but how would Google reliably detect pages like this? I'…
The same way it reliable detects the content of the site and everything else. Take a snapshot and keep the latest history.
You can easily change the content on a page with javascript so Google still needs to load everything to get an accurate representation. Since there are plenty of tools already (including Google's own) that measure page loading speed, it would be trivial to combine them into a performance score for search rankings.
If speed was actually made into a strong signal, major websites would've become as fast as AMP within months.