What's infuriating to me about these types of "signals" to the search rankings is that they have little to do with the content that I'm searching for. Google will hide results that I might find useful because the web master hasn't kept up with whatever Google decided was today's best practices. How about ranking based on the best source for what I'm looking for?
In general, prioritizing speed highly helps small independent sites and over large bloated sites: they typically have less JS, fewer round trips on the critical path, etc. Make your site simple enough (ex: https://danluu.com/) and it will automatically be fast.
(Disclosure: I work for Google, speaking only for myself)