Earlier quoted context omitted.
Except for the difficulty of managing and keeping up to date with TLS (I'd have to upgrade my RPi's OS for example, to do it well), and the fact that it actually reduces performance for small simple sites with local readership, even with HTTP/2 for example. http://www.earth.org.uk/note-on-carbon-cost-of-CDN.html Edit: I note that I got downvoted, though no reply with reasons. I accept that it is currently widespread…
For most sites, the Google organic search penalty for not having https is reason enough.
I don't believe that any penalty is currently significant, nor likely to be soon. There's plenty of good historic sites which SEs don't want to exclude just because someone doesn't have the resources to rework them and keep them maintained with the latest TLS. (Note issues like embedded http resources that would all have to transitively converted to avoid warnings.)