Exactly this.
I tried to learn speed reading and got pretty decent at it, I can still do it to an extent. In all honesty, I remembered more about texts that I speed read than texts I would read normally.
The only real difference was that when speed reading I didn't have the feeling that I know the full text, but when tested the knowledge would just magically appear out of nowhere; knowledge I didn't know I had.
Eventually I stopped speed reading because it's too much effort. Usually when I'm reading it's to relax the mind, give it a low effort activity to keep itself busy while I rest. Speed reading seemed counter productive for that sort of thing.
Also, speed reading is rubbish for fiction books. You get none of the pleasant pictures in your head and watching the book like an imaginative movie, but all of the feeling of consuming information at a breakneck pace without much time to take pleasure in it.
PS: my main issue with normal reading is that it isn't taxing enough. Especially when I'm trying to study something. Then my mind starts looking for a distraction and I suddenly realise I have no idea what I'm reading. Or simply get bored of it. Speed reading very effectivelyf orces you to focus on what you're reading (probably why I have a higher retention rate when speed reading)