Speeding up playback buys you a small gain, but 2x or 3x is an absolute upper limit in most cases. And yes, retention suffers.
It's not that I don't bump up speed in some cases, though it's not all. I also slow down, or replay (sometimes many times) especially good marterial.
(Or especially confusing material, which often proves not to be especially good.)
But ...
... the real gains aren't to be had in playing media at 2x or 3x realtime. It's in selecting sources which afford 10x, or 100x, or 1000x utility in information. Sturgeon's Law applies, there's far more content created (in any format: text, images, audio, video, data, software) than any one person could hope to attend to in 15,000 lifetimes.[1] Probably significantly more.
Instead, one should recognise one's own limits of information exposure, goals in aquiring knowledge, and the available sources, and seek out at least higher quality sources amongst those available. These needn't necessarily be based strictly on truth value --- there's merit in reading discredited philosophy, mythologies, religion, and even perhaps for some, fiction and poetry.[2] Particularly such works as are of cultural significance, or which might help one achieve inner balance or simple amusement.
But listening to crap at 3x speed ... is not true efficiency.
It doesn't look as if anyone's mentioned Mortimer J. Adler's How to Read a Book yet. I'd strongly recommend reading it, quickly or slowly, and incorporating the lessons it offers for how, and what, to read.
________________________________
Notes:
1. Not a number pulled from /dev/ass, for what it's worth, but at least a rough approximation. If a truly dedicated scholar might read 10,000 books over a dilligent lifetime, that's 167 books a year from age 20 onwards, and there are roughly 150 million books published, then reading the collected works of humanity would require roughly 15,000 lifetimes. That might be adjusted up or down a bit but is a good first approximation.
2. An attempte at humour, Dear Reader.