i'm pretty amazed that they didn't do this a long time ago... sites with no content above the fold have been a problem for a long time...
I think they have been trying for a long time, but while it's possible to conceive the idea in a second, it might be harder to find the right algorithms to do it in an effective way. After all they make changes every day and study the response of users randomly distributing new versions of the algorithm to a small group of users that are effectively beta testers without even knowing it. It is a known problem, that's…
a crappy page with nothing but ads is going to get clicks, and since Google is the #1 ad network, it means more money for Google.
if Google organic search results were perfect, people would never click on the ads. the worse the results are, the more likely the ads are better and you get trained to click... and ker-ching!
Google needs to be good enough to (i) discourage mass defection to Bing and Duck-Duck-Go, (ii) not produce public outrage as happened when A list bloggers were getting outranked by duplicate content, and (iii) not get in trouble for anti-trust. (Hint: if you want to run a spam farm, buy a second or third tier search engine.