My assumption & opinion is they know exactly what they are doing. I assume every decision has firm logic & metrics to back it up. However, the very nature of the marketplace is for advertisers to squeeze as much value out of the customer as possible in order to purchase higher visibility. I don't see it as a mistake or a flaw, it is just the end reality of a set of circumstances.
What makes Google's marketplace uniquely challenging is the following: In the physical world, you can go to a store and make a pretty good judgement that the product being sold is real, it is high quality, the cashiers are happy, the (real) customers tell you they have shopped there for years.
Online, everything becomes about human perception of data. It is easier to represent quality when there is none. It is easier to sell a customer one thing and extract a great deal of additional value from them. When that happens, the winner is the one with the most visibility (I think Americans can greatly relate this to US elections.)
Are user reviews and testimonials the solution? Amazon is loaded with fake ones. Many companies exist already to help small business owners "tweak" Yelp reviews. Even when you put a real face behind a review, it can be gamed. A static review will always be quite different than judging something with all of your senses in person.
Load your own site or app up with ads and see what happens to user retention. Prediction based on my own experience: it tanks.
Google is in crises because their behavior tells me their revenue would drop immensely if the result pages had as many ads on them as they did 6 years ago. Google is in crises because the ad revenue is their life blood. They have a massive organization which must have this revenue stream to remain in business.
There is a segment of the market that will click on and respond to bad ads over and over and over again. There is another segment that is more perceptive. There is a whole other segment that doesn't even know any of this has happened because they have used ad blockers for years!
Google might be really good at avoiding the tipping point. I assume that right now they are balancing the flight to mobile with the drop in user retention from "too many ads." Or perhaps I am overestimating the impact of ads, and the real game for Google is making customers think they are just clicking on content.
Some verticals pillage their customers. Many verticals do not, the arms race has not happened for them yet. That is a good thing for Google, and an even better thing for Google's users.
Google can control what they do, however with every step they take in the "show more ads" direction, the opportunity for someone else gets that much bigger.
As pointed out in the original article, in some result pages 90% of above the fold is ads. Would you watch Game of Thrones if it was 54 minutes of ads and 6 minutes of content? I wouldn't.