I agree with your general point that the search quality has gone down, quotes doesn't even always work anymore to get exact results. Looking into your suggested example: That turned out to be interesting and unexpected. So, the exact string you put here was "David, we have been expecting you - this is what you have been searching for - this place, David, is where dreams are born", which is what you get when you copy…
There is incentive to keep users on-site as long as possible; "our engagement metrics are rising".
There is incentive to shovel users, kicking and screaming, to product pages or advertisements.
Google, Youtube, Amazon and other giants have little incentive to improve search beyond "good enough that our users feel like we're trying to answer their query."