To this day, my most public contribution to reddit is that I wrote the code to put the title of the post in the URL. That was done specifically for SEO purposes. It was pretty much the only SEO optimization we ever did (along with a few DOM changes), because shortly after that, Google basically dedicated engineering effort specifically to crawling reddit. So much so that we lost the "crawl rate" button in our SEO adm…
It's mind-boggling to me that Google didn't create a spec to work with special partners whereby they could get syndicated data in a format that was easy to ingest, and easy for their partners to produce. Lighting dollars on fire just to serve up pages to Googlebot, when you could just periodically dump a journal of updates to Google, is just crazy imo. On edit: if only there were some way to do some kind of really si…
Of course, I know that some version of this can and does occur with classic web scraping too, but that is an arms race that a search engine can win