This is great. It means I won't see Reddit content popping up all over search results in other engines. Can Medium do the same? And perhaps Quora?
Google is the only search engine that works on Reddit now, thanks to AI deal
31–40 of 379 posts
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#32They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm)) But I can understand why they made the change they did. The data was being abused. My guess is that this was an oversight -- that they will do an audit and reopen it for search engines aft…
One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#33This is great. It means I won't see Reddit content popping up all over search results in other engines. Can Medium do the same? And perhaps Quora?
Yeah awesome, reddit was one of the last useful results beside the spam blogs and ai generated articles.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#34Earlier quoted context omitted.
This is about Reddit disallowing other search engines. Blame Reddit, not Google.
plenty of blame to go around
You'll have to demonstrate that.
Is Google's contract with Reddit exclusive, so that other search engines aren't given the opportunity to also pay?
I highly doubt that, especially since the DOJ would go after that immediately because of antitrust.
So no, pretty sure the blame here is 100% on Reddit unless you have evidence otherwise.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#35They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm)) But I can understand why they made the change they did. The data was being abused. My guess is that this was an oversight -- that they will do an audit and reopen it for search engines aft…
One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.
I agree with you in theory, but in practice someone has to pay for all this magic.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#36They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm)) But I can understand why they made the change they did. The data was being abused. My guess is that this was an oversight -- that they will do an audit and reopen it for search engines aft…
the article quotes reddit policy change: Reddit considers search and ads commercial activities and thus subject to robot.txt block and exclusion.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#37They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm)) But I can understand why they made the change they did. The data was being abused. My guess is that this was an oversight -- that they will do an audit and reopen it for search engines aft…
Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#38This is an interesting development. How many other sites might have leverage to charge to be indexed? I don't want to live in a world where you have to use X search engine to get answers from Y site - but this seems like the beginning of that world. From an efficiency perspective - it's obviously better for websites to just lease their data to search engines then both sides paying tons of bandwidth and compute to get…
Kagi uses at least Google and Mojeek
edit:
> Realistically, there are only 2 search engines now.
https://seirdy.one/posts/2021/03/10/search-engines-with-own-...
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#39Suppose a crawler or rival search engine doesn’t respect robots.txt, reddit can’t stop them. Make it a bit trickier, yes, but not stop them.
Re: Google is the only search engine that works on Reddit now, thanks to AI deal
#40Earlier quoted context omitted.
One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.
I agree with you in theory, but in practice someone has to pay for all this magic.
This is a false dichotomy. You can have services, and not have them devolve into complete unusability in the name of profit. This isn’t sustainable either. The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, no matter how much you believe in this weird frog-boil internet we’ve inherited now.