Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google is the only search engine that works on Reddit now, thanks to AI deal

404media.co

Post reply on HN
Print view View original

191–200 of 379 posts

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#191

Post by lotsofpulp »

ToucanLoucan wrote: post #49
jedberg wrote: post #35

Earlier quoted context omitted.

I agree with you in theory, but in practice someone has to pay for all this magic.

We did. As in we, the Internet, existed for a long time without anyone making money and we paid for the privilege. Websites were built and hosted at owner's expense, for years, with no expectation that they be financially rewarded. Sure some would run donation drives, or work with sponsors relevant to the community in question, but a whole ton, mine included, just cost me a lot of money over many years. Those website…

>Websites don't cost that much to run.

Popular websites that allow user content to be uploaded or linked do cost that much to run, due to content moderation.

There might be a small (relatively) forum here and there that a few good moderators are willing to slave away at keeping clean, but you will never see a website that allows user content with as many users as Reddit/Youtube/Instagram/etc be cheap.

Although, due to AI, the cost to spam the small forums might be so small that even they might come into the crosshairs.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#192

Post by meiraleal »

jsnell wrote: post #95
meiraleal wrote: post #72

Earlier quoted context omitted.

how can we keep paying the ever-growing profits of multi-trillion dollar companies? This is insane.

Reddit is 100x from being a trillion-dollar company, and is not profitable.

Reddit offers no magic is just a forum. Google used to do some magic decades ago and still profit from it.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#193

Post by Manuel_D »

nomilk wrote: post #77
miyuru wrote: post #68

Earlier quoted context omitted.

reddit blocked datacenter IPs even before this change.

Could a motivated scraper not buy IPs/proxies that aren’t in those ranges, i.e. to blend in with general users?

Proxy IPs are also known and typically blocked. In fact, you can't even browse reddit without logging in when connected to most proxies.

Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern web scraping: https://i.imgur.com/U2RXi5G.jpeg

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#194

Post by dev1ycan »

MichaelZuo wrote: post #165
WarOnPrivacy wrote: post #129

Earlier quoted context omitted.

> Realistically, there are only 2 search engines now. From the article: Many alternatives to GBY [Google, Bing, and Yandex] exist, but almost none of them have their own results; This seems to assert that ~0 other search providers do any crawling at all. Ever. Are we sure that's the case? (they could crawl but never ever return those results == more odd).

Bing provides far fewer verbatim results for pretty much all search queries that I've tested. And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively. And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above. So realistically there's just one search engine for the full web that actually does the work.

Brave has their own search engine, yandex I only use for reverse image search, baidu's interface is really clean and feels like old school google... but I don't speak chinese so I can't use it.

I hope that one day they get a western version

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#195

Post by shadowgovt »

ColinHayhurst wrote: post #140
WarOnPrivacy wrote: post #129

Earlier quoted context omitted.

> Realistically, there are only 2 search engines now. From the article: Many alternatives to GBY [Google, Bing, and Yandex] exist, but almost none of them have their own results; This seems to assert that ~0 other search providers do any crawling at all. Ever. Are we sure that's the case? (they could crawl but never ever return those results == more odd).

It's a very long article so understandable that you did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally. We are at 8 billion pages and totally independent [0], hence expressing our concerns to 404 media after being blanked by Reddit. [0] https://www.mojeek.com/about/why-mojeek

I mean, I didn't read on because it's paid.

I'm not taking their reporting without compensation, but that also means I didn't have the whole story. Such is life in this era of the internet.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#196

Post by dev1ycan »

NoMoreNicksLeft wrote: post #88
voisin wrote: post #67

Makes sense that Google did this deal since their search quality tanked and they became an de facto front end UI for Reddit.

Up until 2016 (I think, +/- 1 year), if you could remember 3 uncommon words in a comment, you could find any reddit post instantly on Google. I'd want to follow up on a thread from weeks ago, and it was magic. Number one result. Then one day that just stopped working, and even adding site:*.reddit.com didn't fix it. At the time, I think, I didn't realize that it was mostly Google's fault, I thought maybe Reddit had c…

it's so bad it's crazy, you can legit not find stuff on the internet anymore, it's the same with youtube, I search something and get like 20 or so results and then everything else is hidden.

it started when youtube removed the ability to search for videos older than 5 years, if I had to guess? cost saving, have every old video in cheaper storage... but it sort of fragments youtube, every couple of years you only get newer content.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#197

Post by pas »

dogleash wrote: post #116
popcalc wrote: post #5

# Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content. # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use. # pol…

> # Reddit believes in an open internet, but not the misuse of public content. Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.

it's even worse. it's not theirs (it's the users'), they are merely hosting it and using it (ToS gives them a fancy irrevocable license I guess).

so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#198

Post by thih9 »

Story / rant warning.

I remember seeing an unhelpful hyperlink for the first time. It was a random word in the body of a random tech site that redirected to a list of articles from that site tagged with that term.

I remember being stunned, my expectation was that the link would lead me to another website, one that would be an authoritative source on that term and freely accessible.

20 years later we get a paywalled article about fragmented web – and we’re not slowing down.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#199

Post by culi »

WarOnPrivacy wrote: post #129
ColinHayhurst wrote: post #38

Earlier quoted context omitted.

Kagi uses at least Google and Mojeek edit: > Realistically, there are only 2 search engines now. https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

> Realistically, there are only 2 search engines now. From the article: Many alternatives to GBY [Google, Bing, and Yandex] exist, but almost none of them have their own results; This seems to assert that ~0 other search providers do any crawling at all. Ever. Are we sure that's the case? (they could crawl but never ever return those results == more odd).

I believe Brave Search is also starting their own index. There are some tiny independent indexes too:

https://www.crawlson.com/ https://search.marginalia.nu/ https://wiby.me/ https://searchmysite.net/

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#200

Post by lmeyerov »

FWIW, we inquired to the reddit sales team about paying for data sometime last year, as we do similar elsewhere for use cases like helping emergency responders, and even though they were launching the program and asking for customers... no email back. Nor on our second and I think third attempt.

I'm not sure what to make of that.

Post reply on HN