Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google is the only search engine that works on Reddit now, thanks to AI deal

404media.co

Post reply on HN
Print view View original

271–280 of 379 posts

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#271

Post by freediver »

WarOnPrivacy wrote: post #169
MichaelZuo wrote: post #165

Earlier quoted context omitted.

Bing provides far fewer verbatim results for pretty much all search queries that I've tested. And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively. And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above. So realistically there's just one search engine for the full web that actually does the work.

> And Yandex isn't much better for non cyrillic search, I like Yandex when I'm rabbit-holing after obscure musicians/music. I routinely have a better experience than I do with DDG or Kagi or Goog.

Kagi also uses Yandex index so it would be unusual that something exists in Yandex and not in Kagi results.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#272

Post by shit_game »

dogleash wrote: post #116
popcalc wrote: post #5

# Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content. # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use. # pol…

> # Reddit believes in an open internet, but not the misuse of public content. Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.

Their license/Eula clearly state that Reddit has perpetual whatever to content posted on Reddit, but relying solely on DMCA for "stolen" content _yet again_ feels like a terrible way to deal with non-original content. Part of me hopes that Reddit gets hit with some new precidence-setting lawsuits regarding non-original content that requires useful attribution, but I double t that will ever happen.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#275

Post by qingcharles »

redcobra762 wrote: post #228
qingcharles wrote: post #222

Earlier quoted context omitted.

At the federal level; but states have their own laws. For instance, it can get you 5 years in prison in Illinois to violate a web site ToS. https://www.ilga.gov/legislation/ilcs/ilcs4.asp?DocName=0720...

Has anyone ever successfully been prosecuted for violating this statute?

I don't know. That data is really hard to put together.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#276

Post by synicalx »

imglorp wrote: post #209
roughly wrote: post #86

Boy, the LLMs have really been an apocalypse moment for the web, haven’t they? Between hoovering up and monetizing every bit of content they can without any attribution or compensation and the absolute flood of mediocre generated content, they’ve really done in the last straggling remains of the open internet. It’s not like everyone wasn’t already pulling the same grift, but quantity really does have a quality all it…

Of course, we have to be careful not to villainize a neutral tech. Instead let's call it what it is: unchecked capitalism and monopolistic behaviors. Capitalism seems to work ok for the common good until you remove all the protections. LLMs provide a defacto monopoly for the owner which must already be a near monopoly: they take vast resources to train; only a giant corp can afford to buy all the content and provisio…

> Of course, we have to be careful not to villainize a neutral tech

This is a very good point IMO. If we're going to chastise LLM's we may as well give servers, switches, routers, fiber-optic cable, and silicone a bollocking as well since that's ultimately what's facilitating all this.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#277

Post by quink »

qingcharles wrote: post #275
redcobra762 wrote: post #228

Earlier quoted context omitted.

Has anyone ever successfully been prosecuted for violating this statute?

I don't know. That data is really hard to put together.

That’s a blink and you miss it joke of the highest calibre - well done!

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#278

Post by maxnevermind »

immibis wrote: post #78
popcalc wrote: post #5

# Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content. # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use. # pol…

Nobody who wants to be successful obeys robots.txt. And I do mean nobody.

Has not NYT tried to sue OpenAI because of them ignoring robots.txt or you mean it's impossible to prove and / or it's still more profitable to just ignore robots.txt?

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#279

Post by 1vuio0pswjnm7 »

"If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn't rely on Google's indexing and search Reddit by using "site:reddit.com," you will not see any results from the last week."

The veracity of this statement is questionable.

I found at least four web search engines not using Google's index that produced results from the last week.

Example: Recent eruption at Yellowstone Black Diamond Pool

https://www.ecosia.org/search?method=index&q=site:reddit.com...

https://search.brave.com/search?q=reddit.com+black+diamond+p...

https://api.yep.com/fs/2/search?client=web&gl=all&no_correct...

   POST /sp/search HTTP/1.0
   host: www.startpage.com
   content-length: 74
   content-type: application/x-www-form-urlencoded
   query=site:reddit.com black diamond pool&abp=-1&t=&lui=english&sc=&cat=web
At least for this example, I got the same desired result using Reddit site search.

https://old.reddit.com/search/?q=black+diamond+pool

If anyone has some good examples of search queries that I can test showing why a search engine must be used, please share.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#280

Post by BeFlatXIII »

Manuel_D wrote: post #193
nomilk wrote: post #77

Earlier quoted context omitted.

Could a motivated scraper not buy IPs/proxies that aren’t in those ranges, i.e. to blend in with general users?

Proxy IPs are also known and typically blocked. In fact, you can't even browse reddit without logging in when connected to most proxies. Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern…

Or, if you're on the shadier parts of the web, hacked IoT devices put to good use instead of DDoS zombies.
Post reply on HN