Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google is the only search engine that works on Reddit now, thanks to AI deal

404media.co

Post reply on HN
Print view View original

311–320 of 379 posts

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#311

Post by sssilver »

tedivm wrote: post #210
chippiewill wrote: post #90

Earlier quoted context omitted.

They changed it to disallow so that scrapers can't just claim the robots.txt gave them permission.

According to the US court systems the robots.txt file is meaningless. If they respond with a 200 status code giving you the access then you can legally scrape it all you want. If they require that you log in then you have to follow the terms you agree to when creating an account. Public means public though, and if Reddit doesn't want to make the content private (put it behind a login) then we can scrape away. Note th…

Fascinating. Where can one learn more about this?

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#312

Post by latexr »

immibis wrote: post #78
popcalc wrote: post #5

# Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content. # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use. # pol…

Nobody who wants to be successful obeys robots.txt. And I do mean nobody.

That’s a weird statement to be absolutist about. The majority of individuals and companies who want to be successful do not do so by scrapping websites, thus have no reason to disobey robots.txt. Most people in the world, ambitious or not, wouldn’t even understand what your sentence refers to.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#313

Post by latexr »

TeaBrain wrote: post #105
BeetleB wrote: post #93

Earlier quoted context omitted.

I know people will hate to hear this, but Reddit it's not important to the A well being of the Internet.

I think it's the other way around, in that people don't like to hear how Reddit has become important due to the death of independent forums and the degree to which information has become concentrated on the site.

Independent forums, like RSS, are not dead. I use both every day.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#314

Post by pas »

throwaway290 wrote: post #260
pas wrote: post #197

Earlier quoted context omitted.

it's even worse. it's not theirs (it's the users'), they are merely hosting it and using it (ToS gives them a fancy irrevocable license I guess). so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregatio…

actually owners/authors like me would not want our stuff crawlable because that gives up our ownership. When I am answering some random dude on reddit with a problem I want that dude to read my solution. I don't want this to be crawled and forever stored (probably deanonymized) or enshrined in a dozen commercial LLMs. There is substack for that stuff.

have you heard about DMs?

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#315

Post by latexr »

talldayo wrote: post #114
JohnMakin wrote: post #40

Earlier quoted context omitted.

This is a false dichotomy. You can have services, and not have them devolve into complete unusability in the name of profit. This isn’t sustainable either. The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, no matter how much you believe in this weird frog-boil internet we’ve inherited now.

> The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, The myopic pursuit of short-term gains is the only playbook that works. Long-term business strategy is a gamble, and today's businesses have all learned that they'd rather make hay when the sun is shining than be remembered as a good business. Twitter tried a long-term playbook to reverse their unprofitab…

> Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website.

From what I recall reading at the time, Twitter was finally becoming profitable before the sale (last two years? It’s hard to find a source now since every story since is about some shit show or other post sale).

> That ended up with them being undervalued and sold to the highest bidder.

You make it seem they were in dire straits and had to be sold for scraps, but that’s far from the case. They sold for more than their valuation to the only bidder because they understood what a good deal it was for them. They forced the buyer to not back out, after all.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#316

Post by latexr »

imglorp wrote: post #209
roughly wrote: post #86

Boy, the LLMs have really been an apocalypse moment for the web, haven’t they? Between hoovering up and monetizing every bit of content they can without any attribution or compensation and the absolute flood of mediocre generated content, they’ve really done in the last straggling remains of the open internet. It’s not like everyone wasn’t already pulling the same grift, but quantity really does have a quality all it…

Of course, we have to be careful not to villainize a neutral tech. Instead let's call it what it is: unchecked capitalism and monopolistic behaviors. Capitalism seems to work ok for the common good until you remove all the protections. LLMs provide a defacto monopoly for the owner which must already be a near monopoly: they take vast resources to train; only a giant corp can afford to buy all the content and provisio…

On the one hand, you’re absolutely right. But on the other hand it’s not like it matters in practice. Isn’t most technology technically neutral? But it’s also made to be used by people, who can do so beneficially or detrimentally. Criticising a technology is a shorthand for criticising how it’s used.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#317

Post by latexr »

synicalx wrote: post #276
imglorp wrote: post #209

Earlier quoted context omitted.

Of course, we have to be careful not to villainize a neutral tech. Instead let's call it what it is: unchecked capitalism and monopolistic behaviors. Capitalism seems to work ok for the common good until you remove all the protections. LLMs provide a defacto monopoly for the owner which must already be a near monopoly: they take vast resources to train; only a giant corp can afford to buy all the content and provisio…

> Of course, we have to be careful not to villainize a neutral tech This is a very good point IMO. If we're going to chastise LLM's we may as well give servers, switches, routers, fiber-optic cable, and silicone a bollocking as well since that's ultimately what's facilitating all this.

No, those are not comparable. If someone criticises the electric chair, it’s not reasonable to defend it with “if we’re going to chastise the electric chair we may as well give wood, metal, chairs, and electricity a bollocking since that's ultimately what's facilitating all this”.

Things are more than the sum of their parts. If you have a ton of beneficial things which can be cobbled together into one bad detrimental thing, the existence of the latter does not remove the benefits of the former.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#318

Post by account42 »

shit_game wrote: post #272
dogleash wrote: post #116

Earlier quoted context omitted.

> # Reddit believes in an open internet, but not the misuse of public content. Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.

Their license/Eula clearly state that Reddit has perpetual whatever to content posted on Reddit, but relying solely on DMCA for "stolen" content _yet again_ feels like a terrible way to deal with non-original content. Part of me hopes that Reddit gets hit with some new precidence-setting lawsuits regarding non-original content that requires useful attribution, but I double t that will ever happen.

An EULA does not change the morality of the situation anyway. They are a leech profiting off users generating content who are now upset about not getting a cut from third-parties also profiting from said user generated content.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#319

Post by account42 »

throwaway4pp24 wrote: post #293
whycome wrote: post #284

Earlier quoted context omitted.

Scrapings laundering. Do we have a term for this?

Yes, right in the law - "fair use"

Even more basic, it's free speech. The data itself is public domain so your free speech is not restricted and you don't need fair use excemptions for those restrictions. On the the access through the official system is restricted.

Re: Google is the only search engine that works on Reddit now, thanks to AI deal

#320

Post by throwaway290 »

pas wrote: post #314
throwaway290 wrote: post #260

Earlier quoted context omitted.

actually owners/authors like me would not want our stuff crawlable because that gives up our ownership. When I am answering some random dude on reddit with a problem I want that dude to read my solution. I don't want this to be crawled and forever stored (probably deanonymized) or enshrined in a dozen commercial LLMs. There is substack for that stuff.

have you heard about DMs?

That's why more and more exchanges are moving to DMs and closed communities. Because we don't plan on stuff we intend for participants in the discussion to be harvested but it more and more is. If reddit breaks that trend I only welcome it.
Post reply on HN