Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google cut a deal with Reddit for AI training data

theverge.com

Post reply on HN
Print view View original

111–120 of 154 posts

Re: Google cut a deal with Reddit for AI training data

#111

Post by gruez »

lolpanda wrote: post #106

Reddit blocks all search engines from crawling their comments, essentially blocking all user generated contents. See this directive in https://www.reddit.com/robots.txt `Disallow: /r/ /comments/ / / /*` But I can still search Reddit on Google. How does Google manage to get the data?

I think you're misinterpreting the rule.

The relevant robots.txt rules are:

    User-Agent: *
    Disallow: */comment/*
    Disallow: /r/*/comments/*/*/*/*
The url for the comment sections looks like: https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/. This doesn't match the above rules because there's only 2 parts after "comments", not 4 parts as specified by the rules.

Re: Google cut a deal with Reddit for AI training data

#112

Post by BadHumans »

rebelde wrote: post #85

Google is paying Reddit instead of just taking it for free like they do from all other websites?

Second line in the article: "In an update on Thursday, Reddit announced it will start providing Google “more efficient ways to train models.”

They will helping Google create a real-time pipeline to Reddit's data.

Re: Google cut a deal with Reddit for AI training data

#113

Post by jiveturkey42 »

kromem wrote: post #83
jiveturkey42 wrote: post #15

AI is about to become really sarcastic, pedantic, and absurdly moralistic

Every LLM almost certainly already is trained on Reddit from before 2015 when the entire site content was compiled for research. This is just adding in more recent content.

> makes a banter

> gets a pedantic reply

were you also trained on reddit?

Re: Google cut a deal with Reddit for AI training data

#114

Post by mike_hearn »

Scoundreller wrote: post #49
mvdtnz wrote: post #44

Earlier quoted context omitted.

And even more confidently wrong.

Can’t wait for heavily downvoted unfortunate truths to be eliminated from AI

That already happens. Standard academic datasets that use Reddit start by eliminating any comments that are under +3, for example.

Re: Google cut a deal with Reddit for AI training data

#117

Post by klipklop »

What incentive is there for posters on Reddit to continue to produce new content that is sold off for a profit? Sure they agreed to it in the TOS, but I suspect people might leave the platform.

Also how many of the posts are already generated by AI? Seems like a raw deal to pay for data that could be a large percentage of bot activity.

Why visit Reddit at all for information when you can just ask Gemini.

Re: Google cut a deal with Reddit for AI training data

#118

Post by Scoundreller »

declaredapple wrote: post #90
Scoundreller wrote: post #48

Earlier quoted context omitted.

> They've never sold it before. Do we know this?

I'm pretty confident they haven't until at least 6 months ago when they freaked out about api usage. It's possible they've sold it and we haven't heard about it since then. I would think they'd want to brag about it publicly though.

I figure in this case, Google was the one that wanted to be public about it (to appear as relevant when AI is the hot thing). I could totally see other orgs happy to keep their data-source private.

Re: Google cut a deal with Reddit for AI training data

#119

Post by mlekoszek »

LorenDB wrote: post #3

Hm, now I have to wonder if it'd be possible to create a tool that poisons your Reddit posts so they look fine to a human but completely trash LLM output. I'm not saying we should do it, but it's a fascinating thought.

What about Markov chain babble? It would be interesting to see Bard learn to mimic people going far even as decided to use even go want to do look more like.

Re: Google cut a deal with Reddit for AI training data

#120

Post by klipklop »

latentcall wrote: post #115
Mo3 wrote: post #109

Just a reminder this exists: https://github.com/j0be/PowerDeleteSuite https://github.com/andrewbanchich/shreddit

This worked amazingly for me when I stopped using Reddit.

Who knows if the "deleted" data can still be sold off. Sure you can't view it on the website, but nobody knows what Reddit will hand to Google. Then again certain laws passed recently might make such a thing illegal...
Post reply on HN