Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reddit is OpenAI’s moat

cyberdemon.org

Post reply on HN
Print view View original

81–90 of 319 posts

Re: Reddit is OpenAI’s moat

#81

Post by LittleShaman »

meowface wrote: post #12

I'm sure I'm missing something, but: aren't there publicly available corpuses of all reddit posts up to a certain date? Why wouldn't researchers train with these? Are they just not recent enough? Even if they aren't very recent, how big of a deal is that when it comes to training models that presumably use lots of other data sources as well?

The author is aware of that, mentions it towards the end:

> Most of Reddit’s current data has been scraped anyway, so the game is to protect Reddit’s data going forward.

But yes, Pushshift archives of all posts and comments until the recent ban [1] are freely available for download [2]

[1] https://old.reddit.com/r/pushshift/comments/135tdl2/a_respon... The ban was followed by allowing the parent non-profit of Pushshift (Network Contagion Research Institute) to use the API provided access is restricted to a use-case Reddit has care for: mod tools. Reddit hasnt replaced those with its own just yet. The rest of us are shut out.

[2] https://academictorrents.com/userdetails.php?id=9863

Re: Reddit is OpenAI’s moat

#82

Post by csdvrx »

toomuchtodo wrote: post #35
csdvrx wrote: post #31

Earlier quoted context omitted.

"Let's now show the new Bing, where high quality data from reddit is better ranked and the selected by default for {whatever_category_you_think_matters} thanks to automatic classification of the search questions by GPT" There are many synergies that aren't properly considered by a naive take. I use site:reddit.com with most of my queries. I now prefer bing to google. Bing market share is growing. Coughing up a few B…

Github acquisition was a savvy, Reddit has more peril involved but similar value.

I'm in full agreement, but I see an even better integration in other product lines (mostly gaming and search)

Also, Microsoft doesn't have a generic social network to mine data from.

They may prefer to stick to the professional world, but then acquiring a few high value sites just to close their API would be another move.

How much $$$ do you think it would take for YC to say "hell yes!" to put HN under Microsoft control?

How valuable would it be to add a few conditions like a very generous giveaway of azure credits to HN companies in return of exclusivity (no more S3 or google cloud for the new batch!)

Re: Reddit is OpenAI’s moat

#83

Post by mguerville »

refulgentis wrote: post #61
zdragnar wrote: post #28

Earlier quoted context omitted.

I genuinely don't understand the appeal of AI for search. The provenance of information is just as important as the resulting information for pretty much anything I search for. I almost never accept a single search result as authoritative unless I'm pretty familiar with the source. Reddit in particular seems like a terrible set of training data. Pretty much any opinion is going to have a counter-opinion somewhere in…

Simple: get the search results, have LLM read and evaluate them. It's devilishly difficult to get citations in there, you're not going to do it with langchain, but its possible (c.f. Bing). Python x langchain x LLMs makes it very easy to create demos so there's been an initial influx of meh stuff, I'm very excited for 6 months from now.

A niche provider that doesn't do generative AI very well but does "AI powered search with citations" is perplexity.ai

I've used for rather obscure queries and liked the summary the AI wrote as well as the links to dive deeper. I imagine that's what AI search will look like across all providers before long.

Re: Reddit is OpenAI’s moat

#84

Post by kajaktum »

They can still scrap the website?

The fact is that many, many internet businesses today not make sense. Many of them are bleeding money with no possible path to profitability ahead. Mostly due to the fact that people's expectations are very different and I would argue even skewed. Clearly hosting Reddit is expensive, but who really pays for it? People don't like ads, don't like to pay for services, don't like to be reminded that they can provide donations (Wikipedia). What else is there? Hoping the customers have the suddenly conscience to fund the product?

I foresee many internet companies dying over the next few years with no real replacement to crop up. But people always forget and history repeats itself; so maybe we will just see an endless reincarnations of similarly failed products.

Re: Reddit is OpenAI’s moat

#85

Post by larperdoodle »

zdragnar wrote: post #28
JimtheCoder wrote: post #9

I may be in the minority here, but if I want the opinion of Redditors on an issue, I will use a search engine to look for it specifically, thus knowing the provenance of the information I am receiving. I don't really want the corpus of Reddit data influencing the output of a generative AI model...because it's Reddit, after all... Even though I am pretty sure it is already included in the training dataset already... I…

I genuinely don't understand the appeal of AI for search. The provenance of information is just as important as the resulting information for pretty much anything I search for. I almost never accept a single search result as authoritative unless I'm pretty familiar with the source. Reddit in particular seems like a terrible set of training data. Pretty much any opinion is going to have a counter-opinion somewhere in…

When you're not sure what to Google (such as when working in an unfamiliar domain), asking ChatGPT for help and then researching it's answer with normal searches is very effective.

Re: Reddit is OpenAI’s moat

#86

Post by rjh29 »

meowface wrote: post #12

I'm sure I'm missing something, but: aren't there publicly available corpuses of all reddit posts up to a certain date? Why wouldn't researchers train with these? Are they just not recent enough? Even if they aren't very recent, how big of a deal is that when it comes to training models that presumably use lots of other data sources as well?

Reddit's API change will not stop LLM training, it is far too easy to manually scrape the site. It only needs to be done once.

The intention of the API change is to kill third-party app users who cannot be as effectively monetized as first-party app users.

Re: Reddit is OpenAI’s moat

#87

Post by flandre »

rjh29 wrote: post #76
duxup wrote: post #64

> There is no question that Reddit is extremely valuable as training data. How often do you append “reddit” to your searches? Is it? When I worked in networking, and later web dev work I found Reddit to be a TERRIBLE place for Q & A type situations. Answers on Reddit are often skewed by truthy answers from people with limited perspective in the industry who are surprisingly sort of militant about a given topic. For e…

Yes. For anything from opinions of movies/games to DIY advice to nerdy stuff like the best watch to buy in a given price range or the best synthesiser. Or people's opinions on a particular episode of TV! Or basically any hobby. It's not great for business stuff because there are way more people using e.g. AWS for their hobby project than running multi-million $ businesses on it. What are the alternatives? Facebook an…

My favorite alternatives are email and going outside.

Re: Reddit is OpenAI’s moat

#88

Post by skilled »

ltadeut wrote: post #70
skilled wrote: post #53

Earlier quoted context omitted.

I agree with you but not in entirety. This comment from spez[0] about blaming the API price changes on LLM's is too far fetched. A lot of commenters here on HN have already pointed that out already too. Unless they build a literal brick wall (paywall) around the site, that data can and will get scraped if the intention is to use for a model. You could get it down to a science where you only scrape any new data whenev…

> , that data can and will get scraped if the intention is to use for a model. How would that work from a legal perspective, though? Let's say there's no paywall and Reddit's terms of use disallow unauthorized commercial use of their data. Wouldn't that be a violation of Reddit's terms and liable to some legal procedure?

Interesting question but sadly I am in no position to answer it.

I think there are probably issues to address with scraping it blindly:

- Can Reddit imprint its data somehow? A watermark?

- Can Reddit prove that certain type of information appeared on Reddit first and thus that serves as proof its data was used without authorization?

If OpenAI can't work around this, I'm not sure they would be willing to cross any lines in terms of copyright, they've already done it with ChatGPT and I am guessing rules are only going to get stricter on this topic.

Re: Reddit is OpenAI’s moat

#89

Post by waihtis »

duxup wrote: post #64

> There is no question that Reddit is extremely valuable as training data. How often do you append “reddit” to your searches? Is it? When I worked in networking, and later web dev work I found Reddit to be a TERRIBLE place for Q & A type situations. Answers on Reddit are often skewed by truthy answers from people with limited perspective in the industry who are surprisingly sort of militant about a given topic. For e…

not only that but 80% of conversations on reddit is idiotic jokes and puns

Re: Reddit is OpenAI’s moat

#90

Post by VMG »

whywhywhywhy wrote: post #63
zdragnar wrote: post #28

Earlier quoted context omitted.

I genuinely don't understand the appeal of AI for search. The provenance of information is just as important as the resulting information for pretty much anything I search for. I almost never accept a single search result as authoritative unless I'm pretty familiar with the source. Reddit in particular seems like a terrible set of training data. Pretty much any opinion is going to have a counter-opinion somewhere in…

> I genuinely don't understand the appeal of AI for search If you're good at googling the flow is: Ask the question > Clock which result isnt spam and click it > Figure out how to dismiss the cookies gate without accepting the cookies > Dismiss the google login box > Dismiss the popover pushing you to install an app > Scroll the page or ctrl+F to find the answer With ChatGPT it's just type your question and your answ…

painful how accurate that is
Post reply on HN