Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reddit is OpenAI’s moat

cyberdemon.org

Post reply on HN
Print view View original

221–230 of 319 posts

Re: Reddit is OpenAI’s moat

#221

Post by haldujai »

refulgentis wrote: post #173
bbor wrote: post #118

Earlier quoted context omitted.

This is a great analysis that really cleared up the impact of that memo for that me. You clearly know but for any readers who aren’t clear: that memo is the opinion of one engineer at google, and far from the (apparent) opinion of the relevant execs

It’s dead wrong, I suggest reading the original. _No one_ thinks models trained on more data would be about the same, since everything flows out of that premise…then throws Current Thing on top…it’s very unhelpful. The “patches” he refers to are LoRA and are treated as deus ex machina. They’re not, ex. playing with Stable Diffusion we can see they’re additive but they’re not nearly as good as training the original mo…

> they’re not nearly as good as training the original model on the data.

How are you defining "nearly as good", can you be more specific?

It's obvious full fine-tuning > [PEFT tuning] but to my understanding the gap isn't that significant, as reported in various papers. (specifically with respects to language models, I'm not familiar with diffusion models).

Re: Reddit is OpenAI’s moat

#222

Post by dmazin »

nostrademons wrote: post #62

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. The remaining misses are facts tha…

> you're ingesting expert knowledge that's highly specific and only discussed in a few forums. Reddit data could perhaps help with the latter

This is exactly where I think Reddit's value lies. I disagree, though, that people don't go to Reddit for it. Here are a couple recent queries where I appended reddit to my search query: * best places to visit from London * best mattress sold in UK * should I bring king mattress from US to UK

(Can you tell I just moved to London?)

Why do people go to Reddit? Because it's guaranteed (at least for now) that the answer came from humans who are not trying to sell you something. They may be wrong but... it's opinions.

Re: Reddit is OpenAI’s moat

#223

Post by jjoonathan »

mathisfun123 wrote: post #204
nostrademons wrote: post #62

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. The remaining misses are facts tha…

>most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. more than anything, AI/ML/DL enables people to make claims that cannot be refuted by any application of the scientific method. it's literally "not even wrong" in exactly the same way as people claim string theory is.

String theory wishes it had (unique) claims as easily falsifiable as "Once you have ingested several terabytes of natural language, you know how to generate natural language."

Obviously you need a page of fine print to make a claim strictly falsifiable, but complaining about the absence of fine print in casual discussion is absurdly uncharitable unless you have reason to believe that agreeable fine print couldn't be drawn up and I'm 99% sure that in this case you have so such reason.

Re: Reddit is OpenAI’s moat

#224

Post by enterprise_cog »

tmpz22 wrote: post #145
nostrademons wrote: post #62

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. The remaining misses are facts tha…

> Reddit is usually not the place to go for expert discourse Its not expert research but Reddit can be used to find probably-real-first-hand-experience on a given subject. Which is enough to prefix "reddit" to a lot of google searches. It just needs to be a marginal improvement to Google blogspam for it to have some intrinsic value.

It used to. Past 7ish years or so I saw a lot more posting that was straight up lies or people pretending to be experts that wasn’t heavily downvoted. Or maybe it was always like that and I got more aware. Either way, not something I’d want ingested into a model.

Re: Reddit is OpenAI’s moat

#225

Post by voytec »

> This blew up on Hacker News. An HN mod edited the title to add a ? at the end. It was not me. The answer to any headline with a ? at the end is “no.” I would not own myself like that.

Agree with the viewpoint. Hacker News is heavily moderated in a non-user-friendly way which results in some comments being nonsensical after moderation editorials and also disturbs searching for content by a title one remembers.

Re: Reddit is OpenAI’s moat

#226

Post by ok123456 »

rchaud wrote: post #136
ok123456 wrote: post #46

Earlier quoted context omitted.

The appeal is that now regular search engines are so bad at giving you useful content that using a LLM is now the equivalent of "google-dorking" to find relevant information.

Only because the LLM didn't have any AI generated blogspam to get trained on. That's going to change very quickly.

Any non-trival LLM that works by scraping the internet is already sufficiently advanced to be able to classify blogspam.

Re: Reddit is OpenAI’s moat

#227

Post by winddude »

>If Reddit merely wanted to restrict the ability to scrape its data, they could have done so without killing off clients – e.g. via licensing deals[1].

They haven't taken any steps to stop scrapping. They made access to the data via api extremely expensive. Calling an API is not crawling/scrapping. You can still crawl/scrape, well once/if the mod protest is over. And also stop calling it scrapping, it web crawling.

I also actually wonder about the validity of it as training data. I've done a few experiments with fine tuning models, with a few hundred thousand samples curated from hundreds of millions of threads https://huggingface.co/winddude/pb_lora_7b_v0.1>. they are interesting, because they end up being so sarcastic. People tend to either be short, or overly opinionated.

At the very least someone would have to do a lot of pre-processing, which would make it a transfomative work anyways.

Re: Reddit is OpenAI’s moat

#228

Post by cma »

ltadeut wrote: post #70
skilled wrote: post #53

Earlier quoted context omitted.

I agree with you but not in entirety. This comment from spez[0] about blaming the API price changes on LLM's is too far fetched. A lot of commenters here on HN have already pointed that out already too. Unless they build a literal brick wall (paywall) around the site, that data can and will get scraped if the intention is to use for a model. You could get it down to a science where you only scrape any new data whenev…

> , that data can and will get scraped if the intention is to use for a model. How would that work from a legal perspective, though? Let's say there's no paywall and Reddit's terms of use disallow unauthorized commercial use of their data. Wouldn't that be a violation of Reddit's terms and liable to some legal procedure?

Reddit don't own the copyright to it, just a license. That plus public web scraping is legal. Reproducing the data directly might violate the user's copyrights, but through an LLM it is assumed not.

Re: Reddit is OpenAI’s moat

#229

Post by haldujai »

hospitalJail wrote: post #200
nostrademons wrote: post #62

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. The remaining misses are facts tha…

>You'd usually be better off training on trade publications, scientific journals, or fandom than Reddit. Bold claim. There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals. That is what is missing. The absurdly specific stuff that reddit gets. Sure you can answer with nonsense that sounds realistic, but you could also answer with the exact text that sol…

> There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals.

There are also plenty of people answering nuanced questions with complete BS.

It's not that bold to claim that as when taken as a whole the signal-noise is worse on Reddit than in a publication.

Re: Reddit is OpenAI’s moat

#230

Post by dmonitor »

hospitalJail wrote: post #200
nostrademons wrote: post #62

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. The remaining misses are facts tha…

>You'd usually be better off training on trade publications, scientific journals, or fandom than Reddit. Bold claim. There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals. That is what is missing. The absurdly specific stuff that reddit gets. Sure you can answer with nonsense that sounds realistic, but you could also answer with the exact text that sol…

The replies of engineers and doctors are significantly outweighed by the responses of bored teenagers.
Post reply on HN