Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Certified 100% AI-free organic content

substack.piszek.com

Post reply on HN
Print view View original

271–275 of 275 posts

Re: Certified 100% AI-free organic content

#271

Post by rcme »

CuriouslyC wrote: post #269
rcme wrote: post #267

Earlier quoted context omitted.

Even if each dataset is biased, I’m not still not sure how you derived logarithmic growth from the general notion of bias in data. For instance, assuming the data is biased, perhaps it is biased in the other direction and contains more common patterns compared to the underlying distribution.

There is a lot to this subject, it might be easier if you took a look at https://martinapugliese.github.io/data/heaps-law-languages/ . Note that when plotting corpus size vs unique words, the log plot is expected to be linear.

Ah, I see what you mean: the number of unique examples increases logarithmicly with data size, which kind of makes sense. Language, in this case, follows a power law.

I think you argument is that this means smaller datasets are ok because they contain "most" of what the larger datasets contains. But I think this data-power-rule implies the opposite. ML models can often get to 80-90% accuracy on some task. Unfortunately, these models often aren't that useful because that missing 10% of accuracy matters a lot to users. So what this data-power-rule implies is that, in order to get the last 10% of gains, you need 10x the amount of data.

Re: Certified 100% AI-free organic content

#272

Post by CuriouslyC »

rcme wrote: post #271
CuriouslyC wrote: post #269

Earlier quoted context omitted.

There is a lot to this subject, it might be easier if you took a look at https://martinapugliese.github.io/data/heaps-law-languages/ . Note that when plotting corpus size vs unique words, the log plot is expected to be linear.

Ah, I see what you mean: the number of unique examples increases logarithmicly with data size, which kind of makes sense. Language, in this case, follows a power law. I think you argument is that this means smaller datasets are ok because they contain "most" of what the larger datasets contains. But I think this data-power-rule implies the opposite. ML models can often get to 80-90% accuracy on some task. Unfortunate…

Well, to get back to my original point, if we're trying to improve the quality and accuracy of model writing, and we want to do that by adding quality and accuracy scores to short token sequences, the power law distribution means we could get coverage on a significant portion of the data set by scoring just the most frequent sequences that aren't linguistic trivia. We could probably get to 50% average coverage fairly cheaply, and while diminishing returns would kick in and make getting to 80 or 90% much more expensive, at that point we could use a model to estimate the remainder, and have a perfectly suitable quality/accuracy scores to condition the model on. The model would output those quality/accuracy scores for the generated token sequence as well, so portions of output that were low quality/of questionable accuracy could be flagged.

Re: Certified 100% AI-free organic content

#273

Post by CuriouslyC »

joshspankit wrote: post #248
CuriouslyC wrote: post #62

Earlier quoted context omitted.

This isn't an issue, because it's possible to add prose quality and content accuracy scores to training data and train the model to predict those quantities during generation, which would allow you to condition the generation on high prose quality/accuracy. It just requires a small update to the model, and a shit ton of data set annotation time. Likewise, images can be scored for aesthetics and consistency and models…

Then you’d just create the AI equivalent of black-hat SEO.

An arms race between generative AI and discriminative AI is a good thing, it'll lead to higher quality models since the only way to "win" the game is to make output that is indistinguishable from legitimate content.

In fact, that is basically the GAN algorithm in meta form.

Re: Certified 100% AI-free organic content

#274

Post by r3trohack3r »

joshspankit wrote: post #246
r3trohack3r wrote: post #25

Earlier quoted context omitted.

I feel like we are nearing peak "uncurated" content, both for humans and machines. Humans are still grappling with our novel abundance problems. As we move forward, suspect we will see an increase in curation services and AI models will do more with less. You can bootstrap a productive adult human on an almost infintismal slice of the training sets we are using for the current gen of AI, can't imagine future approach…

We’d have to adjust capitalism to deal with the “novel abundance” problems. Most of the drive for novel content/audiences is simply to decide which people get a cut of the revenue (and/or audience). If we focused on quality and stopped caring about who gets paid for what I suspect that not only would we have better quality overall but we’d also push the boundaries much faster thus making things even more interesting.

Strong disagree. Capitalism (can argue whether that's what we have or not) uses capital to respond to markets. Markets are signals of what people want.

The scarcity mindset in an the era of information abundance isn't being perpetuated by capitalism, it's being perpetuated by people making their own decisions one transaction at a time.

Re: Certified 100% AI-free organic content

#275

Post by joshspankit »

r3trohack3r wrote: post #274
joshspankit wrote: post #246

Earlier quoted context omitted.

We’d have to adjust capitalism to deal with the “novel abundance” problems. Most of the drive for novel content/audiences is simply to decide which people get a cut of the revenue (and/or audience). If we focused on quality and stopped caring about who gets paid for what I suspect that not only would we have better quality overall but we’d also push the boundaries much faster thus making things even more interesting.

Strong disagree. Capitalism (can argue whether that's what we have or not) uses capital to respond to markets. Markets are signals of what people want. The scarcity mindset in an the era of information abundance isn't being perpetuated by capitalism, it's being perpetuated by people making their own decisions one transaction at a time.

You make some strong logical cases so I'd like to use some of your language to try to give another perspective on my points:

Markets in our current system are (IMO) a distorted view of what people want. For example TVs: How much of the market is people who simply want a nice clean picture and how much is people who are feature-chasing based on the TV marketing? And then how much of the 'market' is actually smoke and mirrors accounting to increase the value of companies?

Some things that I think are only common because of capitalism (or "capitalism") and could therefore be phased out if we transitioned to an AI-supported system:

- "Keeping up with the Jones'"

- Needing short hits of artificial 'excitement' (such as the kind of novelty drip-fed to us by the movie industry)

- Keeping ideas secret as a form of Intellectual Property protection, and therefore the scarcity mindset that there are no good solutions and no new problems to solve (since we'd all be exposed to the latest of those if we chose)

- Capital being used for poor-quality projects (since many of these again are just about trying to grab more money for the people making them)

- People making their own decisions one transaction at a time. This is definitely underrated. Sure some people will continue to think short-term just because that's how their brains are wired, but I suspect the majority will find themselves able to breathe slowly for the first time and they will naturally unwind in to longer and longer-term thinking.

Post reply on HN