Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why wordfreq will not be updated

github.com

Post reply on HN
Print view View original

131–140 of 542 posts

Re: Why wordfreq will not be updated

#131

Post by eptcyka »

Freak_NL wrote: post #42
kevindamm wrote: post #32

Earlier quoted context omitted.

Yes but not quite as far as you imply. The training data is weighted by a quality metric, articles written by journalists and wikipedia contributors are given more weight than Aunt May's brownie recipe and corpoblogspam.

It certainly feels like the amount of regurgitated, nonsensical, generated content (nontent?) has risen spectacularly specifically in the past few years. 2021 sounds about right based on just my own experience, even though I can't point to any objective source backing that up.

I personally am yet to see this beyond some slop on youtube. And I am here for the AI meme videos. I recognize the dangers of this, all I am saying is that I don't feel the effect, yet.

Re: Why wordfreq will not be updated

#132

Post by [deleted] »

dsign wrote: post #53

Somehow related, paper books from before 2020 could be a valuable commodity in a in a decade or two, when the Internet will be full of slop and even contemporary paper books will be treated with suspicion. And there will be human talking heads posing as the authors of books written by very smart AIs. God, why are we doing this????

[deleted]

Re: Why wordfreq will not be updated

#133

Post by Sander_Marechal »

kdmccormick wrote: post #63
shortrounddev2 wrote: post #9

Man the AI folks really wrecked everything. Reminds me of when those scooter companies started just dumping their scooters everywhere without asking anybody if they wanted this.

At least scooters did something useful for the environment.

Did they? A lot of then were barely used, got damaged or vandalized, etc. And when the companies folded or communities outlawed the scooters, they end up as trash. I don't believe for a second that the amount of pollutants and greenhouse gasses saved by usage is larger than the amount produced by manufacturing, shipping and trashing all those scooters.

Re: Why wordfreq will not be updated

#134

Post by coldpie »

aucisson_masque wrote: post #18

Did we (the humans) somehow managed to pollute the internet so much with AI that's it's now barely usable ? In my opinion the internet can be considered as the equivalent of a natural environment like the earth. it's a space where people share, meet, talk, etc. I find it astonishing that after polluting our natural environment we know polluted the internet.

There are smaller, gated communities that are still very valuable. You're posting in one. But yes, the open Internet is basically useless now, thanks ultimately to advertising as a business model.

Re: Why wordfreq will not be updated

#135

Post by MPSimmons »

Lvl999Noob wrote: post #36
aucisson_masque wrote: post #24

It could be used to spot LLM generated text. compare the frequency of words to those used in human natural writings and you spot the computer from the human.

It could be used to differentiate LLM text from pre-LLM human text maybe. The thing, our AIs may not be very good at learning but our brains are. The more we use AI, the more we integrate LLMs and other tools into our life, the more their output will influence us. I believe there was a study (or a few anecdotes) where college papers checked for AI material were marked AI written even though they were written by human…

You're exactly right. You only have to look at the prevalence of the word "unalive" in real life contexts to find an example.

Re: Why wordfreq will not be updated

#136

Post by ilaksh »

ecshafer wrote: post #123
ilaksh wrote: post #116

Reading through this entire thread, I suspect that somehow generative AI actually became a political issue. Polarized politics is like a vortex sucking all kinds of unrelated things in. In case that doesn't get my comment completely buried, I will go ahead and say honestly that even though "AI slop" and paywalled content is a problem, I don't think that generative AI in itself is a negative at all. And I also think t…

Generative AI is inherently a political issue, its not surprising at all. There is the case of what is "truth". As soon as you start to ensure some quality of truth to what is generated, that is political. As soon as generative AI has the capability to take someone's job, that is political. The instant AI can make someone money, it is political. When AI is trained on something that someone has created, and now they c…

Then .. everything is political?

Re: Why wordfreq will not be updated

#137

Post by karaterobot »

I guess a manageable, still-useful alternative would be to curate a whitelist of sources that don't use AI, and without making that list public, derive the word frequencies from only those sources. How to compile that list is left as an exercise for the reader. The result would not be as accurate as a broad sample of the web, but in a world where it's impossible to trust a broad sample of the web, it the option you are left with. And I have no reason to doubt that it could be done at a useful scale.

I'm sure this has occurred to them already. Apart from the near-impossibility of continuing the task in the same way they've always done it, it seems like the other reason they're not updating wordfreq is to stick a thumb in the eye of OpenAI and Google. While I appreciate the sentiment, I recognize that those corporations' eyes will never be sufficiently thumbed to satisfy anybody, so I would not let that anger change the course of my life's work, personally.

Re: Why wordfreq will not be updated

#138

Post by user432678 »

dsign wrote: post #53

Somehow related, paper books from before 2020 could be a valuable commodity in a in a decade or two, when the Internet will be full of slop and even contemporary paper books will be treated with suspicion. And there will be human talking heads posing as the authors of books written by very smart AIs. God, why are we doing this????

And I thought I had some kind of mental illness collecting all those books, barely reading them. Need to do that more now.

Re: Why wordfreq will not be updated

#139

Post by joshdavham »

PeterStuer wrote: post #100

Intuitively I feel like word frequency would be one of the things least impacted by LLM output, no?

Think of an LLM as a person on the internet. Just like everyone else, they have their own vocabulary and preferred way of talking which means they’ll use some words more than others. Now imagine we duplicate this hypothetical person an incredible amount of times and have their clones chatter on the internet frequently. ‘Certainly’ this would have an effect.

Re: Why wordfreq will not be updated

#140

Post by epgui »

jsheard wrote: post #80
kevindamm wrote: post #32

Earlier quoted context omitted.

Yes but not quite as far as you imply. The training data is weighted by a quality metric, articles written by journalists and wikipedia contributors are given more weight than Aunt May's brownie recipe and corpoblogspam.

> The training data is weighted by a quality metric At least in Googles case, they're having so much difficulty keeping AI slop out of their search results that I don't have much faith in their ability to give it an appropriately low training weight. They're not even filtering the comically low-hanging fruit like those YouTube channels which post a new "product review" every 10 minutes, with an AI generated thumbnail…

I don’t think they were talking about the quality of Google search results. I believe they were talking about how the data was processed by the wordfreq project.
Post reply on HN