Live data from Hacker News

Why wordfreq will not be updated

github.com

201–210 of 542 posts

Re: Why wordfreq will not be updated

#201
post #4

I created https://lowbackgroundsteel.ai/ in 2023 as a place to gather references to unpolluted datasets. I'll add wordfreq. Please submit stuff to the Tumblr.

That's exactly the opposite of what the author wanted IMO. The author no more wants to be a part of this mess. Aggregating these sources would just makes it so much more easier for the tech giants to scrape more data.

Re: Why wordfreq will not be updated

#202
post #136

Earlier quoted context omitted.

Generative AI is inherently a political issue, its not surprising at all. There is the case of what is "truth". As soon as you start to ensure some quality of truth to what is generated, that is political. As soon as generative AI has the capability to take someone's job, that is political. The instant AI can make someone money, it is political. When AI is trained on something that someone has created, and now they c…

Then .. everything is political?

"Just because you do not take an interest in politics doesn't mean politics won't take an interest in you." -- Pericles

Re: Why wordfreq will not be updated

#203
I regret the situation led to the OP feel discourage about the NLP community, wo which I belong, and I just want to say "we're not all like that", even though it is a trend and we're close to peak hype (slightly past even?).

The complaint about pollution of the Web with artificial content is timely, and it's not even the first time due to spam farms intended to game PageRank, among other nonsense. This may just mean there is new value in hand-curated lists of high-quality Web sites (some people use the term "small Web").

Each generation of the Web needs techniques to overcome its particular generation of adversarial mechanisms, and the current Web stage is no exception.

When Eric Arthur Blair wrote 1984 (under his pen name "George Orwell"), he anticipated people consuming auto-generated content to keep the masses from away from critical thinking. This is now happening (he even anticipated auto-generated porn in the novel), but the technologies criticized can also be used for good, and that is what I try to do in my NLP research team. Good will prevail in the end.

Re: Why wordfreq will not be updated

#204

I really like the fact that the content of the conventional user content internet is becoming willfully polluted and ever more useless by the incessant influx of "ai"-garbage. At some point all of this will become so awful that nerds will create new and quiet corners of real people and real information while the idiot rabble has to use new and expensive tools peddled by scammy tech bros to handle the stench of automa…

> At some point all of this will become so awful that nerds will create new and quiet corners of real people and real information

It's already happening. There is a growing number of groups that are forming their own "private internets" that is separated from the internet-at-large, precisely because the internet at large is becoming increasingly useless for a whole lot of valuable things.

Re: Why wordfreq will not be updated

#205

Earlier quoted context omitted.

There are smaller, gated communities that are still very valuable. You're posting in one. But yes, the open Internet is basically useless now, thanks ultimately to advertising as a business model.

I've seen plenty of comments here that read like they've been generated by an LLM, if this is a gated community we need a better gate.

Sure, there's bad actors everywhere, but there's really no incentive to do it here so I don't think it's a problem in the same way it is on the open internet, where slop is actively rewarded.

Re: Why wordfreq will not be updated

#206

Earlier quoted context omitted.

Yeah pay an illustrator if this is important to you. See a lot of people upset about AI still using AI image generation because it's not in their field so they feel less strongly about it and can't create art themselves anyway, hypocritical either use it or don't but don't fuss over it then use it for something thats convenient for you.

I have updated my comment with "/s" as that is closer to what I've meant. However, seriously, from ethical point of view it is unlikely illustrators were asked or compensated for their work being used for training AI to produce the image.

I thought the header image was a symbol of AI slop contamination because it looked really off-putting

Re: Why wordfreq will not be updated

#207

"I don't think anyone has reliable information about post-2021 language usage by humans." We've been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now. Especially smaller children don't have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but…

> When I get asked if the person in a video is real, I still feel pretty confident to answer

I don't. I mean, I can identify the bad ones, sure, but how do I know I'm not getting fooled by the good ones?

Re: Why wordfreq will not be updated

#208
post #184

Earlier quoted context omitted.

I don't seem to get it.

To whomever downvoted parent: please don't act against people brave enough to state that they don't know something. This is a desired quality, increasingly less present in IT work environments. People afraid of being shamed for stating knowledge gaps are not the folks you want to work with.

I feel like there's a minimum "due diligence" bar to meet though before asking, otherwise it comes across as "I'm too lazy to google the reference and connect the dots myself, but can someone just go ahead and distill a nice summary for me"

Re: Why wordfreq will not be updated

#210

Earlier quoted context omitted.

I don't seem to get it.

Steel without nuclear contamination is sought after, and only available from pre-war / pre-atomic sources. The analogy is that data is now contaminated with AI like steel is now contaminated with nuclear fallout. https://en.wikipedia.org/wiki/Low-background_steel > Low-background steel, also known as pre-war steel[1] and pre-atomic steel,[2] is any steel produced prior to the detonation of the first nuclear bombs in…

> and only available from pre-war / pre-atomic sources.

From the same wiki you linked:

"Since the end of atmospheric nuclear testing, background radiation has decreased to very near natural levels, making special low-background steel no longer necessary for most radiation-sensitive uses, as brand-new steel now has a low enough radioactive signature"

and

"For the most demanding items even low-background steel can be too radioactive and other materials like high-purity copper may be used"

Post reply on HN