Live data from Hacker News

Why wordfreq will not be updated

github.com

471–480 of 542 posts

Re: Why wordfreq will not be updated

#471

Intuitively I feel like word frequency would be one of the things least impacted by LLM output, no?

If only we had a data set that measured word frequency across the internet as we're getting more and more into AI being used... Maybe with a baseline from before 2021 for comparison... But no let's just stop measuring word frequency entirely because we can just assume what will happen and we're angry.

Re: Why wordfreq will not be updated

#472

Earlier quoted context omitted.

glances nervously at atmospheric CO2

The earth will recover. We may not, but earth will.

And in a few million years, the next intelligent life form will examine remains of human texts, and wonder: with all the tools and knowledge they possessed, how could they not have prevented their demise?

(Sorry for pessimism and offtopicism)

Re: Why wordfreq will not be updated

#473
post #460

Earlier quoted context omitted.

Human me just thought it was a good word for this. It implies some irreversible process of mixing, I think that characterizes this process really well.

There were dozens of 20th Century ideological movements which developed their own forms of "Newspeak" in their own native languages. Largely, natural human dialog between native speakers and between those opposed to the prevailing regime recoils violently at stilted, official, or just "uncool" usages in daily vernacular. So I wouldn't be too surprised to see a sharp downtick in the popular use of any word that become…

humans also lag humans, the future may already be spoken, but the slang is not evenly memed out yet.

Re: Why wordfreq will not be updated

#474
post #225

One of the examples is the increased usage of "delve" which Google Trends confirms increased in usage since 2022 (initial ChatGPT release): https://trends.google.com/trends/explore?date=all&q=delve&hl... It seems however it started increasing most in usage just these last few months, maybe people are talking more about "delve" specifically because of the increase in usage? A usage recursion of some sorts.

We’ve seen this with a couple words and expressions, and I don’t doubt that AI is somewhat likely to “like” some phrases for whatever reason. Big eigenvaues of the latent space or whatever, hahaha (I don’t know AI). But also, words and phrases do become popular among humans, right? It would be a shame if AI caused the language to get more stagnant, as keeping up with which phrases are popular get you labeled as an AI…

Exactly, like how "mindful" and "demure" recently became more popular for seemingly no reason. Humans do this all the time.

And language in general stagnates and shrinks in vocabulary over time ( https://evoandproud.blogspot.com/2019/09/why-is-vocabulary-s... ). (Link that ChatGPT helped me find :P) I think AI will increase the average persons vocabulary, since it appears to in general be better/more professionally written than a lot of what the average person is exposed to online.

Re: Why wordfreq will not be updated

#475

Earlier quoted context omitted.

It certainly feels like the amount of regurgitated, nonsensical, generated content (nontent?) has risen spectacularly specifically in the past few years. 2021 sounds about right based on just my own experience, even though I can't point to any objective source backing that up.

I personally am yet to see this beyond some slop on youtube. And I am here for the AI meme videos. I recognize the dangers of this, all I am saying is that I don't feel the effect, yet.

Looking forward to watch perfect generated videos. We need so much more power and chips but it’s completely worth it. After that? Maybe generated videogames. But the video stuff will be awesome and changing the video dominated social media content for ever. Virtual headsets will become useful finally generating anything you want to see and jump tru space and time.

Re: Why wordfreq will not be updated

#476
post #4

I created https://lowbackgroundsteel.ai/ in 2023 as a place to gather references to unpolluted datasets. I'll add wordfreq. Please submit stuff to the Tumblr.

Congratulations on "shipping", I've had a background task to create pretty much exactly this site for a while. What is your cutoff date? I made this handy list, in research for mine: 2017: Invention of transformer architecture June 2018: GPT-1 February 2019: GPT-2 June 2020: GPT-3 March 2022: GPT-3.5 November 2022: ChatGPT You may want to add kiwix archives from before whatever date you choose. You can find them on t…

I was taking "Release of ChatGPT" as the Trinity date.

Re: Why wordfreq will not be updated

#477
post #4

I created https://lowbackgroundsteel.ai/ in 2023 as a place to gather references to unpolluted datasets. I'll add wordfreq. Please submit stuff to the Tumblr.

FYI: My two datasets, DebateSum and OpenDebateEvidence/OpenCaseList in their current forms qualify for this, as they end at latest in 2022.

You can either add them to the site yourself via Tumblr or send them to me via email (jgc@cloudflare).

Re: Why wordfreq will not be updated

#478
post #210

Earlier quoted context omitted.

> and only available from pre-war / pre-atomic sources. From the same wiki you linked: "Since the end of atmospheric nuclear testing, background radiation has decreased to very near natural levels, making special low-background steel no longer necessary for most radiation-sensitive uses, as brand-new steel now has a low enough radioactive signature" and "For the most demanding items even low-background steel can be t…

reading stuff like this makes me so happy. no matter how fucked up something may be there is always a way to clean right up.

I wouldn't be so optimistic. The thing you call "way" is actually just time. Yes, anything humanity does (good or bad) will fade with time. But do we have the amount of time to clean up X (and i don't refer to X as in "formally twitter")?

Re: Why wordfreq will not be updated

#479
post #466

I agree with the general ethos of the piece (albeit a few of the details are puzzling and unnecessarily partisan - content on X isn't invariably worthless drivel, nor does what Reddit is doing make much intellectual as opposed to economic [IPO-influenced] sense - but this line: 'OpenAI and Google can collect their own damn data. I hope they have to pay a very high price for it, and I hope they're constantly cursing t…

> puzzling and unnecessarily partisan - content on X isn't invariably worthless drivel

Maybe this is because I’m European, but what is partisan about calling X invariably worthless drivel? Seems a lot like facts to me considering what has been going on with the platform moderation since Elon Musk bought it. It’s so bad that the EU consider it a platform for misinformation these days.

Re: Why wordfreq will not be updated

#480

Web before 2021 was still polluted by content farms. The articles were written by humans, but still, they were rubbish. Not compared to current rate of generation, but the web was already dominated by them.

Maybe, it if you’re studying the way humans use language you’re still getting human made data from rubbish. There isn’t any value in AI generated content is what you’re cataloging is human language.
Post reply on HN