Live data from Hacker News

Why wordfreq will not be updated

github.com

11–20 of 542 posts

Re: Why wordfreq will not be updated

#12

Sad. I'd love to see by how much the use of world "delve" has increased since 2021...

From the submission you're commenting on:

> As one example, Philip Shapira reports that ChatGPT (OpenAI's popular brand of generative language model circa 2024) is obsessed with the word "delve" in a way that people never have been, and caused its overall frequency to increase by an order of magnitude.

Re: Why wordfreq will not be updated

#13
I agree in general but the web was already polluted by Google's unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on "indexability" instead of readability, made the web a less than ideal source for such analysis long before LLMs.

It also made the web a less than ideal source for training. And yet LLMs were still fed articles written for Googlebot, not humans. ML/LLM is the second iteration of writing pollution. The first was humans writing for corporate bots, not other humans.

Re: Why wordfreq will not be updated

#14
>"Now Twitter is gone anyway, its public APIs have shut down, and the site has been replaced with an oligarch's plaything, a spam-infested right-wing cesspool called X. Even if X made its raw data feed available (which it doesn't), there would be no valuable information to be found there.

>Reddit also stopped providing public data archives, and now they sell their archives at a price that only OpenAI will pay.

>And given what's happening to the field, I don't blame them."

What beautiful doublethink.

Re: Why wordfreq will not be updated

#15
Enshittification is accelerating. A good 70% of my Facebook feed is now obviously AI generated images with AI generated text blurbs that have nothing to do with the accompanying images likely posted by overseas bot farms. I'm also noticing more and more "books" on Amazon that are clearly AI generated and self published.

Re: Why wordfreq will not be updated

#16
post #10

All those writers who'll soon be out of job and/or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI. (I initially wanted to say 'paid for by the government' but that'd be socialising losses and we've had quite enough of that in the past.)

AI companies are indeed hiring such people to generate customized training data for them.

Re: Why wordfreq will not be updated

#17
post #10

All those writers who'll soon be out of job and/or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI. (I initially wanted to say 'paid for by the government' but that'd be socialising losses and we've had quite enough of that in the past.)

There are already several companies doing this - I do occasional contract work for a couple -, and paying rates sometimes well above what an average earning writer can expect elsewhere. However, the vast majority of writers have never been able to make a living from their writing. The threshold to write is too love, too many people love it, and most people read very little.

Re: Why wordfreq will not be updated

#18
Did we (the humans) somehow managed to pollute the internet so much with AI that's it's now barely usable ?

In my opinion the internet can be considered as the equivalent of a natural environment like the earth. it's a space where people share, meet, talk, etc.

I find it astonishing that after polluting our natural environment we know polluted the internet.

Re: Why wordfreq will not be updated

#19

Sad. I'd love to see by how much the use of world "delve" has increased since 2021...

The fun thing is that while GPTs initially learned from humans (because ~100% of the content was human-generated), future humans will learn from GPTs, because almost all available content would be GPT-generated very soon.

This will surely affect how we speak. It's possible that human language evolution could come to a halt, stuck in time as AI datasets stop being updated.

In the worst case, we will see a global "model collapse" with human languages devolving along with AI's, if future AIs are trained on their own outputs...

Re: Why wordfreq will not be updated

#20

Sad. I'd love to see by how much the use of world "delve" has increased since 2021...

From the submission you're commenting on: > As one example, Philip Shapira reports that ChatGPT (OpenAI's popular brand of generative language model circa 2024) is obsessed with the word "delve" in a way that people never have been, and caused its overall frequency to increase by an order of magnitude.

https://pshapira.net/2024/03/31/delving-into-delve/ "Delving into “delve”"
Post reply on HN