All those writers who'll soon be out of job and/or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI. (I initially wanted to say 'paid for by the government' but that'd be socialising losses and we've had quite enough of that in the past.)
Why wordfreq will not be updated
21–30 of 542 posts
Re: Why wordfreq will not be updated
#22All those writers who'll soon be out of job and/or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI. (I initially wanted to say 'paid for by the government' but that'd be socialising losses and we've had quite enough of that in the past.)
AI companies are indeed hiring such people to generate customized training data for them.
Re: Why wordfreq will not be updated
#23Re: Why wordfreq will not be updated
#24compare the frequency of words to those used in human natural writings and you spot the computer from the human.
Re: Why wordfreq will not be updated
#25Did we (the humans) somehow managed to pollute the internet so much with AI that's it's now barely usable ? In my opinion the internet can be considered as the equivalent of a natural environment like the earth. it's a space where people share, meet, talk, etc. I find it astonishing that after polluting our natural environment we know polluted the internet.
If we haven't already, we will be very soon. I'm sure there are people working on this problem, but I think we're starting to hit a very imminent feedback loop moment. Most of human's recorded information is digitized and most of that is generating non-human content at an incredible pace. We've injected a whole lot of noise into our usable data.
I don't know if the answer is more human content (I'm doing my part!) or novel generative content but this interim period is going to cause some medium-term challenges.
I like to think the LLM more-tokens-equals-better era is fading and we're getting into better use of existing data, but there's a very real inflection point we're facing.
Re: Why wordfreq will not be updated
#26Did we (the humans) somehow managed to pollute the internet so much with AI that's it's now barely usable ? In my opinion the internet can be considered as the equivalent of a natural environment like the earth. it's a space where people share, meet, talk, etc. I find it astonishing that after polluting our natural environment we know polluted the internet.
Re: Why wordfreq will not be updated
#27Sad. I'd love to see by how much the use of world "delve" has increased since 2021...
There are charts / graphs in the link, both since 2021, and since earlier.
The final graph suggests the phenomenon started earlier, possibly correlated in some way to Malaysian / Indian usages of English.
It does seem OpenAI's family of GPTs as implemented in ChatGPT unspool concepts in a blend of India-based-consultancy English with American freshmen essay structure, frosted with superficially approachable or upbeat blogger prose ingratiatingly selling you something.
Anthropic has clearly made efforts to steer this differently, Mistral and Meta as well but to a lesser degree.
I've wondered if this reflects training material (the SEO is ruining the Internet theory), or is more simply explained by selection of pools of Hs hired for RLHF.
Re: Why wordfreq will not be updated
#28>"Now Twitter is gone anyway, its public APIs have shut down, and the site has been replaced with an oligarch's plaything, a spam-infested right-wing cesspool called X. Even if X made its raw data feed available (which it doesn't), there would be no valuable information to be found there. >Reddit also stopped providing public data archives, and now they sell their archives at a price that only OpenAI will pay. >And g…
Given just how many AI bots scrape up everything they can, oftentimes ignoring robots.txt or any rate limits (there have been a few complaint threads on HN about that), I can hardly blame the operators of large online services just cutting off data feeds.
Twitter however didn't stop their data feeds due to AI or because they wanted money, they stopped providing them because its new owner does everything he can to hinder researchers specializing in propaganda campaigns or public scrutiny.
Re: Why wordfreq will not be updated
#29All those writers who'll soon be out of job and/or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI. (I initially wanted to say 'paid for by the government' but that'd be socialising losses and we've had quite enough of that in the past.)
There are already several companies doing this - I do occasional contract work for a couple -, and paying rates sometimes well above what an average earning writer can expect elsewhere. However, the vast majority of writers have never been able to make a living from their writing. The threshold to write is too love, too many people love it, and most people read very little.
Re: Why wordfreq will not be updated
#30Did we (the humans) somehow managed to pollute the internet so much with AI that's it's now barely usable ? In my opinion the internet can be considered as the equivalent of a natural environment like the earth. it's a space where people share, meet, talk, etc. I find it astonishing that after polluting our natural environment we know polluted the internet.