Why wordfreq will not be updated
github.com
Why wordfreq will not be updated
1–10 of 542 posts
Re: Why wordfreq will not be updated
#2It might be fun to collect the same data if not for any other reason than to note the changes but adding the caveat that it doesn’t represent human output.
Might even change the tool name.
Re: Why wordfreq will not be updated
#3Well said
Re: Why wordfreq will not be updated
#4I created https://lowbackgroundsteel.ai/ in 2023 as a place to gather references to unpolluted datasets. I'll add wordfreq. Please submit stuff to the Tumblr.
Re: Why wordfreq will not be updated
#5It might be fun to collect the same data if not for any other reason than to note the changes but adding the caveat that it doesn’t represent human output. Might even change the tool name.
The point was it’s getting harder and harder to do that as things get locked down or go behind a massive paywall to either profit off of or avoid being used in generative AI. The places where previous versions got data is impossible to gather from anymore so the dataset you would collect would be completely different, which (might) cause weird skewing.
Re: Why wordfreq will not be updated
#6Hear, hear!
Re: Why wordfreq will not be updated
#7I wonder if anyone will fork the project. Apart from anything else, the data may still be useful given that we know it is polluted. In fact, it could act as a means of judging the impact of LLMs via that very pollution.
Re: Why wordfreq will not be updated
#8It might be fun to collect the same data if not for any other reason than to note the changes but adding the caveat that it doesn’t represent human output. Might even change the tool name.
The point was it’s getting harder and harder to do that as things get locked down or go behind a massive paywall to either profit off of or avoid being used in generative AI. The places where previous versions got data is impossible to gather from anymore so the dataset you would collect would be completely different, which (might) cause weird skewing.
But that would always be the case. Twitter will not last forever; heck, it may not even be long before an open alternative like Bluesky competes with it. Would be interesting to know what percentage of the original mined data was from Twitter.
Re: Why wordfreq will not be updated
#9Man the AI folks really wrecked everything. Reminds me of when those scooter companies started just dumping their scooters everywhere without asking anybody if they wanted this.
Re: Why wordfreq will not be updated
#10All those writers who'll soon be out of job and/or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI.
(I initially wanted to say 'paid for by the government' but that'd be socialising losses and we've had quite enough of that in the past.)