Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why wordfreq will not be updated

github.com

Post reply on HN
Print view View original

421–430 of 542 posts

Re: Why wordfreq will not be updated

#421

Post by kristopolous »

0xbadcafebee wrote: post #280

I'm going to call it: The Web is dead. Thanks to "AI" I spend more time now digging through searches trying to find something useful than I did back in 2005. And the sites you do find are largely garbage. As a random example: just trying to find a particular popular set of wireless earbuds takes me at least 10 minutes, when I already know the company, the company's website, other vendors that sell the company's goods…

for tech stuff I just use documentation, bug trackers and source code now. Web searching has become useless.

Re: Why wordfreq will not be updated

#422

Post by rovr138 »

jonas21 wrote: post #395

I think the main reason for sunsetting the project is hinted at near the bottom: > The field I know as "natural language processing" is hard to find these days. It's all being devoured by generative AI. Other techniques still exist but generative AI sucks up all the air in the room and gets all the money. Traditional NLP has been surpassed by transformers, making this project obsolete. The rest of the post reads like…

I think the reason to sunset the project is actually near the top.

> I don't think anyone has reliable information about post-2021 language usage by humans.

It's information about language usage by humans. We know the rate at which generated text has increased after 2021. How do we filter this to only have data from humans?

The bottom is just lamenting what's happening in the field (which is pretty much what everyone that's been doing anything with NLP research is also complaining about behind closed doors).

Re: Why wordfreq will not be updated

#423

Post by jgord »

We will soon face another kind of bit-rot : where so much text is generated by LLMs that it pollutes the human natural language corpus available for training, on the web.

Maybe we actually need to preserve all the old movies / documentaries / books in all languages and mark them as pre-LLM / non-LLM.

But I hazard a guess this wont happen, as its a common good that could only be funded by left-leaning taxation policies - no one can make money doing this, unlike burning carbon chains to power LLMs.

Re: Why wordfreq will not be updated

#424

Post by ipaddr »

jgord wrote: post #423

We will soon face another kind of bit-rot : where so much text is generated by LLMs that it pollutes the human natural language corpus available for training, on the web. Maybe we actually need to preserve all the old movies / documentaries / books in all languages and mark them as pre-LLM / non-LLM. But I hazard a guess this wont happen, as its a common good that could only be funded by left-leaning taxation policie…

Old content can make money now and will be more valuable why wouldn't it happen more frequency?

Re: Why wordfreq will not be updated

#425

Post by travisb »

Eisenstein wrote: post #404
bigstrat2003 wrote: post #401

Earlier quoted context omitted.

Gay marriage, for instance. In the 90s (and even the 2000s) it was pretty common even for people on the left to come down on the side of "I don't think we should allow gay marriage". Whereas now, it's so firmly within the Overton window that even most of the right thinks gay marriage is fine.

The fact is that society changes over time and what was unacceptable during one time becomes acceptable in another. This does not indicate a shift to the left, it indicates humanity becoming less intolerant. Are you going to argue that allow black people and white people to marry and repealing miscegenation laws mean that the country shifted to the left?

Uh yeah.

More tolerance is pretty much the definition of the social-left.

Re: Why wordfreq will not be updated

#426

Post by tdb7893 »

bongodongobob wrote: post #231
squigz wrote: post #217

Earlier quoted context omitted.

> Have "good" small webs EVER prevailed? You ask on HN, one of the highest quality sites I've ever visited in any age of the Internet. IRC is still alive and well among pretty much the same audience as always. I'm not sure it's fair to compare that with the others.

It's high quality when the content is within HN's bubble. Anything related to health, politics, or Microsoft is full of misinformation, ignorance, and garbage like any other site. The Microsoft discussions in particular are extremely low quality.

When economics has come up I've been curious and asked my brother about some of the stuff in the more upvoted comments (he has his PhD in economics with a focus on labor specifically) his reaction has always been something like "that doesn't match my understanding of that" or "I think their analysis is a bit oversimplified".

My experience here is that it's pretty good for things outside of tech (at least better than the average internet) but definitely not great.

Re: Why wordfreq will not be updated

#427

Post by freddie_mercury »

0xbadcafebee wrote: post #362
odo1242 wrote: post #316

Earlier quoted context omitted.

Yeah, this is why I still use the web to order things in a nutshell lol

Where do you order things online that you aren't inundated by ads?

It's a lot LOT easier for me as an adult to ignore ads online than it is for my kids in brick and mortar stores to ignore the candy and toys placed at their eye level.

Re: Why wordfreq will not be updated

#428

Post by [deleted] »

adastra22 wrote: post #398
girvo wrote: post #397

Earlier quoted context omitted.

TFA means "the featured article", so in this case the "Why wordfreq will not be updated" link we're talking about.

To be pedantic, the F in TFA has the same meaning as the F in RTFM. It’s the same origin. On Slashdot (the HN of the early 00’s) people would admonish others to RTFA. Then they started using it as a referent: TFA was the thing you were supposed to have read.

[deleted]

Re: Why wordfreq will not be updated

#429

Post by Eisenstein »

travisb wrote: post #425
Eisenstein wrote: post #404

Earlier quoted context omitted.

The fact is that society changes over time and what was unacceptable during one time becomes acceptable in another. This does not indicate a shift to the left, it indicates humanity becoming less intolerant. Are you going to argue that allow black people and white people to marry and repealing miscegenation laws mean that the country shifted to the left?

Uh yeah. More tolerance is pretty much the definition of the social-left.

Are you saying that the Overton window can never move right as long as we maintain more human rights and tolerance than we had decades before whatever the current time period is?
Post reply on HN