Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why wordfreq will not be updated

github.com

Post reply on HN
Print view View original

81–90 of 542 posts

Re: Why wordfreq will not be updated

#81

Post by formerly_proven »

creshal wrote: post #76
smallerfish wrote: post #73

Earlier quoted context omitted.

There seem to be quite a few recipe sharing sites around - e.g. allrecipes.com.

And they're all flooded with low effort trash and useless. The only remaining reliable source - now that many newspapers are axing the remaining staff in favour of LLMs - is pre-2020 print cookbooks. Anything online or printed later must be assumed to be tainted, full of untested sewage and potentially dangerous suggestions.

Well there's https://www.allrecipes.com/author/chef-john/ on that particular site.

Re: Why wordfreq will not be updated

#82

Post by rvnx »

Retr0id wrote: post #78
Miraltar wrote: post #50

Earlier quoted context omitted.

I guess it would be interesting but differentiating pollution from language evolution seems very tricky since getting a non polluted corpus gets harder and harder

Arguably it is a form of language evolution. I bet humans have started using "delve" more too, on average. I think the best we can do is look at the trends and think about potential causes.

“Seamless”, “honed”, “unparalleled”, “delve” are now polluting the landscape because of monkeys repeating what ChatGPT says without even questioning what the words mean.

Everything is “seamless” nowadays. Like I am seamlessly commenting here.

Arguably, the meaning of these words evolve due to misuse too.

Re: Why wordfreq will not be updated

#83

Post by shagie »

creshal wrote: post #52
darby_nine wrote: post #43

Earlier quoted context omitted.

Aunt may's brownie recipe (or at least her thoughts on it) are likely something you'd want if you want to reflect how humans use language. Both news-style and encyclopedia-style writing represent a pretty narrow slice.

That's why search engines rated them highly, and why a million spam sites cropped up that paid writers $1/essay to pretend to be Aunt May, and why today every recipe website has a gigantic useless fake essay in front of their copypasted made up recipes.

I wish more people presented recipes like cooking for engineers. For example - Meat Lasagna https://www.cookingforengineers.com/recipe/36/Meat-Lasagna

Re: Why wordfreq will not be updated

#84

Post by ckemere »

baq wrote: post #29
vidarh wrote: post #17

Earlier quoted context omitted.

There are already several companies doing this - I do occasional contract work for a couple -, and paying rates sometimes well above what an average earning writer can expect elsewhere. However, the vast majority of writers have never been able to make a living from their writing. The threshold to write is too love, too many people love it, and most people read very little.

Transformers read a lot during training, it might actually be beneficial for the companies to the point those works never see the light of day, only machines would read them. That's so dystopian I'd say those works should be published so they eventually get into the public domain.

Rooms full of people writing into a computer is a striking mental picture. It feels like it could be background for a great plot for a book/movie.

Re: Why wordfreq will not be updated

#85

Post by olabyne »

weinzierl wrote: post #64

"I don't think anyone has reliable information about post-2021 language usage by humans." We've been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now. Especially smaller children don't have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but…

I never thought about that. Humans losing their ability to detect AI content from reality ? It's frightening.

Re: Why wordfreq will not be updated

#86

Post by rvnx »

dsign wrote: post #53

Somehow related, paper books from before 2020 could be a valuable commodity in a in a decade or two, when the Internet will be full of slop and even contemporary paper books will be treated with suspicion. And there will be human talking heads posing as the authors of books written by very smart AIs. God, why are we doing this????

To support well-known “philanthropists” like Sam Altman or Mark Zuckerberg that many consider as their heroes here.

Re: Why wordfreq will not be updated

#88

Post by DrillShopper »

kdmccormick wrote: post #63
shortrounddev2 wrote: post #9

Man the AI folks really wrecked everything. Reminds me of when those scooter companies started just dumping their scooters everywhere without asking anybody if they wanted this.

At least scooters did something useful for the environment.

Their batteries on the other hand…

Re: Why wordfreq will not be updated

#89

Post by wraptile »

olabyne wrote: post #85
weinzierl wrote: post #64

"I don't think anyone has reliable information about post-2021 language usage by humans." We've been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now. Especially smaller children don't have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but…

I never thought about that. Humans losing their ability to detect AI content from reality ? It's frightening.

I find issue with this statement as content was never a clean representation of human actions or even thought. It was always driven by editorials, SEO, bot remixing and whatnot that heavily influences how we produce content. One might even argue that heightened content distrust is _good_ for our society.

Re: Why wordfreq will not be updated

#90

Post by ToucanLoucan »

wpietri wrote: post #74
ToucanLoucan wrote: post #66

Earlier quoted context omitted.

This feels like a second, magnitudes larger Eternal September. I wonder how much more of this the Internet can take before everyone just abandons it entirely. My usage is notably lower than it was in even 2018, it's so goddamn hard to find anything worth reading anymore (which is why I spend so much damn time here, tbh).

I think it's an arms race, but it's an open question who wins. For a while I thought email as a medium was doomed, but spammers mostly lost that arms race. One interesting difference is that with spam, the large tech companies were basically all fighting against it. But here, many of the large tech companies are either providing tools to spammers (LLMs) or actively encouraging spammy behaviors (by integrating LLMs in…

> but spammers mostly lost that arms race

I'm not saying this is impossible but that's going to be an uphill sell for me as a concept. According to some quick stats I checked I'm getting roughly 600 emails per day, about 550 of which go directly to spam filtering, and of the remaining 50, I'd say about 6 are actually emails I want to be receiving. That's an impressive amount overall for whoever built this particular filter, but it's also still a ton of chaff to sort wheat from and as a result I don't use email much for anything apart from when I have to.

Like, I guess that's technically usable, I'm much happier filtering 44 emails than 594 emails? But that's like saying I solved the problem of a flat tire by installing a wooden cart wheel.

It's also worth noting there that if I do have an email thats flagged as spam that shouldn't be, I then have to wade through a much deeper pond of shit to go find it as well. So again, better, but IMO not even remotely solved.

Post reply on HN