Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why wordfreq will not be updated

github.com

Post reply on HN
Print view View original

121–130 of 542 posts

Re: Why wordfreq will not be updated

#121

Post by doe_eyes »

voytec wrote: post #13

I agree in general but the web was already polluted by Google's unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on "indexability" instead of readability, made the web a less than ideal source for such analysis long before LLMs. It also made the web a less than ideal source for training. And yet LLMs were still fed articles written for Googlebot, not humans. ML/LLM is the second…

> I agree in general but the web was already polluted by Google's unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on "indexability" instead of readability, made the web a less than ideal source for such analysis long before LLMs.

Blog spam was generally written by humans. While it sucked for other reasons, it seemed fine for measuring basic word frequencies in human-written text. The frequencies are probably biased in some ways, but this is true for most text. A textbook on carburetor maintenance is going to have the word "carburetor" at way above the baseline. As long as you have a healthy mix of varied books, news articles, and blogs, you're fine.

In contrast, LLM content is just a serpent eating its own tail - you're trying to build a statistical model of word distribution off the output of a (more sophisticated) model of word distribution.

Re: Why wordfreq will not be updated

#122

Post by Diti »

Ensorceled wrote: post #91
next_xibalba wrote: post #45

[flagged]

> Also, it is shocking how authoritarian the “left” has become in my lifetime. We are going through a general uptick in authoritarian "discussions" online. It's interesting that you are only seeing it on the "left".

As far as I noticed, the “right” effectively gets the boot in most online communities which abide to a Code of Conduct, leaving mostly the “left” (the most recent example I have in mind of such moderation efforts is the save-nix-together.org open letter). It’s interesting that you don’t notice this happening in the communities you seem to frequent.

Re: Why wordfreq will not be updated

#123

Post by ecshafer »

ilaksh wrote: post #116

Reading through this entire thread, I suspect that somehow generative AI actually became a political issue. Polarized politics is like a vortex sucking all kinds of unrelated things in. In case that doesn't get my comment completely buried, I will go ahead and say honestly that even though "AI slop" and paywalled content is a problem, I don't think that generative AI in itself is a negative at all. And I also think t…

Generative AI is inherently a political issue, its not surprising at all.

There is the case of what is "truth". As soon as you start to ensure some quality of truth to what is generated, that is political.

As soon as generative AI has the capability to take someone's job, that is political.

The instant AI can make someone money, it is political.

When AI is trained on something that someone has created, and now they can generate something similar, it is political.

Re: Why wordfreq will not be updated

#124

Post by jerf »

olabyne wrote: post #85
weinzierl wrote: post #64

"I don't think anyone has reliable information about post-2021 language usage by humans." We've been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now. Especially smaller children don't have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but…

I never thought about that. Humans losing their ability to detect AI content from reality ? It's frightening.

It's even worse than that. Most people have no idea how far CGI has come, and how easily it is wielded even by a couple of dedicated teens on their home computer, let alone people with a vested interest in faking something for some financial reason. People think they know what a "special effect" looks like, and for the most part, people are wrong. They know what CGI being used to create something obviously impossible, like a dinosaur stomping through a city, looks like. They have no idea how easy a lot of stuff is to fake already. AI just adds to what is already there. Heck, to some extent it has caused scammers to overreach, with things like obviously fake Elon Musk videos on YouTube generated from (pure) AI and text-to-speech... when with just a little bit more learning, practice, and amounts of equipment completely reasonable for one person to obtain, they could have done a much better fake of Elon Musk using special effects techniques rather than shoveling text into an AI. The fact that "shoveling text into an AI" may in another few years itself generate immaculate videos is more a bonus than a fundamental change of capability.

Even what's free & open source in the special effects community is astonishing lately.

Re: Why wordfreq will not be updated

#125

Post by dweinus »

> Now the Web at large is full of slop generated by large language models, written by no one to communicate nothing.

Fair and accurate. In the best cases the person running the model didn't write this stuff and word salad doesn't communicate whatever they meant to say. In many cases though, content is simply pumped out for SEO with no intention of being valuable to anyone.

Re: Why wordfreq will not be updated

#126

Post by actionfromafar »

darby_nine wrote: post #57
creshal wrote: post #52

Earlier quoted context omitted.

That's why search engines rated them highly, and why a million spam sites cropped up that paid writers $1/essay to pretend to be Aunt May, and why today every recipe website has a gigantic useless fake essay in front of their copypasted made up recipes.

Ok, but what i said is true regardless of SEO, and that SEO has also fed back into english before LLMs were a thing. If you only train on those subsets you'll also end up with a chatbot that doesn't speak in a way we'll identify as natural english.

Yet. Give it time. The LLMs will train our future children.

Re: Why wordfreq will not be updated

#127

Post by hn_throwaway_99 »

olabyne wrote: post #85
weinzierl wrote: post #64

"I don't think anyone has reliable information about post-2021 language usage by humans." We've been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now. Especially smaller children don't have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but…

I never thought about that. Humans losing their ability to detect AI content from reality ? It's frightening.

I mean, it's already apparent to me that a lot of people don't have a basic process in place to detect fact from fiction. And it's definitely not always easy, but when I hear some of the dumbest conspiracy theories known to man actually get traction in our media, political figures, and society at large, I just have to shake my head and laugh to keep from crying. I'm constantly reminded of my favorite saying, "people who believe in conspiracy theories have never been a project manager."

Re: Why wordfreq will not be updated

#128

Post by wpietri »

ToucanLoucan wrote: post #90
wpietri wrote: post #74

Earlier quoted context omitted.

I think it's an arms race, but it's an open question who wins. For a while I thought email as a medium was doomed, but spammers mostly lost that arms race. One interesting difference is that with spam, the large tech companies were basically all fighting against it. But here, many of the large tech companies are either providing tools to spammers (LLMs) or actively encouraging spammy behaviors (by integrating LLMs in…

> but spammers mostly lost that arms race I'm not saying this is impossible but that's going to be an uphill sell for me as a concept. According to some quick stats I checked I'm getting roughly 600 emails per day, about 550 of which go directly to spam filtering, and of the remaining 50, I'd say about 6 are actually emails I want to be receiving. That's an impressive amount overall for whoever built this particular…

I'm having a hard time finding reliably sourced statistics here, but I suspect you're an outlier. My personal numbers are way better, both on Gmail and Fastmail, despite using the same email addresses for decades.

Re: Why wordfreq will not be updated

#129

Post by Lalabadie »

kevindamm wrote: post #32
voytec wrote: post #13

I agree in general but the web was already polluted by Google's unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on "indexability" instead of readability, made the web a less than ideal source for such analysis long before LLMs. It also made the web a less than ideal source for training. And yet LLMs were still fed articles written for Googlebot, not humans. ML/LLM is the second…

Yes but not quite as far as you imply. The training data is weighted by a quality metric, articles written by journalists and wikipedia contributors are given more weight than Aunt May's brownie recipe and corpoblogspam.

The current state of things leads me to believe that Google's current ranking system has been somehow too transparent for the last 2-3 years.

The top of search results is consistently crowded by pages that obviously game ranking metrics instead of offering any value to humans.

Re: Why wordfreq will not be updated

#130

Post by grues-dinner »

shagie wrote: post #83
creshal wrote: post #52

Earlier quoted context omitted.

That's why search engines rated them highly, and why a million spam sites cropped up that paid writers $1/essay to pretend to be Aunt May, and why today every recipe website has a gigantic useless fake essay in front of their copypasted made up recipes.

I wish more people presented recipes like cooking for engineers. For example - Meat Lasagna https://www.cookingforengineers.com/recipe/36/Meat-Lasagna

And here I thought my defacement of printed recipes by bracketing everything that goes together at each stage was just me. There are, well, maybe not dozens but at least two of us! Saves a lot of bowls when you know without further checking that you can, say, just dump the flour and sugar, butter and eggs into the big bowl without having to prepare separately because they're in the "1: big bowl" bracket.
Post reply on HN