Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why wordfreq will not be updated

github.com

Post reply on HN
Print view View original

321–330 of 542 posts

Re: Why wordfreq will not be updated

#321

Post by commodoreboxer »

FrustratedMonky wrote: post #219
commodoreboxer wrote: post #197

Earlier quoted context omitted.

I thought so, but the rest of the comment is worded quite reasonably, so I decided to not interpret it as hyperbole or irony.

"quite reasonably" Joke is, it was GPT 4o. I didn't phrase it to be like an LLM, I asked GPT to defend itself, and used it's text un-edited. Except adding the 'but but but" There is a lot out there that people are no longer able to determine if it is an LLM. They are getting better. That they are no longer obvious is the scary part.

Ah, I've never really been able to discern. There is obvious LLM text, but it's harder to tell the difference in short samples, and with some prompting to get it to write in a particular style, it's been easy to make LLM output non-obvious for a while now.

Re: Why wordfreq will not be updated

#322

Post by bbarn »

0xbadcafebee wrote: post #280

I'm going to call it: The Web is dead. Thanks to "AI" I spend more time now digging through searches trying to find something useful than I did back in 2005. And the sites you do find are largely garbage. As a random example: just trying to find a particular popular set of wireless earbuds takes me at least 10 minutes, when I already know the company, the company's website, other vendors that sell the company's goods…

No disagreement for the most part.

I used to be able to say search for Trek bike derailleur hanger and the first result would be what I wanted. Now I have to scroll past 5 ads to buy a new bike, one that's a broken link to a third party, and if I'm really lucky, at the bottom of page 1 will be the link to that part's page.

The shitification of the web is real.

Re: Why wordfreq will not be updated

#323

Post by lbhdc »

weinzierl wrote: post #319
doe_eyes wrote: post #121

Earlier quoted context omitted.

> I agree in general but the web was already polluted by Google's unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on "indexability" instead of readability, made the web a less than ideal source for such analysis long before LLMs. Blog spam was generally written by humans. While it sucked for other reasons, it seemed fine for measuring basic word frequencies in human-written tex…

Isn't it the other way around? SEO text carefully tuned to tf-idf metrics and keyword stuffed to them empirically determined threshold Google just allows should have unnatural word frequencies. LLM content should just enhance and cement the status quo word frequencies. Outliers like the word "delve" could just be sentinels, carefully placed like trap streets on a map.

> LLM content should just enhance and cement the status quo word frequencies.

TFA mentions this hasn't been the case.

Re: Why wordfreq will not be updated

#324

Post by hcks »

Okay but how big of a sample size do we even actually need for word frequencies? Like what’s the goal here? It looks like the initial project isn’t even stratified per year/decade

Re: Why wordfreq will not be updated

#325

Post by bane »

thechao wrote: post #304
bane wrote: post #300

This is one of the vanguards warning of the changes coming in the post-AI world. >> Generative AI has polluted the data Just like low-background steel marks the break in history from before and after the nuclear age, these types of data mark the distinction from before and after AI. Future models will begin to continue to amplify certain statistical properties from their training, that amplified data will continue to…

That went off the rails quickly. Calm down dude: my mother-in-law isn't going to forget words because of AI; she's gonna forget words because she's 3 glasses of crappy Texas wine into the evening.

But your children's children will never learn about love because that word will have been mechanically trained out of existence.

Re: Why wordfreq will not be updated

#326

Post by weinzierl »

apricot wrote: post #207
weinzierl wrote: post #64

"I don't think anyone has reliable information about post-2021 language usage by humans." We've been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now. Especially smaller children don't have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but…

> When I get asked if the person in a video is real, I still feel pretty confident to answer I don't. I mean, I can identify the bad ones, sure, but how do I know I'm not getting fooled by the good ones?

That is very true, but for now we have a baseline of videos that we either remember or that we remember key details of, like the persons in the video. I'm pretty sure if I watch The Primeagen or Tom Scott today, that they are real. Ask me in year, I might not be so sure anymore.

Re: Why wordfreq will not be updated

#327

Post by aanet »

fer wrote: post #306
bane wrote: post #300

This is one of the vanguards warning of the changes coming in the post-AI world. >> Generative AI has polluted the data Just like low-background steel marks the break in history from before and after the nuclear age, these types of data mark the distinction from before and after AI. Future models will begin to continue to amplify certain statistical properties from their training, that amplified data will continue to…

> Future models will begin to continue to amplify certain statistical properties from their training, that amplified data will continue to pollute the public space from which future training data is drawn. That's why on FB I mark my own writing as AI generated, and the AI generated slop as genuine. Because what is disguised as "transparency disclaimer" is just flagging content of what's a potential dataset to train f…

You, Sir, may have stumbled upon the just the -hack- advice needed to post on social media.

Apropos of nothing in particular, see LinkedIn now admitting [1] it is training its AI models on "all users by default"

[1] https://www.techmeme.com/240918/p34#a240918p34

Re: Why wordfreq will not be updated

#328

Post by _proofs »

bongodongobob wrote: post #239
diggan wrote: post #225

One of the examples is the increased usage of "delve" which Google Trends confirms increased in usage since 2022 (initial ChatGPT release): https://trends.google.com/trends/explore?date=all&q=delve&hl... It seems however it started increasing most in usage just these last few months, maybe people are talking more about "delve" specifically because of the increase in usage? A usage recursion of some sorts.

Delves are a new thing in World of Warcraft released 9/10 this year. Delve is also an M365 product that has been around for some time and is being discontinued in December. So no, that has nothing to do with LLMs.

Delve was also an addition to PoE, which I imagine had its own spike in google searches relative to that word.

Re: Why wordfreq will not be updated

#329

Post by bane »

midnitewarrior wrote: post #309
bane wrote: post #300

This is one of the vanguards warning of the changes coming in the post-AI world. >> Generative AI has polluted the data Just like low-background steel marks the break in history from before and after the nuclear age, these types of data mark the distinction from before and after AI. Future models will begin to continue to amplify certain statistical properties from their training, that amplified data will continue to…

From the day of the first spoken word, humans have guided the development of language through conversational use and institution. With the advent of AI being used to publish documents into the open web, humans have given up their exclusive domain. What would it take for Open AI overlords to inject words they want to force into usage in their models and will new words into use? Few have had the power to do such things…

It's the ultimate seizure of the means of production, and in the end it will be the capitalists who realize that revolution.

Re: Why wordfreq will not be updated

#330

Post by sangnoir »

aryonoco wrote: post #308
sangnoir wrote: post #294

Earlier quoted context omitted.

> Back in pre-LLM days, it's not like I would have hired a x264 expert to do this job for me. I would have either had to spend hours more on this task, or more likely, this 97 year old man would never have seen his great granddaughter's dance Didn't most DVD burning software include video transcoding as a standard feature? Back in the day, you'd have used Nero Burning ROM, or Handbrake - granted, the quality may not…

Back in the day they did. I checked handbrake but now there's nothing specific about DVD compatibility there. I could have picked something like Super HQ 576p, and there's a good chance that would have sufficed, but old DVD players were extremely finicky about filenames, extensions, interlacing, etc. I didn't want to risk the DVD traveling half way across the world only to find that it's not playable.

I mentioned Handbrake without checking its DVD authoring capability - probably used it to rip DVDs many years ago and got it mixed up with burning them; a better FLOSS alternative for authoring would have been DeVeDe or bombono.
Post reply on HN