Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why does ChatGPT love the word "eager" so much?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#11

Post by jolt42 »

JohnFen wrote: post #3
solardev wrote: post #2

Can it just be a frequency illusion, where you tend to notice a new-to-you phenomenon again and again at first https://en.wikipedia.org/wiki/Frequency_illusion?wprov=sfla1 ? Eager isn't an especially uncommon word (eg "eager beavers" is a somewhat common saying), even though it's not used in most convos. I feel like "delve" is a YouTube phenomenon (as in "let's delve into this topic") as a weird proxy for "deep dive"…

> Eager isn't an especially uncommon word I'd say it's very common, at least in my part of the US. It's one of the words I hear on a daily basis, anyway. "Delve" used to be a very commonly used term before "deep dive" largely replaced it. I'm sure there are a whole lot of writings online that use "delve" because of the time period they were produced in. As a graybeard, I'm personally still much more likely to say "de…

Seems like "chef's kiss" is replacing "icing on the cake" or "cherry on top", although I think it really means "stamp of approval", so that one has been bugging me since I hear it all the time now, it seems.

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#13

Post by kgeist »

wietze wrote: post #7

Although "eager" isn't called out, a recent study of academic publications shows that the use of LLMs can be measured through word frequency analysis [1], finding certain words are disproportionally represented: > We study vocabulary changes in 14 million PubMed abstracts from 2010–2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. 1: https://arxiv.org/html/24…

I wonder why some words are overrepresented. Isn't the whole idea of language models to model word distribution as close as possible? Does it have something to do with RLHF? Or it's the training data?

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#14

Post by PeterisP »

wietze wrote: post #7

Although "eager" isn't called out, a recent study of academic publications shows that the use of LLMs can be measured through word frequency analysis [1], finding certain words are disproportionally represented: > We study vocabulary changes in 14 million PubMed abstracts from 2010–2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. 1: https://arxiv.org/html/24…

I don't want to look for the source of analysis right now, but I recall reading a study demonstrating that a large part, if not most of the word frequency shift was caused by RLHF training done on data predominantly generated by people hired from lower income English-speaking countries which simply have a different dialect of English with a noticeably different frequency of certain phrases and expressions, so e.g. at least some versions of ChatGPT got RLHF-trained to speak more in a Nigerian English dialect.

Since there isn't a single English (English learners generally get informed about the choice of UK vs US English only, but most English is spoken outside of UK and USA in other places and other dialects), but multiple different Englishes, any English speaker will probably find something to be surprised by, and there is an economic incentive to get data from people other than the relatively expensive native speakers of UK or USA English.

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#15

Post by soco »

agubelu wrote: post #5

Another giveaway of GPT content for me are overusing importance adjectives like "crucial" or "essential", and of course an extreme overuse of enumerations.

An extreme overuse of everything. Like "give me a formula to color every second Excel row blue". "Sure! First make sure you have installed Excel, here are the steps..."

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#17

Post by llm_nerd »

PeterisP wrote: post #14
wietze wrote: post #7

Although "eager" isn't called out, a recent study of academic publications shows that the use of LLMs can be measured through word frequency analysis [1], finding certain words are disproportionally represented: > We study vocabulary changes in 14 million PubMed abstracts from 2010–2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. 1: https://arxiv.org/html/24…

I don't want to look for the source of analysis right now, but I recall reading a study demonstrating that a large part, if not most of the word frequency shift was caused by RLHF training done on data predominantly generated by people hired from lower income English-speaking countries which simply have a different dialect of English with a noticeably different frequency of certain phrases and expressions, so e.g. at…

There wasn't a study or analysis. It was just lazy speculation that felt good because it could be bound up in a "evil white countries exploiting the developing world" narrative. Where exploiting was "paying to do a job".

It was submitted as https://news.ycombinator.com/item?id=40623629

Again, there is effectively zero real data showing this. Further, RLHF isn't likely to reinforce such word selection regardless.

A more logical, likely scenario is that training data is biased heavily towards higher grade level material, so word selection veers towards writings that you find in those realms.

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#18

Post by ben_w »

kgeist wrote: post #13
wietze wrote: post #7

Although "eager" isn't called out, a recent study of academic publications shows that the use of LLMs can be measured through word frequency analysis [1], finding certain words are disproportionally represented: > We study vocabulary changes in 14 million PubMed abstracts from 2010–2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. 1: https://arxiv.org/html/24…

I wonder why some words are overrepresented. Isn't the whole idea of language models to model word distribution as close as possible? Does it have something to do with RLHF? Or it's the training data?

Language models would be fairly useless for most people if they accurately modelled the source distribution, no better than autocomplete. In fact, they were fairly useless when they modelled the source distribution, that's why ChatGPT was an instant hit whereas GPT-3 was mainly only interesting to other AI reasearchers.

What made LLMs suddenly interesting was that the responses were much more like answers and much less like additional questions in the same vein as the prompt.

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#19

Post by vundercind »

JohnFen wrote: post #3
solardev wrote: post #2

Can it just be a frequency illusion, where you tend to notice a new-to-you phenomenon again and again at first https://en.wikipedia.org/wiki/Frequency_illusion?wprov=sfla1 ? Eager isn't an especially uncommon word (eg "eager beavers" is a somewhat common saying), even though it's not used in most convos. I feel like "delve" is a YouTube phenomenon (as in "let's delve into this topic") as a weird proxy for "deep dive"…

> Eager isn't an especially uncommon word I'd say it's very common, at least in my part of the US. It's one of the words I hear on a daily basis, anyway. "Delve" used to be a very commonly used term before "deep dive" largely replaced it. I'm sure there are a whole lot of writings online that use "delve" because of the time period they were produced in. As a graybeard, I'm personally still much more likely to say "de…

All of that’s way more common in (American) business English than other registers, I’d say, including “eager”.

Re: Ask HN: Why does ChatGPT love the word "eager" so much?

#20

Post by supercoffee »

solardev wrote: post #2

Can it just be a frequency illusion, where you tend to notice a new-to-you phenomenon again and again at first https://en.wikipedia.org/wiki/Frequency_illusion?wprov=sfla1 ? Eager isn't an especially uncommon word (eg "eager beavers" is a somewhat common saying), even though it's not used in most convos. I feel like "delve" is a YouTube phenomenon (as in "let's delve into this topic") as a weird proxy for "deep dive"…

My new frequency illusion is the phrase "ride or die". I'm 32 and I had never heard that in my life until I watched that show Cruel Summer and now it seems like I hear it everywhere, mostly on tv/ads/podcasts and almost never when talking to people I know. I refuse to say it because I don't fully understand what it means and when I do hear it, it always sounds a little cringey.
Post reply on HN