Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Delving into ChatGPT usage in academic writing through excess vocabulary

arxiv.org

Post reply on HN
Print view View original

21–30 of 112 posts

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#21

Post by josephg »

AlienRobot wrote: post #15

This is so disappointing.

Eh. Its a tool, just like a calculator, a computer or a drug. Just like everything else that has potency, you can use it well or poorly.

I got some help from ChatGPT while writing a recent paper. At one point while writing, I couldn't find a straightforward way to express myself. I was stuck - and then every attempt at writing the same paragraph came out worse than the last. Eventually I gave my draft to chatgpt and asked it to come up with a few suggestions on how to express my ideas more clearly. That really helped. I adapted some of its writing and ended up with a better final result.

I'm not sure if anything chatgpt wrote is in the final paper. But I really appreciated the LLM assistance to help me express myself.

I'm sure there's a lot of much more egregious examples where chatgpt wholesale wrote large sections of some papers.

But its really not the tool that's at fault. Its how the tool is being used. And we simply don't have social norms around that yet, because figuring out norms takes time.

The calculator was the same. When is a calculator acceptable in a classroom? In an exam? Our teachers at the time justified "no calculator" policies by saying "You won't have a calculator in your pocket when you're going through life!". And, well, that turned out to be very wrong.

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#23

Post by daemonologist »

usef- wrote: post #18
lioeters wrote: post #3

> We study vocabulary changes in 14 million PubMed abstracts from 2010-2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. "Delving".. Sounds like the authors might have used an LLM while writing this paper as well.

I'm surprised people make such a big deal out of "delving", it doesn't seem rare. Maybe it was less common in the US than elsewhere? (and do I have to deliberately avoid it now, to avoid sounding like AI?)

I think it's not an extraordinarily rare word in the US, but it does feel to me more at home in a blog post or business strategy-ish document than a PubMed abstract. People are making a big deal of it just because the signal is so strong (25x more common now than in 2022).

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#24

Post by dweekly »

usef- wrote: post #18
lioeters wrote: post #3

> We study vocabulary changes in 14 million PubMed abstracts from 2010-2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. "Delving".. Sounds like the authors might have used an LLM while writing this paper as well.

I'm surprised people make such a big deal out of "delving", it doesn't seem rare. Maybe it was less common in the US than elsewhere? (and do I have to deliberately avoid it now, to avoid sounding like AI?)

The spike is global.

https://trends.google.com/trends/explore?date=all&q=delving&...

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#26

Post by hdhshdhshdjd »

AlienRobot wrote: post #15

This is so disappointing.

While it would never be published, I imagine if they broke it down by author surname you’d see most of this is non English speakers using a juiced up grammar check.

I see no evidence actual findings are changing.

Lots of people have good science to contribute and suck at writing English. Making findings easier to digest isn’t bad per se.

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#27

Post by cortesoft »

silver_silver wrote: post #17

It’s possible this is caused by the editors rather than the authors. An old partner edited papers for a large publisher - largely written by non-native speakers and already heavily machine translated - and would almost always use ChatGPT for the first pass when extensive changes were needed. She was paid by the word and also had a pretty intense daily minimum quota so it was practically required to get enough done to…

This seems strange… this is an old partner, who had a job long enough to know that the only way to hit a quota to was to use ChatGPT, which came out less than two years ago?

It seems strange that something that experimental became essential that quickly.

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#28

Post by daemonologist »

refibrillator wrote: post #7

One confounding factor here is the proliferation of autocorrect and grammar “advisors” in popular apps like gmail etc. One algorithm tweak could change a lot of writing at that scale. While the word frequency stats are damning, there doesn’t seem to be any evidence presented that directly ties the changes to LLMs specifically.

I think they address this to some degree by checking different year pairings (end of page 2), where the only excess usage they found was of words related to current events (ebola, coronavirus, etc.) and even then not to the same degree as the 2022-24 pair.

It would be interesting to analyze how well a language model is able to predict each abstract. In theory if the text was largely written by a model then a similar model might be able to predict it more accurately than it would a human-written abstract. (Of course the variety of models and frequency at which they're updated makes this more difficult.)

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#29

Post by nicce »

Animats wrote: post #14

Their list: delves crucial potential these significant important They're not the first to make this observation. Others have picked up that LLM's like the word "delves". LLMs are trained on texts which contain much marketing material. So they tend to use some marketing words when generating pseudo-academic content. No surprise there. I'm surprised it's not worse. What happens if you use a prompt containing "Write in…

Traditional grammar correction applications usually drop the "important" words as well and suggests something else.

Re: Delving into ChatGPT usage in academic writing through excess vocabulary

#30

Post by noizejoy »

cortesoft wrote: post #27
silver_silver wrote: post #17

It’s possible this is caused by the editors rather than the authors. An old partner edited papers for a large publisher - largely written by non-native speakers and already heavily machine translated - and would almost always use ChatGPT for the first pass when extensive changes were needed. She was paid by the word and also had a pretty intense daily minimum quota so it was practically required to get enough done to…

This seems strange… this is an old partner, who had a job long enough to know that the only way to hit a quota to was to use ChatGPT, which came out less than two years ago? It seems strange that something that experimental became essential that quickly.

> this is an old partner

“Old” might be intended to mean “former” or “ex”, which could be as recent as merely weeks ago. (i.e. ESL / lost in translation issue).

Post reply on HN