Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When ChatGPT broke the field of NLP: An oral history

quantamagazine.org

Post reply on HN
Print view View original

161–170 of 181 posts

Re: When ChatGPT broke the field of NLP: An oral history

#161

Post by mgraczyk »

more-nitor wrote: post #119
mgraczyk wrote: post #87

Earlier quoted context omitted.

It currently costs around $2200 to run Gemini flash lite on all of Wikipedia English. It would probably cost around 10x that much to run sentiment analysis on every Yelp review ever posted. It's true that LLMs still cost a lot for some use cases, but for essentially any business case it's not worth using traditional NLP any more

idk why are you changing targets for comparison? it's like: "does apple cure cancer in monkeys?" vs "does blueberry cure diabetes in pigs?"

More like "does apple cure cancer in monkeys?" vs "no, but some do cure diabetes in pigs"

Re: When ChatGPT broke the field of NLP: An oral history

#162

Post by canjobear »

suddenlybananas wrote: post #137
canjobear wrote: post #81

Earlier quoted context omitted.

There's a lot of debate about it. Here's one view: https://arxiv.org/abs/2501.17047

This paper is awful. They bizarrely argue the fact that transformers are not very sensitive to word order as a positive of transformers despite the fact that's not how languages work. There's also this absurd passage. >However, a closer look at the statistical structure of language use reveals that word order contains surprisingly little information over and above lexical information. To see this intuitively, imagine…

I don't see how it's absurd. The point isn't that languages don't have word order constraints, it's that they're organized in such a way that word order is usually redundant with other things when it comes to expressing meaning. This redundancy is a real and nontrivial property of language which becomes obvious when you analyze it in a statistical and usage-based way, but which isn't so clear if you're focused only on the categorical formal structure.

Re: When ChatGPT broke the field of NLP: An oral history

#163

Post by ahz001 »

teruakohatu wrote: post #31

I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…

I work in a non-profit and continue to use traditional NLP for the same reasons. I have lots of text, and LLMs are expensive. Also, our organization has restrictive policies on AIs, especially LLMs.

I try to get the best of both words by using LLMs to generate synthetic data to train NLP classifiers. First, I use LLMs to generate variations of human-labeled data. Second, I use LLMs to label unlabeled data.

In a future challenge, I want to train LLMs to generate data to train NER for segmenting documents and extracting information.

Re: When ChatGPT broke the field of NLP: An oral history

#165

Post by aidenn0 »

wahern wrote: post #93
simonw wrote: post #35

Earlier quoted context omitted.

It's called the AI effect: https://en.wikipedia.org/wiki/AI_effect > The author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."

The flip side of that is that every time a new AI approach becomes popular even more people proclaim "this is what thinking is", believing that the new technology reflects the underlying process of human intelligence. This phenomenon goes back further than AI as a formal discipline, to early computers, and even during the age of mechanical computers. There are parallels with robotics, where for centuries anything tha…

> Contemporary machine learning may rightly be called artificial intelligence, but to conflate it with human intelligence is folly. It's clearly not human intelligence. It's something else. The same way dolphin intelligence isn't human intelligence, or a calculator isn't human intelligence. These things may be able to tell us something about the contours and limits of human intelligence, especially in contrast, but equivocations or even simple direct comparisons only serve to obfuscate and constrain how we think of intelligence.

This is something I mostly agree with. One quibble:

The process in LLMs clearly differs from human intelligence, but lumping it in with a the intelligence of a calculator is, IMO, making a mistake in the opposite direction.

Re: When ChatGPT broke the field of NLP: An oral history

#166

Post by chunkmonke99 »

charcircuit wrote: post #143
Al-Khwarizmi wrote: post #140

Earlier quoted context omitted.

Yes, my comment focused on NLP research but the importance of university teaching has also taken a hit - not that I fear bullying, but now students can have a dedicated custom teacher with infinite time and patience and that can answer questions at 3 AM, and obviously that reduces the relevance of the professor. While the human interaction in in-person teaching still provides some exclusive value, demand logically sh…

>but they wouldn't have been possible without the academics that kept working on neural networks while that field was actively scorned during the 90s-2000s AI winter The bottleneck was on compute power. Industry would have also worked on neural networks once the compute power for it existed.

I think the point is that universities and public research broadly speaking tolerate a lot of risk that private institution are not built to shoulder. I'm no AI historian but ANNs were an intellectual backwater after the 1st/2nd AI Winter and the only entities "foolish" enough to keep investing in them were Canadian public research universities. If it weren't for Hinton's dogged quasi-autistic special obsession with them ... no for profit entity would have bothered to invest in "computer power".

Re: When ChatGPT broke the field of NLP: An oral history

#167

Post by chunkmonke99 »

horsh1 wrote: post #141
simianwords wrote: post #139

Earlier quoted context omitted.

Not totally related but I have wondered how someone who thinks they are an expert in a field may deal with contradictions presented by GPT. For example, you may consider yourself an expert on some niche philosophy like say Orientalism. A student can now contradict any theories you can come up with using GPT and the scary thing is that they will be sensible contradictions. I feel like the bar to consider yourself an e…

Why in this story students use GPT and professors don't? If you are an expert -- sit down and start working with GPT on your own. See what it can and what it can not do. See where it helps. See where it hands down lose.

Yeah I agree! I use GPTs (Gemini 2 series and o3 etc.) and they are excellent! But even they sometimes don't quite get the nuance of things or subtly misses the point! There are certain "meta-cognitive" limitations ... I would never bet against the AGI industry and I can only assume that these issues will eventually be solved after we build planets of computronium and force every word and thought any human ever utters to be recorded and trained on. But for now there remain some limitations.

Re: When ChatGPT broke the field of NLP: An oral history

#168

Post by chunkmonke99 »

Al-Khwarizmi wrote: post #132

As an NLP professor, yes, I think we're mostly screwed - saying LLMs are a dead end or not a big deal, like some of the interviewed say, is just wishful thinking. A lot of NLP tasks that were subject of active research for decades have just been wiped out. Ironically, the tasks that still aren't solved well by LLMs and can still have a few years of life in them are the most low-level ones, that had become unfashionab…

[flagged]

Re: When ChatGPT broke the field of NLP: An oral history

#169

Post by suddenlybananas »

canjobear wrote: post #162
suddenlybananas wrote: post #137

Earlier quoted context omitted.

This paper is awful. They bizarrely argue the fact that transformers are not very sensitive to word order as a positive of transformers despite the fact that's not how languages work. There's also this absurd passage. >However, a closer look at the statistical structure of language use reveals that word order contains surprisingly little information over and above lexical information. To see this intuitively, imagine…

I don't see how it's absurd. The point isn't that languages don't have word order constraints, it's that they're organized in such a way that word order is usually redundant with other things when it comes to expressing meaning. This redundancy is a real and nontrivial property of language which becomes obvious when you analyze it in a statistical and usage-based way, but which isn't so clear if you're focused only o…

The redundancy is not a property of language it's a property of communication. This just speaks to the pseudoscientific nature of usage based linguistics; it's as if physicists claimed that heavier objects fell faster than lighter ones because a piece of paper will fall slower than a brick. You can't just look at the most frequent cases, you need to look at the actual edge cases to understand a phenomenon.

Re: When ChatGPT broke the field of NLP: An oral history

#170

Post by AndrewKemendo »

skissane wrote: post #110
AndrewKemendo wrote: post #74

Earlier quoted context omitted.

Humans are basically incapable of recognizing that there’s something that’s more powerful than them They’re never going to actively collectively admit that that’s the case, because humans collectively are so so systematically arrogant and self possessed that they’re not even open to the possibility of being lower on the intelligence totem pole The only possible way forward for AI is to create the thing that everybody…

> Humans are basically incapable of recognizing that there’s something that’s more powerful than them > They’re never going to actively collectively admit that that’s the case, because humans collectively are so so systematically arrogant and self possessed that they’re not even open to the possibility of being lower on the intelligence totem pole For most of human history, the clear majority of humans have believed…

Religious people - depending on type - use the God as cosplay or as some kind of deus ex machina that they are merged with in their mind.

Ask a deeply religious person about the separation between themselves and God and each tradition has their own version of being one with God or that they’ll become God etc…

It’s all the same because it’s ultimately about their relationship with how they define their God - having a “religious experience” where you realize “you are a part of god and get benefit etc..” is frankly a requirement to be a religion

In part that’s why it’s hard to call some versions of Buddhism religions and even harder for eg Hindus

Post reply on HN