Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When ChatGPT broke the field of NLP: An oral history

quantamagazine.org

Post reply on HN
Print view View original

131–140 of 181 posts

Re: When ChatGPT broke the field of NLP: An oral history

#131

Post by GardenLetter27 »

whoaann_92 wrote: post #121

As someone deeply involved in NLP, I’ve observed the field’s evolution: from decades of word counting and statistical methods to a decade of deep learning enabling “word arithmetic.” Now, with Generative AI, we’ve reached a new milestone, a universal NLP engine. IMHO, the path to scalability often involves using GPT models for prototyping and cold starts. They are incredible at generating synthetic data, which is inv…

Reminds me of the whole Chomsky vs. Norvig debate - https://norvig.com/chomsky.html

Re: When ChatGPT broke the field of NLP: An oral history

#132

Post by Al-Khwarizmi »

As an NLP professor, yes, I think we're mostly screwed - saying LLMs are a dead end or not a big deal, like some of the interviewed say, is just wishful thinking. A lot of NLP tasks that were subject of active research for decades have just been wiped out.

Ironically, the tasks that still aren't solved well by LLMs and can still have a few years of life in them are the most low-level ones, that had become unfashionable in the last 15 years or so - part-of-speech tagging, syntactic parsing, NER. Of course, they have lost a lot of importance as well: you no longer need them for user-oriented downstream tasks. But they may still get some use: for example NER for its own sake is used in biomedical domains, and parsing can be useful for scientific studies of language (say, language universals related to syntax, language evolution, etc.). Which is more than you can say about translation or summarization, which have been pretty much obsoleted by LLMs. Still, even with these tasks, NLP will go from broad applicability to niche.

I'm not too worried for my livelihood at the moment (partly because I have tenure, and partly because the academic system works in such a way that zombie fields keep walking for quite long - there are still journals and conferences on the semantic web, which has been a zombie for who knows how long). But it's a pity: I got into this because it was fun and made an impact and now it seems most of my work is going to be irrelevant, like those semantic web researchers I used to look down at. I guess my consolation is that computers that really understand human language was the dream that got me into this in the first place, and it has been realized early. I can play with it and enjoy it while I sink into irrelevant research, I guess :/ Or try to pivot into discrete math or something.

Re: When ChatGPT broke the field of NLP: An oral history

#133

Post by jsemrau »

philipkglass wrote: post #54
darkteflon wrote: post #47

Earlier quoted context omitted.

Would love to read that post - we’re considering using GliNER for discrete parts of our ingestion pipeline where we assumed it would be a great perf/$ drop-in for larger models.

This looks like the post: https://jdsemrau.substack.com/p/finer-gliner-and-smolagents-...

Thank you

Re: When ChatGPT broke the field of NLP: An oral history

#134

Post by coffeeaddict1 »

motorest wrote: post #92
coffeeaddict1 wrote: post #40

Earlier quoted context omitted.

This to me is a weak argument. You have the ability to appreciate and judge something as good as Bob Dylan and Tom Petty. That's what makes you intelligent.

> This to me is a weak argument. You have the ability to appreciate and judge something as good as Bob Dylan and Tom Petty. That's what makes you intelligent. What if you don't? Do you think that makes someone not intelligent? Think about it for a second.

Yes. If you do not possess the potential ability to judge other human beings and/or their work, you lack intelligence.

Re: When ChatGPT broke the field of NLP: An oral history

#135

Post by larodi »

Al-Khwarizmi wrote: post #132

As an NLP professor, yes, I think we're mostly screwed - saying LLMs are a dead end or not a big deal, like some of the interviewed say, is just wishful thinking. A lot of NLP tasks that were subject of active research for decades have just been wiped out. Ironically, the tasks that still aren't solved well by LLMs and can still have a few years of life in them are the most low-level ones, that had become unfashionab…

Let's admit it - overnight it became much much harder to be a convincing professor, given each student can use GPTs of all sorts to contradict or otherwise intimidate you. Only a seasoned professor knows the feeling of being bullied by a smart-ass student. Which brings down the total value, the incentive, to teach, and also given the avoidance GPTs silently imprint in students. I mean - why write a program/paper/research when the GPT can do it for you, and save you the suffering.

The whole area of algorithms suddenly became more challenging, as now you also have to understand folding multi-dimensional spaces, and retell this all as a nice story for students to remember.

We are very likely heading into some dark uncharted era for academia, which will very likely lead to academia shrinking massively. And given the talk of 'all science now happens in big corpos'... I can expect the universities to go back to the original state they started from - monasteries.

Saying this all having spent 20+ years as part-time contributor to one such monastery.

Re: When ChatGPT broke the field of NLP: An oral history

#136

Post by suddenlybananas »

AndrewKemendo wrote: post #74
aidenn0 wrote: post #4

Earlier quoted context omitted.

Some people will never be convinced that a machine demonstrates intelligence. This is because for a lot of people, intelligence exists a subjective experience that they have and the belief that others have it too is only inasmuch as others appear to be like the self.

Humans are basically incapable of recognizing that there’s something that’s more powerful than them They’re never going to actively collectively admit that that’s the case, because humans collectively are so so systematically arrogant and self possessed that they’re not even open to the possibility of being lower on the intelligence totem pole The only possible way forward for AI is to create the thing that everybody…

Are you really that misanthropic that you think we need AI to show us how meaningless we are?

Re: When ChatGPT broke the field of NLP: An oral history

#137

Post by suddenlybananas »

canjobear wrote: post #81
softwaredoug wrote: post #41

I’m curious how have large language models impacted linguistics and particularly the idea of a universal grammar?

There's a lot of debate about it. Here's one view: https://arxiv.org/abs/2501.17047

This paper is awful. They bizarrely argue the fact that transformers are not very sensitive to word order as a positive of transformers despite the fact that's not how languages work. There's also this absurd passage.

>However, a closer look at the statistical structure of language use reveals that word order contains surprisingly little information over and above lexical information. To see this intuitively, imagine we give you a set of words {dogs, bones, eat} without telling you the original order of the words. You can still reconstruct the meaning based entirely on (1) the meanings of the words in isolation and (2) your knowledge of how the world works—dogs usually eat bones; bones rarely eat dogs. Indeed, many languages show a high level of nondeterminism in word order (Futrell et al., 2015b; Koplenig et al., 2017), and word order cues are often redundant with meaning or case markers (Pijpops and Zehentner, 2022; Mahowald et al., 2023). The fact that word order is relatively uninformative in usage also partly explains why bag-of-words methods dominated NLP tasks until around 2020, consistently outperforming much more sophisticated approaches: it turns out that most of the information in sentences is in fact present in the bag of words.

While it is certainly possible to guess that your interlocutor meant "dogs eat bones", the sentence "bones eat dogs" is entirely possible (if unlikely)! For example, imagine a moving skeleton in a video game or something. The idea that word order isn't vital to meaning is deeply unserious. (Of course there are languages where word order matters less, but there are still important rules about constituent structure, clausal embedding etc, which constrain word order).

Re: When ChatGPT broke the field of NLP: An oral history

#138

Post by suddenlybananas »

qzw wrote: post #69
hn_throwaway_99 wrote: post #22

Earlier quoted context omitted.

This is pretty much correct. I'd have to search for it but I remember an article from a couple years back that detailed how LLMs blew up the field of NLP processing overnight. Although I'd also offer a slightly different lens through which to look at the reaction of other researchers. There's jealousy, sure, but overnight a ton of NLP researchers basically had to come to terms with the fact that their research was us…

Yes, I think a lot of NLP folks must’ve had their “God does not play dice with the univers(al grammar)” moment.

The majority of NLP people were not into universal grammar at all.

Re: When ChatGPT broke the field of NLP: An oral history

#139

Post by simianwords »

larodi wrote: post #135
Al-Khwarizmi wrote: post #132

As an NLP professor, yes, I think we're mostly screwed - saying LLMs are a dead end or not a big deal, like some of the interviewed say, is just wishful thinking. A lot of NLP tasks that were subject of active research for decades have just been wiped out. Ironically, the tasks that still aren't solved well by LLMs and can still have a few years of life in them are the most low-level ones, that had become unfashionab…

Let's admit it - overnight it became much much harder to be a convincing professor, given each student can use GPTs of all sorts to contradict or otherwise intimidate you. Only a seasoned professor knows the feeling of being bullied by a smart-ass student. Which brings down the total value, the incentive, to teach, and also given the avoidance GPTs silently imprint in students. I mean - why write a program/paper/rese…

Not totally related but I have wondered how someone who thinks they are an expert in a field may deal with contradictions presented by GPT.

For example, you may consider yourself an expert on some niche philosophy like say Orientalism. A student can now contradict any theories you can come up with using GPT and the scary thing is that they will be sensible contradictions.

I feel like the bar to consider yourself an expert is much higher - you not only have to beat your students but also know enough to beat GPT.

Re: When ChatGPT broke the field of NLP: An oral history

#140

Post by Al-Khwarizmi »

larodi wrote: post #135
Al-Khwarizmi wrote: post #132

As an NLP professor, yes, I think we're mostly screwed - saying LLMs are a dead end or not a big deal, like some of the interviewed say, is just wishful thinking. A lot of NLP tasks that were subject of active research for decades have just been wiped out. Ironically, the tasks that still aren't solved well by LLMs and can still have a few years of life in them are the most low-level ones, that had become unfashionab…

Let's admit it - overnight it became much much harder to be a convincing professor, given each student can use GPTs of all sorts to contradict or otherwise intimidate you. Only a seasoned professor knows the feeling of being bullied by a smart-ass student. Which brings down the total value, the incentive, to teach, and also given the avoidance GPTs silently imprint in students. I mean - why write a program/paper/rese…

Yes, my comment focused on NLP research but the importance of university teaching has also taken a hit - not that I fear bullying, but now students can have a dedicated custom teacher with infinite time and patience and that can answer questions at 3 AM, and obviously that reduces the relevance of the professor. While the human interaction in in-person teaching still provides some exclusive value, demand logically should go down. Although don't underestimate the power of inertia - one could also think that small noname universities would go out of business when the likes of MIT began offering their courses to everyone online, and it didn't happen. I do think LLMs bring higher risk than that, and a shrinking will indeed happen, but maybe not so dramatic. Let's see.

Regarding science, if we leave it exclusively to corporations we won't get very far, because most corporations aren't willing to do the basic/foundational science work. The Transformers and most of the relevant followup work that led to LLMs were developed in industry, but they wouldn't have been possible without the academics that kept working on neural networks while that field was actively scorned during the 90s-2000s AI winter. So I think research universities still should have a role to play. Of course, convincing the funders that this is indeed the case might be a different story.

Post reply on HN