Live data from Hacker News

When ChatGPT broke the field of NLP: An oral history

quantamagazine.org

61–70 of 181 posts

Re: When ChatGPT broke the field of NLP: An oral history

#61

Great seeing Ray Mooney (who I took a graduate class with) and Emily Bender (a colleague of many at the UT Linguistics Dept., and a regular visitor) sharing their honest reservations with AI and LLMs. I try to stay as far away from this stuff as possible because when the bottom falls out, it's going to have devastating effects for everyone involved. As a former computational linguist and someone who built similar too…

Don't try and say anything pro-linguistics here, people are weirdly hostile if you think it's anything but probabilities.

Over my years in academia, I noticed that the linguistics departments were always the most fiercely ideological. Almost every comment of a talk would be get contested by somebody from the audience.

It was annoying, but as a psych guy I was also jealous of them for having such clearly articulated theoretical frameworks. It really helped them develop cohesive lines of research to delineate the workings of each theory

Re: When ChatGPT broke the field of NLP: An oral history

#62

Great seeing Ray Mooney (who I took a graduate class with) and Emily Bender (a colleague of many at the UT Linguistics Dept., and a regular visitor) sharing their honest reservations with AI and LLMs. I try to stay as far away from this stuff as possible because when the bottom falls out, it's going to have devastating effects for everyone involved. As a former computational linguist and someone who built similar too…

How exactly is the bottom going to fall out? And are you really trying to present that you have practical experience building comparable tools to an LLM prior to the Transformer paper being written?

Now, there does appear to be some shenanigans going on with circular financing involving MSFT, NVIDIA, and SMCI (https://x.com/DarioCpx/status/1917757093811216627), but the usefulness of all the modern LLMs is undeniable. Given the state of the global economy and the above financial engineering issues I would not be surprised that at some point there isn't a contraction and the AI hype settles down a bit. With that said, LLMs could be made illegal and people would still continue running open source models indefinitely and organizations will build proprietary models in secret, b/c LLMs are that good.

Since we are throwing out predictions, I'll throw one out. Demand for LLMs to be more accurate will bring methods like formal verification to the forefront and I predict eventually model/agents will start to be able to formalize solved problems into proofs using formal verification techniques to guarantee correctness. At that point you will be able to trust the outputs for things the model "knows" (i.e. has proved) and use the probably correct answers the model spits out as we currently do today.

Probably something like the following flow:

1) Users enter prompts

2) Model answers questions and feeds those conversations to another model/program

3) Offline this other model uses formal verification techniques to try and reduce the answers to a formal proof.

4) The formal proofs are fed back into the first model's memory and then it uses those answers going forward.

5) Future questions that can be mapped to these formalized proofs can now be answered with almost no cost and are guaranteed to be correct.

Re: When ChatGPT broke the field of NLP: An oral history

#63

CNNs were outperforming traditional methods on some tasks before 2017. Problem was that all of the low level tasks , like part of speech tagging, parsing, named entity recognition , etc. never resulted in a good summarizing system or translating system. Probabilistic graphical models worked a bit but not much. Transformers were a leap, where none of the low level tasks had to be done for high level ones. Pretty sure…

> never resulted in a good ... translating system that seems too broad > all obliterated with an end to end cnn you mixed your nouns.. what you were saying about transformers was about transformers.. that specifically replaced cnn. So,no

In NLP, transformers replaced RNNs. In computer vision, CNNs replaced previous methods (e.g. feature descriptors), and recently got replaced by visual transformers, though modern CNNs are still pretty good.

Re: When ChatGPT broke the field of NLP: An oral history

#64
post #60

Has there been an LLM that reliably does not ignore the word "not"? Because I'm pretty sure that's a regression compared to most prior NLP.

> Has there been an LLM that reliably does not ignore the word "not"?

Curious. I would expect most of them to get that right, unless it's an intentionally tricky question. Do you have an example?

Re: When ChatGPT broke the field of NLP: An oral history

#65
post #56

I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…

*consigned a good chunk of historical research into the bin

To be fair it is still stuck there, so

Re: When ChatGPT broke the field of NLP: An oral history

#66

I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…

Sentiment analysis using traditional means is really lacking. I can’t talk about the current project I’m working on. But I needed a more nuanced sentiment. Think of something like people commenting on the Uber Eats app versus people commenting on a certain restaurant.

Re: When ChatGPT broke the field of NLP: An oral history

#67

Earlier quoted context omitted.

> never resulted in a good ... translating system that seems too broad > all obliterated with an end to end cnn you mixed your nouns.. what you were saying about transformers was about transformers.. that specifically replaced cnn. So,no

In NLP, transformers replaced RNNs. In computer vision, CNNs replaced previous methods (e.g. feature descriptors), and recently got replaced by visual transformers, though modern CNNs are still pretty good.

yikes! yes typing too fast..

Re: When ChatGPT broke the field of NLP: An oral history

#68
post #35
post #4

Earlier quoted context omitted.

Some people will never be convinced that a machine demonstrates intelligence. This is because for a lot of people, intelligence exists a subjective experience that they have and the belief that others have it too is only inasmuch as others appear to be like the self.

It's called the AI effect: https://en.wikipedia.org/wiki/AI_effect > The author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."

> somebody figured out how to make a computer do something

Well, I would argue that in most deterministic AI systems the thinking was all done by the AI researchers and then encoded for the computer. That’s why historically it’s been easy to say, “No, the machine isn’t doing any thinking, but only applying thinking that’s embedded within.” I think that line of argument becomes less obvious when you have learning systems where the behavior is training dependent. It’s still fairly safe to argue that the best LLMs today are not yet thinking, at least not in a way a human does. But in another generation or two? It will become much harder to deny.

Re: When ChatGPT broke the field of NLP: An oral history

#69
post #10

Earlier quoted context omitted.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

This is pretty much correct. I'd have to search for it but I remember an article from a couple years back that detailed how LLMs blew up the field of NLP processing overnight. Although I'd also offer a slightly different lens through which to look at the reaction of other researchers. There's jealousy, sure, but overnight a ton of NLP researchers basically had to come to terms with the fact that their research was us…

Yes, I think a lot of NLP folks must’ve had their “God does not play dice with the univers(al grammar)” moment.

Re: When ChatGPT broke the field of NLP: An oral history

#70

Earlier quoted context omitted.

> using VADER over something like DistilBERT is NLP malpractice in 2025. Ouch. Was that necessary? I used $1000 worth of GPU credits and threw in VADER because it’s basically free both in time and credits. I usually do this on large dataset out of pure interest in how it correlates with expensive methods on English language text. I am well aware of how VADER works and its limitations, I am also aware of the limitatio…

Curious how big your dataset was if you used $1000 of GPU credits on DistilBERT. I've run BERT on CPU on moderate cloud instances no problem for datasets I've worked with, but which admittedly are not huge.

If I'm reading correctly, they used $1000 running a Llama model, not DistilBERT.
Post reply on HN