I’m curious how have large language models impacted linguistics and particularly the idea of a universal grammar?
When ChatGPT broke the field of NLP: An oral history
41–50 of 181 posts
Re: When ChatGPT broke the field of NLP: An oral history
#42For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.
The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…
do transformers not use a symbolic and a probabilistic approach?
Re: When ChatGPT broke the field of NLP: An oral history
#43For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.
The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…
Even 15-ish years ago when I was in school, the NLP folks viewed probabilistic models with suspicion. NLP treated everyone from our Math department with suspicion and gave them a hard time. It created so many politics that some folks who wanted to do statistical approaches would call themselves CS so that the NLP old guard wouldn't give them a hard time.
Re: When ChatGPT broke the field of NLP: An oral history
#44CNNs were outperforming traditional methods on some tasks before 2017.
Problem was that all of the low level tasks , like part of speech tagging, parsing, named entity recognition , etc. never resulted in a good summarizing system or translating system.
Probabilistic graphical models worked a bit but not much.
Transformers were a leap, where none of the low level tasks had to be done for high level ones.
Pretty sure that equivalent leap happened in computer vision a bit before.
People were fiddling with low level pattern matching and filters and then it was all obliterated with an end to end cnn .
Re: When ChatGPT broke the field of NLP: An oral history
#45"It helps to have tenure when something like this happens."
Re: When ChatGPT broke the field of NLP: An oral history
#46I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…
That's because VADER is just a dictionary mapping each word to a single sentiment weight and adding it up with some basic logic for negations and such. There's an ocean of smaller NLP ML between that naive approach and LLMs. LLMs are trained to do everything. If all you need is a model trained to do sentiment analysis, using VADER over something like DistilBERT is NLP malpractice in 2025.
Price isn't a real issue in almost every imaginable use case either. Even a small open source model would outperform and you're going to get a lot of tokens per dollar with that.
Re: When ChatGPT broke the field of NLP: An oral history
#47I was contrasting FiNER, GliNER, and Smolagents in a recent blog post on my substack and while the first two are fast and provide somewhat good results, running a LLM locally is 10x better easily.
Would love to read that post - we’re considering using GliNER for discrete parts of our ingestion pipeline where we assumed it would be a great perf/$ drop-in for larger models.
Re: When ChatGPT broke the field of NLP: An oral history
#48Great seeing Ray Mooney (who I took a graduate class with) and Emily Bender (a colleague of many at the UT Linguistics Dept., and a regular visitor) sharing their honest reservations with AI and LLMs. I try to stay as far away from this stuff as possible because when the bottom falls out, it's going to have devastating effects for everyone involved. As a former computational linguist and someone who built similar too…
Don't try and say anything pro-linguistics here, people are weirdly hostile if you think it's anything but probabilities.
The interesting question is whether just a gigantic set of probabilities somehow captures things about language and cognition that we would not expect ...
Re: When ChatGPT broke the field of NLP: An oral history
#49For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.
Well, if you’ve built a career on something, you will usually actively resist anything that threatens to destroy it. In other words, what is progress for the field might not be progress for you ! This reminds me of Thomas Kuhn’s excellent book ´the structure of scientific revolutions’ https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
It reminds me much more of Paul Feyerabend's even better book "Against Method" https://en.wikipedia.org/wiki/Against_Method
Re: When ChatGPT broke the field of NLP: An oral history
#50Earlier quoted context omitted.
This is why I want the field to go straight to building indistinguishable agents- specifically, you should be able to video chat with an avatar that is impossible to tell from a human. Then we can ask "if this is indistinguishable from a human, how can you be sure that anybody is intelligent?" Personally I suspect we can make zombies that appear indistinguishable from humans (limited to video chat; making a robot tha…
"There is considerable overlap between the intelligence of the smartest bears and the dumbest tourists." LLMs are not artificial intelligence but artificial stupidity . LLMs will happily hallucinate. LLMs will happily tell you total lies with complete confidence. LLMs will give you grammatically perfect completely vapid content. etc. And yet that is still better than what most humans could do in the same situation. W…
> LLMs will happily hallucinate. LLMs will happily tell you total lies with complete confidence.
Perhaps we should avoid anthropomorphizing them too much. LLMs don't inhabit a "real world" where they can experiment and learn. Their training data is their universe, and it's likely filled with conflicting, peculiar, and untestable information.
Yes, the output is sometimes "a lie" if we apply it to our world, but in "their world" stuff is might be just strangely different. And it's not like the real world has only "hard simple truths" - quantum mechanics comes to mind about how strange stuff can be.