I’m curious how have large language models impacted linguistics and particularly the idea of a universal grammar?
There's a lot of debate about it. Here's one view: https://arxiv.org/abs/2501.17047
When ChatGPT broke the field of NLP: An oral history
81–90 of 181 posts
Re: When ChatGPT broke the field of NLP: An oral history
#82I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…
a) you can save costs on llama by running it locally
b) compute costs are plummeting. inference in the cloud costs has dropped over 80% in 1 year
c) similar to a), spending a little more and having a beefy enough machine is functionally cheaper after just a few projects
d) everyone trying to do sentiment analysis is trying to make waaaay more money anyway
so I dont see NLP’s even lower costs of being that relevant. its like pointing out that I could use assembly instead of 10 layers of abstraction. It doesnt really matter
Re: When ChatGPT broke the field of NLP: An oral history
#83Earlier quoted context omitted.
The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…
I think you greatly understate the impact as EVERYONE is freaking the fuck out about AI, not just NLP researchers. AI is obliterating the usefulness of all mental work. Look at the high percentage of HN articles trying to figure out whether LLMs can eliminate software developers. Or professional writers. Or composers. Or artists. Or lawyers. Focusing on the NLP researchers really understates the scope of the insecuri…
As someone in NLP who lived through this experience: there's something uniquely ironic and cruel about building the wave that washes yourself away.
Re: When ChatGPT broke the field of NLP: An oral history
#84For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.
The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…
I wouldn't say NLP as a field was resistant to probabilistic approaches or even neural networks. From maybe 2000-2018 almost all the papers were about using probabilistic methods to figure out word sense disambiguation or parsing or sentiment analysis or whatever. What changed was that these tasks turned out not to be important for the ultimate goal of making language technologies. We thought things like parsing were going to be important because we thought any system that can understand language would have to do so on the basis of the parse tree. But it turns out a gigantic neural network text generator can do nearly anything we ever wanted from a language technology, without dealing with any of the intermediate tasks that used to get so much attention. It's like the whole field got short-circuited.
Re: When ChatGPT broke the field of NLP: An oral history
#85My view is that "traditional" NLP will get re-incorporated into LLMs (or their successors) over time. We just didn't get to it yet. Appropriate inductive biases will only make LLMs better, faster and cheaper.
There will always be trouble in LLM "paradise" and desire to take it to the next level. Use raw-accessed (highest performing) LLM, intensely, for coding and you will rack up $10-$20/hr bill. China is not supposed to have adequate GPUs at their disposal, - they will come up with smaller and more efficient models. Etc, etc, etc...
Re: When ChatGPT broke the field of NLP: An oral history
#86I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…
I’d love to hear your thoughts on BERTs - I’ve dabbled a fair bit, fairly amateurishly, and have been astonished by their performance.
I’ve also found them surprisingly difficult and non-intuitive to train, eg deliberately including bad data and potentially a few false positives has resulted in notable success rate improvements.
Do you consider BERTs to be the upper end of traditional - or, dunno, transformer architecture in general to be a duff? Am sure you have fascinating insight on this!
Re: When ChatGPT broke the field of NLP: An oral history
#87I am in academia and worked in NLP although I would describe myself as NLP adjacent. I can confirm LLMs have essentially confined a good chunk of historical research into the bin. I suspect there are probably still a few PhD students working on traditional methods knowing full well a layman can do better using the mobile ChatGPT app. That said traditional NLP has its uses. Using the VADER model for sentiment analysis…
It currently costs around $2200 to run Gemini flash lite on all of Wikipedia English. It would probably cost around 10x that much to run sentiment analysis on every Yelp review ever posted. It's true that LLMs still cost a lot for some use cases, but for essentially any business case it's not worth using traditional NLP any more
Re: When ChatGPT broke the field of NLP: An oral history
#88Earlier quoted context omitted.
Don't try and say anything pro-linguistics here, people are weirdly hostile if you think it's anything but probabilities.
Over my years in academia, I noticed that the linguistics departments were always the most fiercely ideological. Almost every comment of a talk would be get contested by somebody from the audience. It was annoying, but as a psych guy I was also jealous of them for having such clearly articulated theoretical frameworks. It really helped them develop cohesive lines of research to delineate the workings of each theory
Are there really all that many parallels between linguistics (the study of langauge) and computational-linguistics/NLP (subject of discussion)?
Re: When ChatGPT broke the field of NLP: An oral history
#89Great seeing Ray Mooney (who I took a graduate class with) and Emily Bender (a colleague of many at the UT Linguistics Dept., and a regular visitor) sharing their honest reservations with AI and LLMs. I try to stay as far away from this stuff as possible because when the bottom falls out, it's going to have devastating effects for everyone involved. As a former computational linguist and someone who built similar too…
How exactly is the bottom going to fall out? And are you really trying to present that you have practical experience building comparable tools to an LLM prior to the Transformer paper being written? Now, there does appear to be some shenanigans going on with circular financing involving MSFT, NVIDIA, and SMCI ( https://x.com/DarioCpx/status/1917757093811216627 ), but the usefulness of all the modern LLMs is undeniabl…
> And are you really trying to present that you have practical experience building comparable tools to an LLM prior to the Transformer paper being written?
I believe (could be wrong) they were talking about their prior GOFAI/NLP experience when referencing scaling systems.
In any case, is it really necessary to be so harsh about over-confidence and then go on to predict the future of solving hallucinations with your formal verification ideas?
Talk is cheap. Show me the code.
Re: When ChatGPT broke the field of NLP: An oral history
#90Earlier quoted context omitted.
Or Planck's principle - "Science progresses one funeral at a time".
And the more general version, “Humanity progresses one funeral at a time.” Which is why the hyper-longevity people are basically trying to freeze all human progress.
Or Effective Altruism's long-termism that effectively makes everyone universally poor now. Interestingly, Guillaume Verdon (e/acc) is friends with Bryan Johnson and seems to be pro-longevity.