Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When ChatGPT broke the field of NLP: An oral history

quantamagazine.org

Post reply on HN
Print view View original

21–30 of 181 posts

Re: When ChatGPT broke the field of NLP: An oral history

#21

Post by Tainnor »

dekhn wrote: post #10
sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

I agree with criticism of Noam Chomsky as a linguist. I was raised in the typological tradition which has its very own kind of beef with Chomsky due to other reasons (his singular focus on English for constructing his theories amongst other things), but his dislike of statistical methods was of course equally suspect.

Nevertheless there is something to be said for classical linguistic theory in terms of constituent (or dependency) grammars and various other tools. They give us much simpler models that, while incomplete, can still be fairly useful at a fraction of the cost and size of transformer architectures (e.g. 99% of morphology can be modeled with finite state machines). They also let us understand languages better - we can't really peek into a transformer to understand structural patterns in a language or to compare them across different languages.

Re: When ChatGPT broke the field of NLP: An oral history

#22

Post by hn_throwaway_99 »

dekhn wrote: post #10
sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

This is pretty much correct. I'd have to search for it but I remember an article from a couple years back that detailed how LLMs blew up the field of NLP processing overnight.

Although I'd also offer a slightly different lens through which to look at the reaction of other researchers. There's jealousy, sure, but overnight a ton of NLP researchers basically had to come to terms with the fact that their research was useless, at least from a practical perspective.

For example, imagine you just got your PhD in machine translation, which took you 7 years of laboring away in grad/post grad work. Then something comes out that can do machine translation several orders of magnitude better than anything you have proposed. Anyone can argue about what "understanding" means until they're blue in the face, but for machine translation, nobody really cares that much - people just want to get text in another language that means the same thing as the original language, and they don't really care how.

Tha majority of research leads to "dead ends", but most folks understand that's the nature of research, and there is usually still value in discovering "OK, this won't work". Usually, though, this process is pretty incremental. With LLMs all of a sudden you had lots of folks whose life work was pretty useless (again, from a practical perspective), and that'd be tough for anyone to deal with.

Re: When ChatGPT broke the field of NLP: An oral history

#23

Post by bpodgursky »

sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

> No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

Well, they're unhappy that an unrelated field of research more-or-less accidentally solved NLP. All the specialized NLP techniques people spent a decade developing were obviated by bigger deep learning models.

Re: When ChatGPT broke the field of NLP: An oral history

#24

Post by Workaccount2 »

dekhn wrote: post #13
mistrial9 wrote: post #12

Earlier quoted context omitted.

> most effective way to make forward progress powerful response but.. "fit for what purposes" .. All of human writings are not functionally equivalent. This has been discussed at length. e.g. poetry versus factual reporting or summation..

https://www.amazon.com/dp/B0DYDGZTMV makes the case that DeepSeek is a poet.

At least the author is upfront that the poetry is a showcase of AI.

Re: When ChatGPT broke the field of NLP: An oral history

#25

Post by levocardia »

dekhn wrote: post #10
sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

Sounds like the bitter lesson is bitter indeed!

Re: When ChatGPT broke the field of NLP: An oral history

#26

Post by suddenlybananas »

Tainnor wrote: post #21
dekhn wrote: post #10

Earlier quoted context omitted.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

I agree with criticism of Noam Chomsky as a linguist. I was raised in the typological tradition which has its very own kind of beef with Chomsky due to other reasons (his singular focus on English for constructing his theories amongst other things), but his dislike of statistical methods was of course equally suspect. Nevertheless there is something to be said for classical linguistic theory in terms of constituent (…

That is simply false about UG only being based on English. Maybe in 1950 but any modern generativist theory uses data from many, many languages and English has been re-analysed in light of other languages (see here for an example of quantifiers being analysed in English on the basis of data in a Salish language https://philpapers.org/rec/MATQAT )

Re: When ChatGPT broke the field of NLP: An oral history

#27

Post by foobarian »

dekhn wrote: post #10
sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

The progression reminds me of how brute force won out in the chess AI game long ago with Deep Blue. Custom VLSI and FPGA acceleration and all.

Re: When ChatGPT broke the field of NLP: An oral history

#28

Post by suddenlybananas »

languagehacker wrote: post #5

Great seeing Ray Mooney (who I took a graduate class with) and Emily Bender (a colleague of many at the UT Linguistics Dept., and a regular visitor) sharing their honest reservations with AI and LLMs. I try to stay as far away from this stuff as possible because when the bottom falls out, it's going to have devastating effects for everyone involved. As a former computational linguist and someone who built similar too…

Don't try and say anything pro-linguistics here, people are weirdly hostile if you think it's anything but probabilities.

Re: When ChatGPT broke the field of NLP: An oral history

#29

Post by macleginn »

dekhn wrote: post #10
sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…

The way I have experienced this, starting from circa 2018, it was a bit more incremental. First, LSTMs and then transformers lead to new heights on the old tasks, such as syntactic parsing and semantic role labelling, which was sad for the previous generation, but at least we were playing the same game. But then not only the old tools of NLP, but the research questions themselves became irrelevant because we could just ask a model nicely and get good results on very practical downstream tasks that didn't even exist a short while ago. NLP suddenly turned into general document/information processing field, with a side hustle in conversational assistants. Already GPT2 essentially mastered the grammar of English, and what difficulties remain are super-linguistic and have more to do with general reasoning. I would say that it's not that people are bitter that other people make progress, it's more that there is not much progress to be had in the old haunts at all.

Re: When ChatGPT broke the field of NLP: An oral history

#30

Post by rdedev »

sp1nningaway wrote: post #7

For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.

It's a truly bitter pill to swallow when your whole area of research goes redundant.

I have a bit of background in this field so it's nice to see even people who were at the top of the field raise concerns that I had. That comment about LHC was exactly what I told my professor. That the whole field seems to be moving in a direction where you need a lot of resources to do anything. You can have 10 different ideas on how to improve LLMs but unless you have the resources there is barely anything you can do.

NLP was the main reason I pursued an MS degree but by the end of my course I was not longer interested in it mostly because of this.

Post reply on HN