Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When ChatGPT broke the field of NLP: An oral history

quantamagazine.org

Post reply on HN
Print view View original

151–160 of 181 posts

Re: When ChatGPT broke the field of NLP: An oral history

#151

Post by whoaann_92 »

GardenLetter27 wrote: post #131
whoaann_92 wrote: post #121

As someone deeply involved in NLP, I’ve observed the field’s evolution: from decades of word counting and statistical methods to a decade of deep learning enabling “word arithmetic.” Now, with Generative AI, we’ve reached a new milestone, a universal NLP engine. IMHO, the path to scalability often involves using GPT models for prototyping and cold starts. They are incredible at generating synthetic data, which is inv…

Reminds me of the whole Chomsky vs. Norvig debate - https://norvig.com/chomsky.html

Thanks for the link, just read it, and the Chomsky transcript. Chomsky wanted deep structure, Norvig bet on stats, but maybe Turing saw it coming, kids talk before they know grammar and so did the machines. It turns out we didn’t need to understand language to automate it.

Re: When ChatGPT broke the field of NLP: An oral history

#152

Post by karel-3d »

Al-Khwarizmi wrote: post #146
karel-3d wrote: post #144

Earlier quoted context omitted.

I used to study NLP but before transformers, and now I don't work with NLP/ML/LLMs at all. Can you explain to me this view? LLMs are NLP? We have a model that works, and that works great for many people and many different usages, shouldn't NLP be at its top now? LLMs are not conceptionally that different to other models? I worked with GIZA/MOSES statistical MT back in the day during my studies, it's at the end of the…

NLP is indeed at its top, NLP professors aren't :) Imagine if you had stayed in academia and kept working in MT for the last two decades. First of all, now you would see how LLMs render all your work pretty much obsolete. That's already hard for many people. Not so much for me, as I consider myself to be rather adaptable, and maybe not for you either - you can embrace the new thing and start working on it yourself, r…

Ah, okay. The scale is just too big for academia, makes sense.

I remember that in my study days (just masters, not PhD) we have been unable to beat Google Translate already for general tasks, but we could beat it on weird language pairs, just because we could get more training data (mostly by downloading some ebooks online) and tinker with the algorithm a bit.

But the scale argument - more data will be better - was true even back then... it was just easy to get better training data than Google for weird subtasks.

(Actually one of my professors is now Big Guy of that "European Commission Approved LLM" research project that was in news few months ago. I am ... interested how that turns out. https://openeurollm.eu/ )

Re: When ChatGPT broke the field of NLP: An oral history

#153

Post by cainxinth »

Lots of great quotes in this piece but this one stuck out for me:

> TAL LINZEN: It’s sometimes confusing when we pretend that there’s a scientific conversation happening, but some of the people in the conversation have a stake in a company that’s potentially worth $50 billion.

Re: When ChatGPT broke the field of NLP: An oral history

#154

Post by tomrod »

ActorNightly wrote: post #124
vjerancrnjak wrote: post #44

CNNs were outperforming traditional methods on some tasks before 2017. Problem was that all of the low level tasks , like part of speech tagging, parsing, named entity recognition , etc. never resulted in a good summarizing system or translating system. Probabilistic graphical models worked a bit but not much. Transformers were a leap, where none of the low level tasks had to be done for high level ones. Pretty sure…

There was no leap in research. Everything had to do with availability of compute. Neural nets are quite old, and everyone knew that they were universal function approximators. The reason why models never took off was because it was very expensive to train a model even of a limited size. There was no real available hardware to do this on short of supercomputer clusters, which were just all cpus, and thus wildly ineffi…

> ever get like ASIC for ml

Is this what you're mentioning?

[0] https://linearmicrosystems.com/using-asic-chips-for-artifici...

Re: When ChatGPT broke the field of NLP: An oral history

#155

Post by drpbl »

ahnick wrote: post #62
languagehacker wrote: post #5

Great seeing Ray Mooney (who I took a graduate class with) and Emily Bender (a colleague of many at the UT Linguistics Dept., and a regular visitor) sharing their honest reservations with AI and LLMs. I try to stay as far away from this stuff as possible because when the bottom falls out, it's going to have devastating effects for everyone involved. As a former computational linguist and someone who built similar too…

How exactly is the bottom going to fall out? And are you really trying to present that you have practical experience building comparable tools to an LLM prior to the Transformer paper being written? Now, there does appear to be some shenanigans going on with circular financing involving MSFT, NVIDIA, and SMCI ( https://x.com/DarioCpx/status/1917757093811216627 ), but the usefulness of all the modern LLMs is undeniabl…

I have argued the same theme elsewhere. Formal reasoning over LLM output is the next step for AI. Where do we go for funding?

Re: When ChatGPT broke the field of NLP: An oral history

#156

Post by AIPedant »

petesergeant wrote: post #115
wahern wrote: post #93

Earlier quoted context omitted.

The flip side of that is that every time a new AI approach becomes popular even more people proclaim "this is what thinking is", believing that the new technology reflects the underlying process of human intelligence. This phenomenon goes back further than AI as a formal discipline, to early computers, and even during the age of mechanical computers. There are parallels with robotics, where for centuries anything tha…

> It's clearly not human intelligence I don't think argument by assertion is appropriate where there's a lot of people who "clearly" believe that it's a good approximation of human intelligence. Given we don't understand how human intelligence works, asserting that one plausible model (a continuous journey through an embedding space held in neurons) that works in machines isn't how humans do it seems too strong.

It is demonstrably true that artificial neurons have nothing to do with cortical neurons in mammals[1] so even if this model of human intelligence is useful, transformers/etc aren't anywhere close to actually implementing the model faithfully. Perhaps by Turing completeness o3 or whatever has stumbled into a good implementation of this model, but that is implausible. o3 still wildly confabulates worse than any dementia patient, still lacks the robust sense of folk physics we see in infants, etc. (This is even more apparent in video generators, Veo2 is SOTA and it still doesn't understand object permanence or gravity.) It does not make sense to say something is a model of human intelligence if it can do PhD-level written tasks but is outsmarted by literal babies (also chimps, dogs, pigeons...)

AI people toss around the term "neuron" way too freely.

[1] https://www.quantamagazine.org/how-computationally-complex-i...

Re: When ChatGPT broke the field of NLP: An oral history

#157

Post by emp17344 »

AndrewKemendo wrote: post #2

If Chomsky was writing papers in 2020 his paper would’ve been “language is all you need.” That is clearly not true and as the article points out wide scale very large forecasting models beat that hypothesis that you need an actual foundational structure for language in order to demonstrate intelligence when in fact is exactly the opposite. I’ve never been convinced by that hypothesis if for no other reason that we ca…

> As we’re finding: solving the markov process iteratively is the foundation of intelligence

No, the markov process allows LLMs to make connections between existing representations of human intelligence. LLMs are nothing without a data set developed by an existing intelligence.

Re: When ChatGPT broke the field of NLP: An oral history

#158

Post by AndrewKemendo »

suddenlybananas wrote: post #136
AndrewKemendo wrote: post #74

Earlier quoted context omitted.

Humans are basically incapable of recognizing that there’s something that’s more powerful than them They’re never going to actively collectively admit that that’s the case, because humans collectively are so so systematically arrogant and self possessed that they’re not even open to the possibility of being lower on the intelligence totem pole The only possible way forward for AI is to create the thing that everybody…

Are you really that misanthropic that you think we need AI to show us how meaningless we are?

I’m curious as to how you derive your position of “misanthropic”

The answer to your question though is I do not believe that humans have at any point in history prevented an existential threat from actually being realized before it was realized

Re: When ChatGPT broke the field of NLP: An oral history

#159

Post by aidenn0 »

coffeeaddict1 wrote: post #40
aidenn0 wrote: post #15

Earlier quoted context omitted.

I can't write something as good as Bob Dylan and Tom Petty. Ergo I'm not intelligent.

This to me is a weak argument. You have the ability to appreciate and judge something as good as Bob Dylan and Tom Petty. That's what makes you intelligent.

1. I'm sure if I were to ask an LLM for opinions on Dylan and Petty, it would provide them.

2. I don't know if this was the point the original was making, but I personally think Dylan is a bit overrated as a songwriter (and the one time I saw him live, he was only so-so as a performer, but I don't think that's exactly a hot take).

Post reply on HN