71–80 of 181 posts
For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.
Earlier quoted context omitted.
Well, if you’ve built a career on something, you will usually actively resist anything that threatens to destroy it. In other words, what is progress for the field might not be progress for you ! This reminds me of Thomas Kuhn’s excellent book ´the structure of scientific revolutions’ https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
Or Planck's principle - "Science progresses one funeral at a time".
Earlier quoted context omitted.
The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…
This is pretty much correct. I'd have to search for it but I remember an article from a couple years back that detailed how LLMs blew up the field of NLP processing overnight. Although I'd also offer a slightly different lens through which to look at the reaction of other researchers. There's jealousy, sure, but overnight a ton of NLP researchers basically had to come to terms with the fact that their research was us…
Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in other words, someone with a great perspective on everything in the top article).
Re: Machine Translation, note that Transformers were introduced for this task, and built on one of the earlier notions of attention in sequence models: https://arxiv.org/abs/1409.0473 (2014, 38k citations)
That's not to say there weren't holdouts or people who really were "hurt" by a huge jump in MT capability - just that this is a logical progression in language understanding methods as seen by some folks (though who could have predicted the popularity of chat interfaces).
If Chomsky was writing papers in 2020 his paper would’ve been “language is all you need.” That is clearly not true and as the article points out wide scale very large forecasting models beat that hypothesis that you need an actual foundational structure for language in order to demonstrate intelligence when in fact is exactly the opposite. I’ve never been convinced by that hypothesis if for no other reason that we ca…
Some people will never be convinced that a machine demonstrates intelligence. This is because for a lot of people, intelligence exists a subjective experience that they have and the belief that others have it too is only inasmuch as others appear to be like the self.
They’re never going to actively collectively admit that that’s the case, because humans collectively are so so systematically arrogant and self possessed that they’re not even open to the possibility of being lower on the intelligence totem pole
The only possible way forward for AI is to create the thing that everybody is so scared of so they can actually realize their place in the universe
Earlier quoted context omitted.
It's called the AI effect: https://en.wikipedia.org/wiki/AI_effect > The author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."
> somebody figured out how to make a computer do something Well, I would argue that in most deterministic AI systems the thinking was all done by the AI researchers and then encoded for the computer. That’s why historically it’s been easy to say, “No, the machine isn’t doing any thinking, but only applying thinking that’s embedded within.” I think that line of argument becomes less obvious when you have learning syst…
I agree completely.
> But in another generation or two? It will become much harder to deny.
Unless there is something ... categorically different about what an LLM does and in a generation or two we can articulate what that is (30 years of looking at something makes it easier to understand ... sometimes).
For me as a lay-person, the article is disjointed and kinda hard to follow. It's fascinating that all the quotes are emotional responses or about academic politics. Even now, they are suspicious of transformers and are bitter that they were wrong. No one seems happy that their field of research has been on an astonishing rocketship of progress in the last decade.
The way I see this is that for a long time there was an academic field that was working on parsing natural human language and it was influenced by some very smart people who had strong opinions. They focused mainly on symbolic approaches to parsing, rather than probabilistic. And there were some fairly strong assumptions about structure and meaning. Norvig wrote about this: https://norvig.com/chomsky.html and I think…
AI is obliterating the usefulness of all mental work. Look at the high percentage of HN articles trying to figure out whether LLMs can eliminate software developers. Or professional writers. Or composers. Or artists. Or lawyers.
Focusing on the NLP researchers really understates the scope of the insecurity induced by AI.
If Chomsky was writing papers in 2020 his paper would’ve been “language is all you need.” That is clearly not true and as the article points out wide scale very large forecasting models beat that hypothesis that you need an actual foundational structure for language in order to demonstrate intelligence when in fact is exactly the opposite. I’ve never been convinced by that hypothesis if for no other reason that we ca…
Some people will never be convinced that a machine demonstrates intelligence. This is because for a lot of people, intelligence exists a subjective experience that they have and the belief that others have it too is only inasmuch as others appear to be like the self.
This ad hominem is really irritating. People have complained since Alan Turing that AI research ignores simpler intelligence, instead trying to bedazzle people with fancy tricks that convey the illusion of human intelligence. Still true today: lots of talk about o3's (exaggerated) ability to do fancy math, little talk about its appallingly bad general quantitative reasoning. The idea of "jagged AI" is unscientific horseshit designed to sweep this stuff under the rug.
Has there been an LLM that reliably does not ignore the word "not"? Because I'm pretty sure that's a regression compared to most prior NLP.
> Has there been an LLM that reliably does not ignore the word "not"? Curious. I would expect most of them to get that right, unless it's an intentionally tricky question. Do you have an example?
This particular example is too small to regularly trip AIs, but as a general rule I do not consider it tricky to try to textually negative-prompt to remove a commonly-valid-but-not-currently-wanted response. (Obviously, if you manually tweak the weights to forbid something rather than using the word "not", this fails.)
From my very rough observations, for models that fit on a local device, it typically starts to happen maybe 10% of the time when the prompt reaches 300 characters or so (specifying other parts of what you want); bigger models just need a bit more input before they fail. Reasoning models might be better, but we can watch them literally burn power running nonsensical variations through the thought pane so they're far from a sure answer.
This happens in any context you can think of: from scratch or extending an existing work; single or list; information lookup, prose generation, code generation (consider "Extend this web app to do lengthy-description-of-some-task. Remember I am not using React you stupid piece of shit AI!").
Earlier quoted context omitted.
Some people will never be convinced that a machine demonstrates intelligence. This is because for a lot of people, intelligence exists a subjective experience that they have and the belief that others have it too is only inasmuch as others appear to be like the self.
It's called the AI effect: https://en.wikipedia.org/wiki/AI_effect > The author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."
Well, there’s the clue it is not really thinking if somebody told the machine how to do things. My Roomba isn’t intelligent because it’s been programmed to clean the floor, now is it?
Wake me up when machines learn to do something on their own. I know everybody is on the AI hype train, but please show your extraordinary evidence to your extraordinary claims first.