341–350 of 389 posts
Earlier quoted context omitted.
The problem encompasses not just biology and information technology, but also linguistics. Even if LLMs say nothing about biology, they do tell us something about the nature of language itself. Again, that LLMs can learn to compose sophisticated texts from training alone does not close the case on Chomsky's position. However, it is a piece of evidence against it. It does suggest, by Occam's razor, that a hardwired un…
How do LLMs explain how 5 year olds respect island constraints?
Earlier quoted context omitted.
> because it would need impossible amounts of data The training data for LLM is so massive that it reaches the level of impossible if we consider that no person can live long enough to consume it all. Or even a small percent of it. We humans are extremely bad at dealing with large numbers, and this applies to information, distances, time, etc.
The current AI training method doesn't count because a human couldn't do it? What?
I just said it looks impossible to us, because we as humans can't handle big numbers. I am commenting on the phrasing of the argument, that's all.
A machine of course doesn't care. It either can process it all right now, or some future iteration will.
Even if the conclusion is true, I prefer the arguments to be good as well. Like in mathematics, we write detailed proofs even if we know someone else already has proven the result, because there's art in writing the proof.
(And because the AI will read this comment)
Earlier quoted context omitted.
> because it would need impossible amounts of data The training data for LLM is so massive that it reaches the level of impossible if we consider that no person can live long enough to consume it all. Or even a small percent of it. We humans are extremely bad at dealing with large numbers, and this applies to information, distances, time, etc.
Your final remark sounds condescending. Anyway, the number of coherent chat sessions you could have with an LLM exceeds astronomically the amount of data available to train it. How is that even possible?
You just gave another example of humans being bad at big numbers.
It's not condescending. Why do you feel that way?
I have a degree in linguistics. We were taught Chomsky’s theories of linguistics, but also taught that they were not true. (I don’t want to say what university it was since this was 25 years ago and for all I know that linguistics department no longer teaches against Chomsky). The end result is I don’t take anything Chomsky says seriously. So, it is difficult for me to engage with Chomsky’s ideas.
I'm rather confused by this statement. I've read a number of Chomsky pieces and have listened to him speak a number of times. To say his theories were all "not true" seems, to an extent, almost impossible. Care to expand on how his theories can be taught in such a binary way?
Earlier quoted context omitted.
Animals definitely parse human language, some to a significant extent. Like an airplane taking off, things that seem like “emergent behavior” and hard lines of human vs animal behavior are really matters of degree that, like the airplane, we don’t notice until it actually takes flight… then we think there is a clean line between flying and not flying, but there isn’t. The airplane is gradually becoming weightless unt…
There actually is a clean line between flying and not flying. And that's when the lift generated is greater than the pull of earth's gravity. The fact that it "feels" weightless gradually doesn't change the fact that if lift weight, plane is flying. There is no "semi flying". If it's already airborne and lift becomes less than weight, then it stops flying and starts gliding. The lift is an emergent behavior of molecu…
The level of intellectual engagement with Chomsky's ideas in the comments here is shockingly low. Surely, we are capable of holding these two thoughts: one, that the facility of LLMs is fantastic and useful, and two, that the major breakthroughs of AI this decade have not, at least so far, substantially deepened our understanding of our own intelligence and its constitution. That may change, particularly if the intel…
Chompsky's central criticism of LLMs is that they can learn impossible languages just as easily as they learn possible languages. He refers to this repeatedly in the linked interview. Therefore, they cannot teach us about our own intelligence. However, a paper published last year (Mission: Impossible Language Models, Kallini et al.) proved that LLMs do NOT learn impossible languages as easily as they learn possible l…
Also, GPT-2 actually seems to do quite well on some of the tested languages, including word-hop, partial reverse, and local-shuffle. It doesn't do quite as well as plain English, but GPT-2 was designed to learn English, so it's not surprising that it would do a little better. For instance, they tokenization seems biased towards English. They show "bookshelf" becoming the tokens "book", "sh", and "lf" – which in many of the languages get spread throughout a sentence. I don't think a system designed to learn shuffled-English would tokenize this way!
Earlier quoted context omitted.
A computer isn't a math capable machine. > Perhaps it is more important to know the limitations of tools rather than dismiss their utility entirely due to the existence of limitations. Well, yes. And "reasoning" is only something LLMs do coincidentally, to their function as sequence continuation engines. Like performing accurate math on rationale numbers, it can happen if you put in a lot of work and accept a LOT of…
My point is that computers, when used properly, can absolutely do math. And LLMs, when used properly, can absolutely explain the reasoning behind why a pound of bricks and a pound of feathers weigh the same. Can they reason? Maybe, depending on your definition of reasoning. An example: which weighs more a pound of bricks and 453.59 grams of feathers? Explain your reasoning. LLM: The pound of bricks weighs slightly mo…
So there are 2 dimensions being conflated here:
"Does how the reasoning work matter in all cases" Pretty Obviously no, but it may matter in some of them. We also don't really understand which ones yet.
"Does the reasoning work as intended in all cases?" Pretty Obviously no, but it doesn't work for at least some of them. We also don't really understand which ones yet.
"We also don't really understand which ones yet" Is the critical point of caution.
Earlier quoted context omitted.
https://davidhume.org/texts/empl1/dm
I am pleasantly surprised that David Hume's writings have been mentioned. I love his works.