Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

They’re made out of weights

maxleiter.com

Post reply on HN
Print view View original

711–720 of 739 posts

Re: They’re made out of weights

#711

Post by ludwik »

trick-or-treat wrote: post #659
ludwik wrote: post #483

Earlier quoted context omitted.

Is it? Both supervised learning and reinforcement learning are ways of training the model, and the difference between them is not that big. I would say that innate means "in the weights", while non-innate means things the model learned during inference, during its "lifetime".

Maybe you're right. In the weights might be the right way to frame that. What do you mean by "during its lifetime"? Do you mean things like system prompts or things in Claude.md? It sounds like you're framing a session as a "lifetime". Whch might be right, I haven't thought of it like that before though. So when I /compact my session what's that even the equivalent of I wonder.

> Do you mean things like system prompts or things in Claude.md?

All of it - system prompts, user prompts, few-shot examples, Claude.md, things that an agent learned by exploring its environment...

> So when I /compact my session what's that even the equivalent of I wonder.

Sleep? :)

Re: They’re made out of weights

#712

Post by simonh »

JdeBP wrote: post #582
yencabulator wrote: post #359

Earlier quoted context omitted.

I mean, they did, and we have, and we've also stopped doing that. https://en.wikipedia.org/wiki/Words_(Unix)

We should start telling them again, then. (-: In the current versions of FreeBSD, NetBSD, DragonFlyBSD, Illumos, and Debian, it is still /usr/share/dict . * https://cgit.freebsd.org/src/tree/share/dict/ * https://cvsweb.netbsd.org/bsdweb.cgi/src/share/dict/ * https://gitweb.dragonflybsd.org/?p=dragonfly.git;a=tree;f=sh... * https://cvsweb.openbsd.org/src/share/dict * https://refspecs.linuxfoundation.org/FHS_3.0/fhs/c…

Sure, but the fact that people are doing something isn't evidence that it isn't a mistake. Also they may be stuck due to concerns about backwards compatibility. There may be games and utilities they are shipping, that come from upstreams, that rely on these files.

Re: They’re made out of weights

#713

Post by f_klem »

ACCount37 wrote: post #708
f_klem wrote: post #704

Earlier quoted context omitted.

> Your references that "back that claim", which are in "books you mentioned", which you "mentioned" who knows where. Yeah, no. I'm not walking that chain. If you want to, do it, but for now, I'm filing it as "has no evidence and knows it". You are free not to believe me and dismiss the whole point. I do have evidence and I know it, no need to prove that (to begin with, the references are there. Read them if you want…

Knowing what exact algorithm "thinking" is isn't a requirement. Automata class is enough to say "a Turing machine can implement it". There are exactly two possibilities: thinking can be expressed as computation, or thinking requires hypercomputation. I did acknowledge both, explicitly. Which one? I'm betting hard against the second one, by the way. Because it requires hypercomputational magic fairy dust to: 1) exist…

> Knowing what exact algorithm "thinking" is isn't a requirement. Automata class is enough to say "a Turing machine can implement it".

I don't know what you are referring to by the word 'thinking'. But in any case, if you declare that it is not necessary to know the algorithm about thinking, how can you say then that a Turing machine can implement it? How can you say you implemented something you don't know how it works and how it is constituted? The only option I see then is that you implement something that is phenomenically identical to human intelligence, provided that you exhaust all possible combinations of human intelligence phenomena in a descriptive, extensional way (which, if you assume a finite extension of such phenomena, in any case, and most probably, gets you in the trouble of counting uncountable finite sets).

> There are exactly two possibilities: thinking can be expressed as computation, or thinking requires hypercomputation.

Again, if you do not define what 'thinking' is and how and on what assumptions it can be described as a computational process, this claim is empty.

So as far as I see it, you are still trapped by the assumption that the brain or mind are fundamentally similar to the kind of machines we can build.

> But that's the name of the game, isn't it? Anything but admitting that your mind is a glorified math construct implemented in wet meat.

Again here some assumptions operate, that tell you that the brain is some kind of hardware. And again: there is no real evidence that the body/consciousness 'construct' has any relation or analogy to the hardware/software/machine idea. Quite the contrary. Since the science that occupies itself on these topics is on the very frontier of knowledge and experimentation, reading science literature only will not clarify your thoughts. You will need additional guidance, and that guidance is called philosophy.

I recognize that the references I posted in my original comment are hard to read. But that's the point with the AI/mind debate: it is a tough, bitter topic. Just reading AI research won't bring anyone to the level this research space needs in order to discuss these topics.

Re: They’re made out of weights

#714

Post by ordu »

f_klem wrote: post #705
ordu wrote: post #688

Earlier quoted context omitted.

Let me start with: > if you read the sources on AI research through its own history, you will see how the AI research space is full of such biases and assumptions. You are repeating it in different forms all the time, so it seems really important to you. I should state, that I don't believe that AI research has some special insights into human nature or into what agency is. I'm sure that they have some biases and ass…

I'm not trying to convince anyone. I am just baffled that a large part of the tech/software and AI research community do not question their own assumptions, when those assumptions are being actively questioned in other fields (namely, philosophy and linguistics). Reaching AGI or human-level intelligence might be possible, but not on the basis of dismissing what other fields already said something about . That is arro…

> I am just baffled that a large part of the tech/software and AI research community do not question their own assumptions

Why are you so concerned about it? If philosophers and linguists are right, then what the sequence of evens should we expect? AI developments will slow down and stop, AI bubble will burst, and AI researchers will be humiliated and forced to accept their failings.

> That is arrogant, and does not help. Even more, this has already happened in the 60s/70s.

Exactly like that time. Or maybe we should say "those times". Doesn't matter really.

Re: They’re made out of weights

#715

Post by wisty »

slopinthebag wrote: post #93
wisty wrote: post #76

Earlier quoted context omitted.

AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either). Yeah, the weights not updating online makes them less like a living organism that ca…

> remove a brain from a body and it is no longer concious either What is no longer conscious, the brain? Or the body? Or some other entity? If consciousness is weakly emergent, how do we know it emerges from the solely from the brain and not, say, brain + body? Or brain + body + or environment. Or from the universe itself?

You can draw a line somewhere. Peole argue against conciousness by constructing (deliberately or not) an example where the line is drawn in a silly place.

Re: They’re made out of weights

#716

Post by LudwigNagasena »

f_klem wrote: post #703
LudwigNagasena wrote: post #678

Earlier quoted context omitted.

What's exactly the fallacy? How do the works help avoid stepping into that "fallacy" if they don't try to solve the issue of consciousness.

The issue of consciousness appears when you think of the world in a mechanistic way: since all there is are laws of physics and materiality, then how could we explain our though processes and our perceptual experience? If the world itself (in a general, existential way) is only made of laws of physics and matter, the consciousness needs to be an emergent characteristic of physical systems, and needs to follow the law…

Of course any research programme requires some assumptions. But I don’t see any reason to call it a fallacy. Saying that something may be “challenged” or is problematic is just weasel wording.

Either there are some serious issues that makes such theories ”flawed in the sense that they cannot account for subjective experience and agency, amongst other things”, or they are just normal theories.

Re: They’re made out of weights

#717

Post by f_klem »

LudwigNagasena wrote: post #716
f_klem wrote: post #703

Earlier quoted context omitted.

The issue of consciousness appears when you think of the world in a mechanistic way: since all there is are laws of physics and materiality, then how could we explain our though processes and our perceptual experience? If the world itself (in a general, existential way) is only made of laws of physics and matter, the consciousness needs to be an emergent characteristic of physical systems, and needs to follow the law…

Of course any research programme requires some assumptions. But I don’t see any reason to call it a fallacy. Saying that something may be “challenged” or is problematic is just weasel wording. Either there are some serious issues that makes such theories ”flawed in the sense that they cannot account for subjective experience and agency, amongst other things”, or they are just normal theories.

True, any research programme requires assumptions. The problem lies when those assumptions are either false or theoretical (unproved), and the community derives facts or claims from them.

Behind the actual AI programme operate the following assumptions (at least):

1. A biological assumption, that states that the brain works similar to a digital computer. The reality is that we do not know.

2. An epistemological assumption, that states that we know how our brain works (or an even worse assumption, that states that we don't even need to know how it works, it is sufficient to replicate its observed behavior). This is rather simplified, the assumption in reality being (as stated by Dreyfus) that we think all intelligent behavior can be formalized as heuristic rules (Dreyfus' critique is based on GOFAI, since the book is pre-GAN/RL AI systems). But the assumption still applies: we think all intelligent behavior can be sampled, captured and formalized in (albeit complex) statistical systems.

Dreyfus describes 4 or 5 in total, one of them is the psychological assumption, which states that the mind itself can be described as a digital computer (I think it might be outdated, since the actual debate is if something we could call 'mind' exists at all).

There is also a fallacy called first-step fallacy, which states that if the first step towards intelligence is met, then the rest of the steps are of similar nature (technical).

Re: They’re made out of weights

#718

Post by f_klem »

ozgung wrote: post #695
f_klem wrote: post #534

Earlier quoted context omitted.

> On the contrary, I highly recommend people in Philosophy of mind and linguistics should start reading AI research papers because their theories and ideas are highly outdated, even ancient. Your books are from 1927 and 1972 respectively and Turing's article is from 1950s. And they are relatively new with respect to other works in Philosophy. People in philosophy and cognitive linguistics do read AI research. Don't g…

> Maybe you can clarify why they are outdated. The First Edition (1995) of the classic textbook Artificial Intelligence: A Modern Approach by Russell and Norvig talks about the criticisms of Dreyfus quite extensively. In the second edition (2003) they conclude: "In sum, many of the issues Dreyfus has focused on-background commonsense knowledge, the qualification problem, uncertainty, learning, compiled forms of decis…

I have the third edition, so I can only speak for it. Being the 3rd edition of the book, I assume that it is the 3rd time the text is revised, so I expect the other two editions (1st and 2nd) to adolesce from the same problem, which I state in the following paragraphs.

The mention to Dreyfus in the 3rd edition of Artificial Intelligence, a Modern Approach, by Stuart Russell and Peter Norvig, is made in 4 different places of the book, referencing four different problems.

The first mention is in page 279, effectively in the bibliographical notes, and it is about something called the 'frame problem'. Dreyfus presents this problem in the 1972 edition of the book, as a problem pertaining 'how to differentiate figure from ground', or 'how to account for what is important and what is not in a specific scenario'. But the solution to the problem that Norvig and Russel cite (Ray Reiter, 1991) is from a paper that _changes the conditions of the problem_, even _change the problem completely_ (by reductionism) to 'how to detect objects that do not change after an action'. They claim the problem solved, but they are actually not addressing Dreyfus criticisms, and misleading the reader to think that the problem is actually solved. The frame problem, by now, is still unsolved (and is one of the most difficult problems to solve).

The second mention is in page 1024, under a section called 'Weak AI: Can machines act intelligently?', and subsection 'The argument from informality'. The section mentions the books What Computers can't do (1972) and What Computers still can't do (1992), as well as Mind over Machine (1986). Unfortunately, this section completely misunderstands the critique of AI that Dreyfus exposes in those books. The whole section is misleading, obfuscating or tergiversing the critique from Dreyfus to fit the purpose of Norvig and Russell (mainly, to show that advances in machine learning and AI can make a solid base for machines that 'act intelligently').

The third mention is in page 1049, and it tries to undermine the first-step fallacy (which is similar to the fallacy of composition). Again, they do it by completely dismissing Dreyfus' critique, not addressing the issue. Then they go on talking about 'rationality' (as explained in chapter 1), but with a trick: only in terms of machines, goal-oriented expectations, computing resources. Dreyfus' critique is about the overall AI enterprise and the search for 'artificial' intelligence, Russell and Norvig discourse in this section first reduce Dreyfus' critique to what they can handle, to their own terms. That is, they evade the issue.

The fourth and final mention, in page 1072, is the bibliographical citation.

Re-reading the non-technical, but more theoretical parts of the book just made me realize how poorly constructed the book is. For example, the definitions given about AI in page 2 are just laughable. Compare with an introductory text on Psychology [0].

[0] https://pressbooks.openeducationalberta.ca/saitintropsycholo...

Re: They’re made out of weights

#719

Post by noduerme »

DangitBobby wrote: post #583
noduerme wrote: post #175

Earlier quoted context omitted.

I didn't read it as coming to the same conclusion as the original, because the meat story presupposes that we who are meat already know that the aliens are wrong. (Maybe that's a humanist reading of the original, but okay). I didn't read this one as trying to make a case that we are fools for assuming that matrix multiplication can't be intelligent... I think its point was that it can't be intelligent, and that peopl…

Don't take this as a criticism, but I think overwhelmingly people took it the other way. The fact that the author admits at the end that the story was written with the assistance of "weights" is a tell, to me. I just have to assume the author's genuine position (which I believe to be, we don't know that LLMs aren't conscious or that they could never be conscious) is so absurd to you that the thing comes across as sat…

I appreciate you taking a moment to write this. I was a little confused by the downvote. I think I have a tendency to credit satire at times when it's not intended... my own sense of humor has a lot to do with tweaking people's expectations, and coming from a family of tricksters, no one wants to be the one who doesn't get the joke. So maybe it's a me problem. Having said that, the situation with the aliens is that they can't conceive of intelligent meat, because they can't conceive of how that could work. We do understand how matrix multiplication works and how it gives rise to apparently emergent behavior, because we theorized it and we engineered it. So I can't help taking the idea that we'd be baffled at "that's it, just numbers?" as anything but tongue in cheek.

I'd only add that if it's not intentional satire, it's an even more profound example of the unintentional variety.

Re: They’re made out of weights

#720

Post by noduerme »

cortesoft wrote: post #457
noduerme wrote: post #162

Earlier quoted context omitted.

In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativi…

> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about…

Maybe we do. I think it's a human tendency at large to ascribe pattern or intelligence or spirit where there is only noise. If we can't even prove our own intelligence, doesn't that reinforce the idea that we're in no position to claim intelligence has emerged by running our own intellectual output through a fixed set of weights, the training of which we also designed? At best, any such intelligence would be entirely self-refential and exposed to the question of whether we ourselves are intelligent. If your position is that we are not, then there's no way an LLM could be.
Post reply on HN