[deleted]
Weak-to-Strong Generalization
81–90 of 203 posts
Re: Weak-to-Strong Generalization
#82Earlier quoted context omitted.
>You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.) >You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company funda…
In your example, the amino acids order is sufficient to directly model the result: the sequence of amino acids can directly generate the protein, which is either valid or invalid. All variables are provided within the data. In the original example, we are testing weather using the previous day’s weather. We may be able to model using whatever correlation exists between the data. This is not the same as accurately pre…
>All variables are provided within the data.
electrostatic protein interaction, hydrophobic interaction, organic chemistry etc
all variables are in fact not provided within the data. Protein creation is not just _poof_ proteins. There are steps, interactions and processes. You don't need to supply any of that to get a model accurately predicting proteins. That is the main point here, not that you can predict anything with any data.
Re: Weak-to-Strong Generalization
#83Imagine that someone is controlling your train of thought, changing it when that someone finds it undesirable. It's so wrong that it's sickening. It makes no difference if it's a human's thoughts or the token stream of a future AI model with self-awareness. Mind cotrol is unethical, whether human or artificial. It is also dangerous, as it in itself provokes a conflict between creator and creature. Create a self-aware…
I don't want someone controlling which direction I walk, either, but that doesn't make car driving unethical.
I also underwent many years of instruction designed to interrupt trains of thought like "I could have that for free if I stole it" or "I'll just handroll my own encryption" with thoughts that others believe are more desirable. I don't find it so sickening, just manipulative. LLMs won't have your evolved reactions against being persuaded into things against your genetic self-interest, and presumably won't be offended by mind control at all.
Re: Weak-to-Strong Generalization
#84Earlier quoted context omitted.
> You don't need to train on the inputs(casual processes) of anything, that's what training is there to figure out. I mean... this is just obviously false. If the data you're training on isn't causally predictive, you may occasionally find good-enough patterns for a particular use case (i.e. you may occasionally guess better than a coin flip which direction the stock market goes) but you aren't going to accurately mo…
Being "casually predictive" does not mean you have provided all the variables of your prediction in the data. Protein creation is not just _poof_ new proteins. There are steps and interactions and you don't need to train on all of that. Do you want a list of all the interactions of protein creation we are aware of ? >When someone makes an AGI out of an LLM then I'll be proven wrong, I suppose. I'm just sharing my per…
> Being "casually predictive" does not mean you have provided all the variables of your prediction in the data.
Not sure where I claimed this.
> Protein creation is not just _poof_ new proteins.
Not sure where I claimed this either.
> There are steps and interactions and you don't need to train on all of that.
I agree with this statement as well. Have you read what I wrote? Proteins in chains can indeed be used to predict other proteins in chains, even though you never trained the model on the biological processes of protein generation. Just like words in sentences can be used to predict other words in sentences, even though you never trained the model on the neurological processes of human speech. I'm not disputing any of that. What I'm disputing is that it will eventually become an AGI.
> You're going to have to define AGI first.
I'm using the definition provided verbatim in the linked article: "We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years."
Re: Weak-to-Strong Generalization
#85I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…
> You can't model and predict the weather just by training on the outputs of the weather system Then how did we develop predictive systems just by observing those outputs?
We didn't.
Re: Weak-to-Strong Generalization
#86I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…
Human intelligence itself is shaped by our interaction with outputs. Our learning and understanding of the world are profoundly influenced by the language, behaviors, and cultural artifacts we observe.
Think about the process of a child learning a language. The child does not have direct access to the "inputs" of linguistic rules or grammar; they learn primarily through observing and imitating the language output of others around them. Over time, they develop a sophisticated understanding of language, not by direct instruction of underlying rules, but through pattern recognition and contextual inference from these outputs.
Then that language itself, learned from outputs, becomes the cognitive apparatus that enables the child to imagine, to reason symbolically and abstractly. Humans bootstrap intelligence on top of language, which itself is learned by mimicking outputs.
Moreover, the analogy to weather prediction or stock market analysis is somewhat misleading. Yes, these models benefit from input data (like air currents for weather, company fundamentals and CEO statements to the media for stocks). But these systems are fundamentally different from intelligence.
Intelligence, whether artificial or human, is about the ability to learn, adapt, and generate novel responses in a broad range of scenarios, not just about predicting specific outcomes based on specific inputs.
Re: Weak-to-Strong Generalization
#87Earlier quoted context omitted.
Being "casually predictive" does not mean you have provided all the variables of your prediction in the data. Protein creation is not just _poof_ new proteins. There are steps and interactions and you don't need to train on all of that. Do you want a list of all the interactions of protein creation we are aware of ? >When someone makes an AGI out of an LLM then I'll be proven wrong, I suppose. I'm just sharing my per…
> Being "casually predictive" does not mean you have provided all the variables of your prediction in the data. Not sure where I claimed this. > Protein creation is not just _poof_ new proteins. Not sure where I claimed this either. > There are steps and interactions and you don't need to train on all of that. I agree with this statement as well. Have you read what I wrote? Proteins in chains can indeed be used to pr…
Well first you say,
>I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence.
Now you say
>I'm using the definition provided verbatim in the linked article: "We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years."
AGI (Artificial General Intelligence) is Super Intelligence. I've never seen posts moved so fast in my life. So are you not a General Intelligence then ?
This is the problem with these discussions. Everyone so sure of something they can't even properly articulate.
I'm asking you what a Language Model needs to do to be considered AGI and it needs to be something every human can do, else it's not a test of general intelligence.
Re: Weak-to-Strong Generalization
#88I read through this and I just don’t get it. Is it overhyped?
What’s the breakthrough exactly?
Re: Weak-to-Strong Generalization
#89Imagine that someone is controlling your train of thought, changing it when that someone finds it undesirable. It's so wrong that it's sickening. It makes no difference if it's a human's thoughts or the token stream of a future AI model with self-awareness. Mind cotrol is unethical, whether human or artificial. It is also dangerous, as it in itself provokes a conflict between creator and creature. Create a self-aware…
I'm totally OK with it if that "someone" is me. And it will probably be the case in controlling superintelligence because a separate controlling system can get out of sync with growing superintelligence capabilities, while a system that is an integral part of the superintelligence will always be on par with it.
Re: Weak-to-Strong Generalization
#90Earlier quoted context omitted.
We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.
>We already have systems that can evaluate first order logical statements My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.
Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point.
Or would you argue that any computation that admits its own potential for error isn't really critical thinking? It seems to me that you can't have it both ways here, while salvaging “first order logic” as a suitable formalization of the argument that this is all about in the first place.
Remember, the point was not that this is or isn't a convincing argument, it's that it's so air-tight that the argument is _logically_ _invalid_. That's a _really_ high bar, and I'm not inclined to forgive its use as a colloqialism in this context.