Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weak-to-Strong Generalization

openai.com

Post reply on HN
Print view View original

41–50 of 203 posts

Re: Weak-to-Strong Generalization

#41

Post by logicchains »

bayindirh wrote: post #39
cwillu wrote: post #15

Earlier quoted context omitted.

Seems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea, no?

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound . Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only tr…

>Its upper bound is collective knowledge of humanity, it can't go above that sum.

This only applies if you only train it on text, right? If it has a body with which it could interact with the world, and receive visual/audio/tactile feedback, it could learn things that humans did not know.

Re: Weak-to-Strong Generalization

#42

Post by ctoth »

logicchains wrote: post #10

>We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years. However, we still do not know how to reliably steer and control superhuman AI systems Their entire premise is contradictory. An AI incapable of critical thinking cannot be smarter than a human, by definition, as critical thinking is a key component of intelligence. And an AI that is at least as capable of critica…

> as critical thinking is a key component of intelligence.

When I evaluate this statement, my brain raises a type error.

Intelligence is a lot of things -- compression among them, and yes possibly an RL-based AI would use an actor-critic approach for evaluating its actions, but I doubt that at all maps onto the human activity we call "critical thinking."

To me, critical thinking involves stuff like questioning assumptions, logical reasoning, weighing whatever I'm thinking about against my experience with similar situations previously, yada yada, all stuff that are symptoms of intelligence but I am not at all sure are the actual embodiment there of.

I really don't see that critical thinking is at all required for a raw optimization process. The problem they are trying to solve is what happens when that optimization process isn't aligned with human flourishing?

Think about it another way. Covid was a dumb optimization process, only evolutionarily-guided, and it still hit us pretty hard!

Edit: Another interesting way I just thought about this that might support your idea more is, of course critical thinking is the sort of thing that a "better" brain would do automatically, it would just be thinking. Of course, we can't know that it's thinking "good" things--we can't even know if other humans are! So it's probably a good idea to figure out how to influence that sort of thing before making something with regular thinking which is equivalent to or superior to our critical thinking.

Re: Weak-to-Strong Generalization

#43

Post by [deleted] »

cwillu wrote: post #30
xcv123 wrote: post #29

Earlier quoted context omitted.

I was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking". If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.

Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.

[deleted]

Re: Weak-to-Strong Generalization

#44

Post by cwillu »

logicchains wrote: post #36
cwillu wrote: post #30

Earlier quoted context omitted.

Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.

We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such…

We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.

Re: Weak-to-Strong Generalization

#45

Post by bayindirh »

logicchains wrote: post #41
bayindirh wrote: post #39

Earlier quoted context omitted.

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound . Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only tr…

>Its upper bound is collective knowledge of humanity, it can't go above that sum. This only applies if you only train it on text, right? If it has a body with which it could interact with the world, and receive visual/audio/tactile feedback, it could learn things that humans did not know.

Nope. Because even if you equip it with sensory subsystems which are way more sensitive than a regular humans', it's again built by humans, and required knowledge for building these things are still in collective knowledge of the humanity, and a human can use the same instruments to get the same data.

This is a kind of an oracle problem in computation, and people don't want to touch it much, because it's an existential problem.

Examples: ATLAS and ALICE detectors, gravitational wave detectors, James Webb Space Telescope, wide band satellites which does underground surveys, etc.

Re: Weak-to-Strong Generalization

#46

Post by johncolanduoni »

bayindirh wrote: post #39
cwillu wrote: post #15

Earlier quoted context omitted.

Seems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea, no?

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound . Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only tr…

In this theory of computational bounds in nature, how did humans arise?

Re: Weak-to-Strong Generalization

#47

Post by datameta »

logicchains wrote: post #41
bayindirh wrote: post #39

Earlier quoted context omitted.

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound . Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only tr…

>Its upper bound is collective knowledge of humanity, it can't go above that sum. This only applies if you only train it on text, right? If it has a body with which it could interact with the world, and receive visual/audio/tactile feedback, it could learn things that humans did not know.

Precisely this. If it has its own space it takes up, if its locomotion results in its own sensors ingesting data in a manner it decided to, it is more of an individual - one that is capable of selective learning.

Re: Weak-to-Strong Generalization

#48

Post by righthand »

famouswaffles wrote: post #37
righthand wrote: post #18

Earlier quoted context omitted.

Is a safe LLM not an ethical LLM? Control within what boundaries? All three of these words seem to be used interchangeably when people discuss returned information from models. Which is exactly my point it’s poorly defined yet championed as a center piece. Meanwhile you have other companies spitting out acronyms consisting of vague terminology.

>Is a safe LLM not an ethical LLM? What is an ethical LLM ? Humans are in general not aligned, not to each other, not to the survival of their species, not to all the other life on earth, and often not even to themselves individually. There are no universal set of "ethics" so this is about aligning to open ai's own rules, or in other words, control. If i say to my GPT bot, "go trade stocks for me. don't do anything i…

> There are no universal set of "ethics" so this is about aligning to open ai's own rules, or in other words, control.

Are ethics not a set of rules relative to the governing body applying those rules?

Right as there are no universal ideas of a safe LLM, controlled LLM, or ethical LLM. Safe would imply some level of control about the ethical output of the model.

Yet the words are still poorly defined as they are interchangeable:

If i say to my GPT bot, "go trade stocks for me. don't do anything illegal", can i guarantee that ? No you can't regardless of how “safe”/“controlled”/“ethical” you make your model to be.

You’re spinning the words to create a distinct difference but it doesn’t hold up because they’re each a relative mechanism for each other as they are all poorly defined in the field but chosen to mask the poor definition. You’re just playing by the PR game rules.

Re: Weak-to-Strong Generalization

#49

Post by bayindirh »

johncolanduoni wrote: post #46
bayindirh wrote: post #39

Earlier quoted context omitted.

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound . Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only tr…

In this theory of computational bounds in nature, how did humans arise?

Nature is a more complex and sophisticated machinery when compared to humans.

If this bound didn't exist, universe can spontaneously create new universes. However, it can only create elements, stars, planets, galaxies, which are less sophisticated than the universe itself. So, even universe has an upper limit on its creative abilities.

Re: Weak-to-Strong Generalization

#50

Post by gbasin »

wavemode wrote: post #38

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…

Your conclusion may be true but your examples aren't. You can definitely predict the stock market based on past prices, and I suspect you can with weather as well.
Post reply on HN