Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weak-to-Strong Generalization

openai.com

Post reply on HN
Print view View original

121–130 of 203 posts

Re: Weak-to-Strong Generalization

#121

Post by ShamelessC »

novaRom wrote: post #100
JZL003 wrote: post #33

This reminds me of a thing cory doctorow talks about how tech companies control the narrative to focus on fun sexy problems while they have fundamental problems which expose the lie. For example uber/self driving cars always talking about the trolley problem, as if the current (or near future) problem is that self-driving cars are so good they have to choose which one. Not the current very difficult problem of gettin…

OpenAI do probably realize they will not win long term vs Open Source (see AI Alliance). Their way of centralized cloud models is simply too risky and not sustainable. What we see instead is more liberation, open source, cooperation, down-scaling, local models. Just look how many more tools and models is available today than even a year ago. And where is OpenAI? Still the same chatGPT, still the same DALL-E, nothing…

That’s a bit reductive. Lots of new closed models have come out since then too. And, chatGPT and DALLE (while closed) have both received consistent upgrades and remain competitive with state of the art.

I’m hopeful that you’re correct but it’s perhaps not guaranteed that the people with all the money wind up failing in this regard. And I say this as someone who makes open contributions in that space.

Re: Weak-to-Strong Generalization

#122

Post by alach11 »

logicchains wrote: post #10

>We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years. However, we still do not know how to reliably steer and control superhuman AI systems Their entire premise is contradictory. An AI incapable of critical thinking cannot be smarter than a human, by definition, as critical thinking is a key component of intelligence. And an AI that is at least as capable of critica…

In a July post [0] they said "while superintelligence seems far off now, we believe it could arrive this decade". Now they're saying "within the next 10 years". I wonder if that reflects a shift in thinking on timelines?

[0] https://openai.com/blog/introducing-superalignment

Re: Weak-to-Strong Generalization

#123

Post by Jensson »

panarky wrote: post #86
wavemode wrote: post #38

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…

Human intelligence itself is shaped by our interaction with outputs. Our learning and understanding of the world are profoundly influenced by the language, behaviors, and cultural artifacts we observe. Think about the process of a child learning a language. The child does not have direct access to the "inputs" of linguistic rules or grammar; they learn primarily through observing and imitating the language output of…

> Over time, they develop a sophisticated understanding of language, not by direct instruction of underlying rules, but through pattern recognition and contextual inference from these outputs.

Kids learn through supervised learning. Children don't develop strong language skills without parents or other people to correct them when they use language incorrectly.

We don't use supervised learning on LLMs. There is no way we can train an LLM by using human supervisors to rate every output. It works for humans since humans learn with so few examples, supervising a kids language learning doesn't take much effort, a single person can easily manage it while doing it for an LLM would consume the whole worlds workforce for many years.

Re: Weak-to-Strong Generalization

#124

Post by wavemode »

ladberg wrote: post #118
wavemode wrote: post #103

Earlier quoted context omitted.

You've linked me a neural network that was trained on decades of climate data like air pressure, wind direction, soil temperature, cloud cover, and hundreds of other features. So... I was correct? https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+docume...

Maybe we have different definitions of inputs and outputs, but all of those things seem like outputs to me? Maybe in this scenario the inputs and outputs are really the same set of variables so there isn't a huge distinction. The reason I linked that article is that all the inputs are outputs of the network, so by definition it's training on just outputs (under my personal definition of outputs).

I specified what I meant by inputs and outputs in my original comment. Whether it rained today, whether it was cloudy yesterday, how hot is it going to get, that sort of thing, are the observable weather, the things most people care about when checking a weather forecast. But you can't only train on those things and expect to come up with accurate predictions. You also need to train on the things that cause that observable weather (air currents, warm fronts, etc.)

Re: Weak-to-Strong Generalization

#125

Post by Jensson »

orbital-decay wrote: post #96
wavemode wrote: post #38

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…

> I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. This is irrelevant because OpenAI's definition of AGI [1] doesn't imply similarity or equivalence to humans at all: >artificial general intelligence (AGI)—by which we mean highly autonomous systems that o…

>artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work

Then we already have AGI, automated farming equipment outperforms humans in 90% of jobs*.

*Jobs in 1700. As things got automated the jobs changed and we now do different things.

Re: Weak-to-Strong Generalization

#126

Post by unlikelymordant »

logicchains wrote: post #53
ctoth wrote: post #42

Earlier quoted context omitted.

> as critical thinking is a key component of intelligence. When I evaluate this statement, my brain raises a type error. Intelligence is a lot of things -- compression among them, and yes possibly an RL-based AI would use an actor-critic approach for evaluating its actions, but I doubt that at all maps onto the human activity we call "critical thinking." To me, critical thinking involves stuff like questioning assump…

>I really don't see that critical thinking is at all required for a raw optimization process. The problem they are trying to solve is what happens when that optimization process isn't aligned with human flourishing. I agree it's possible to have a dangerous AI that lacks human "critical thinking", but I don't think it's reasonable to refer to an AI as much more intelligent than humans if there's any class of intellec…

If a 'superintelligence' achieves the same outcomes as humans without engaging in the same class of intellectual tasks that humans do, wouldnt it still be a superitelligence? Deep blue was beating everybody at chess without engaging in the same process as humans. If chess is a metaphor for life, it seems some algorithm might do better at all the things a human does while not arriving at its decisions in a remotely similar way.

Re: Weak-to-Strong Generalization

#127

Post by johnfn »

novaRom wrote: post #100
JZL003 wrote: post #33

This reminds me of a thing cory doctorow talks about how tech companies control the narrative to focus on fun sexy problems while they have fundamental problems which expose the lie. For example uber/self driving cars always talking about the trolley problem, as if the current (or near future) problem is that self-driving cars are so good they have to choose which one. Not the current very difficult problem of gettin…

OpenAI do probably realize they will not win long term vs Open Source (see AI Alliance). Their way of centralized cloud models is simply too risky and not sustainable. What we see instead is more liberation, open source, cooperation, down-scaling, local models. Just look how many more tools and models is available today than even a year ago. And where is OpenAI? Still the same chatGPT, still the same DALL-E, nothing…

Both ChatGPT and DALL-E have received major updates over the last year (3 to 4 turbo and 2 to 3, respectively).

Re: Weak-to-Strong Generalization

#128

Post by wavemode »

famouswaffles wrote: post #105
wavemode wrote: post #102

Earlier quoted context omitted.

> or is this just a vague "i'll know it when i see it" assertion No, it's actually exceptionally easy to quantify. Take a look at the current leaderboard for LLM reasoning capabilities: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

The state of the art model isn't even on that list. Okay, you've at least given me numbers. You're still not answering my question. Which number signals agi ? Let's look at the top model in that list (which again isn't close to the best performing LLM) that you say isn't agi. so are you telling me that every human can do those tests and perform better than every model on that list. Is that what you are saying ? becau…

> so are you telling me that every human can do those tests and perform better than every model on that list

This is a strange bar to require clearing. Every human is not even capable of reading, or speaking, or thinking at all. But those humans who can't are not the benchmark of general human intelligence.

Let's make this simple - the average human IQ is around 100. Will an LLM ever have an IQ of 100? No, I don't believe they'll ever even be close.

How can we gauge this abstract reasoning ability? Lots of ways. Try to teach them mathematics. Try to teach them physics. Try to explain to an LLM the rules to a card game and have it actually play the game with you accurately.

Re: Weak-to-Strong Generalization

#129

Post by haltist »

davecap1 wrote: post #70
spookie wrote: post #58

Earlier quoted context omitted.

I'm sorry in advance, but aren't proteins glorified Lego?

There's a lot more to protein sequences than legos. I think the argument is that you don't need to train a model on fundamental organic chemistry/biochemistry, electrostatic protein interaction, hydrogen bonding, hydrophobic interaction, quantum mechanics, etc... in order for it to accurately predict protein sequences.

The data that AlphaFold was trained on included all that information and more. The database they used for training included software simulations (and real world data) that accounted for atomic (quantum) interactions. The 3D structure of proteins includes all the quantum interactions.

More generally, AI models (aka very large function graphs) are trained on tuples that represent mappings of inputs to outputs (input -> output). The idea then is that whatever structure exists in those pairs/tuples/mappings is discovered by the training process with the help of gradient descent which tunes the parameters of the model/graph to optimally compress the information contained in the data. This means the model must uncover the quantum effects (or some close proxy of it) and then encode them into the parameters in a way that makes compression/prediction possible [1].

None of this is magic, compressing data requires uncovering structures and symmetries that can be used to reduce the size of the data and it turns out gradient descent with lots of parameters manages to do that for a large class of problems albeit at a very steep computational cost that requires billions of dollars for hardware and software (including nuclear power plants [2]). We are not going to get AGI with this approach but fortunately I know how to make it happen for a mere $80B.

1: https://arxiv.org/abs/2305.15614

2: https://www.cnbc.com/2023/09/25/microsoft-is-hiring-a-nuclea...

Re: Weak-to-Strong Generalization

#130

Post by guybedo »

What does it even mean to align an intelligence? does it mean we want it to behave in a way that doesn't break moral/ethical rules, that aligns with our society rules ? Meaning do no crime, do no harm, etc...

Well, maybe we should acknowledge that we've never even been able to do that with humans. There's crime, there's war, etc...

We can see crime in our societies as a human alignment problem. If humans were "properly aligned", there wouldn't be any crime or misbehavior.

So yeah i'm rather skeptical about aligning a superhuman intelligence that would dwarf us by its capabilities.

Post reply on HN