Weak-to-Strong Generalization • Hacker News Board

Re: Weak-to-Strong Generalization

#61

Post by wavemode » Thu, Dec 14, 2023, 7:26 PM UTC

gbasin wrote: post #50

wavemode wrote: post #38

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…

Your conclusion may be true but your examples aren't. You can definitely predict the stock market based on past prices, and I suspect you can with weather as well.

> You can definitely predict the stock market based on past prices

This is only true if you consider occasionally doing slightly better than random chance, "predicting the stock market". Unfortunately, while this would be enough to make a trader a net positive return over time, we have more stringent requirements for a system to become AGI.

> I suspect you can with weather as well

You suspect wrong.

Re: Weak-to-Strong Generalization

#62

Post by bayindirh » Thu, Dec 14, 2023, 7:29 PM UTC

ctoth wrote: post #60

bayindirh wrote: post #39

Earlier quoted context omitted.

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound . Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only tr…

> There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. The Lorenz attractor, Conway's Game of Life, fractals, and of course... The humble Turing machine itself all argue against this idea. Edit: Now it[0] is stuck in my head. [0]: https://www.youtube.com/watch?v=QrztrxV9OtQ

They're crowd engines. It's akin to how human clans can achieve more than a single human, but can scale up to a certain point.

The funny thing is I had this discussion during my theory of computation course with my professor, and I'm trying to disprove it daily for decades. I was unable to find a single, real world example.

Fractals are also found in the nature, however since we need to zoom into them, they end at a certain point.

Also, nature is a fractal in a greater sense.

Stars follow an orbit in a galaxy. Planets follow an orbit around a star. Satellites follow an orbit around a planet. While an edge case, flying bugs follow an orbit around a light source. At the end electrons follow an orbit around a nucleus.

IOW, a fractal is not more complex than nature itself.

Re: Weak-to-Strong Generalization

#63

Post by refulgentis » Thu, Dec 14, 2023, 7:29 PM UTC

JZL003 wrote: post #33

This reminds me of a thing cory doctorow talks about how tech companies control the narrative to focus on fun sexy problems while they have fundamental problems which expose the lie. For example uber/self driving cars always talking about the trolley problem, as if the current (or near future) problem is that self-driving cars are so good they have to choose which one. Not the current very difficult problem of gettin…

After some meditation, I don't find this line of inquiry to bear fruit:

I don't recall any entity, nor the entities named (Uber / self-driving cars) talking about the trolley problem - that's a well-known thought experiment in philosophy, but not something covered as a stark binary choice in self-driving cars planner systems.

I also don't recall traffic cones being a very difficult problem beyond Cruise + cones on windshield in SF. I have no love for Cruise. But its straightforward to pause if there's a large object on the windshield.

I don't think Corey's observation w/r/t A) loss-making companies over years B) focusing investors towards speculative advancements that would make their current business model profitable without changing applies here, OpenAI is _very_ successful.

After all that, I'm left at "Corey would take a bit of offense towards their thoughts on corporate responsibility via 'Uber is a predatory massively unprofitable company lying about odds they'll invent self-driving via talking about trolley problem' misshapen to critique a very profitable company funding fundamental research in the interest of safety that would be needed if their current rate of improvement continues.

Re: Weak-to-Strong Generalization

#64

Post by myk9001 » Thu, Dec 14, 2023, 7:32 PM UTC

bayindirh wrote: post #49

johncolanduoni wrote: post #46

Earlier quoted context omitted.

In this theory of computational bounds in nature, how did humans arise?

Nature is a more complex and sophisticated machinery when compared to humans. If this bound didn't exist, universe can spontaneously create new universes. However, it can only create elements, stars, planets, galaxies, which are less sophisticated than the universe itself. So, even universe has an upper limit on its creative abilities.

OK, in this theory of computational bounds in nature, how did the universe arise?

Re: Weak-to-Strong Generalization

#65

Post by bzbz » Thu, Dec 14, 2023, 7:36 PM UTC

famouswaffles wrote: post #52

wavemode wrote: post #38

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…

>You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.) >You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company funda…

In your example, the amino acids order is sufficient to directly model the result: the sequence of amino acids can directly generate the protein, which is either valid or invalid. All variables are provided within the data.

In the original example, we are testing weather using the previous day’s weather. We may be able to model using whatever correlation exists between the data. This is not the same as accurately predicting results, if the real-world weather function is determined by the weather of surrounding locations, time of year, and moon phase. If our model does not have this data, and it is essential to model the result, how can you accurately model?

In other words: “Garbage in, garbage out”. Good luck modeling an n-th degree polynomial function, given a fraction of the variables to train on.

Re: Weak-to-Strong Generalization

#66

Post by logicchains » Thu, Dec 14, 2023, 7:38 PM UTC

cwillu wrote: post #44

logicchains wrote: post #36

Earlier quoted context omitted.

We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such…

We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.

>We already have systems that can evaluate first order logical statements

My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.

Re: Weak-to-Strong Generalization

#67

Post by johncolanduoni » Thu, Dec 14, 2023, 7:40 PM UTC

bayindirh wrote: post #49

johncolanduoni wrote: post #46

Earlier quoted context omitted.

In this theory of computational bounds in nature, how did humans arise?

Nature is a more complex and sophisticated machinery when compared to humans. If this bound didn't exist, universe can spontaneously create new universes. However, it can only create elements, stars, planets, galaxies, which are less sophisticated than the universe itself. So, even universe has an upper limit on its creative abilities.

By what mechanism would a universe spontaneously create a new universe? As a human, can I spontaneously create anything simpler than me?

Also, under what theory of cosmology are you operating, and how do you determine when one thing is simpler than another? Under the Big Bang theory, the very early state of the universe (e.g. prior to initial nucleosynthesis) seems simpler to me than a galaxy.

Re: Weak-to-Strong Generalization

#68

Post by bayindirh » Thu, Dec 14, 2023, 7:40 PM UTC

myk9001 wrote: post #64

bayindirh wrote: post #49

Earlier quoted context omitted.

Nature is a more complex and sophisticated machinery when compared to humans. If this bound didn't exist, universe can spontaneously create new universes. However, it can only create elements, stars, planets, galaxies, which are less sophisticated than the universe itself. So, even universe has an upper limit on its creative abilities.

OK, in this theory of computational bounds in nature, how did the universe arise?

In all seriousness, this a question of great interest for me, too, and I'm playing with it for a quite some time.

Trying to answer it or at least starting to search for the answer steered me to astronomy, thinking going deeper on that front may bring me closer to the answer, but it was a bit too much for my younger self, so I continued to dig that issue on a more casual level.

This doesn't mean that I don't spend considerable amount of time thinking about it today, and will put that issue to rest any time soon. At the core, this kind of questioning brought me to here in life, and I'm not gonna let this side of mine to rest or whither and die.

Re: Weak-to-Strong Generalization

#69

Post by wavemode » Thu, Dec 14, 2023, 7:40 PM UTC

famouswaffles wrote: post #52

wavemode wrote: post #38

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air curr…

>You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.) >You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company funda…

> You don't need to train on the inputs(casual processes) of anything, that's what training is there to figure out.

I mean... this is just obviously false. If the data you're training on isn't causally predictive, you may occasionally find good-enough patterns for a particular use case (i.e. you may occasionally guess better than a coin flip which direction the stock market goes) but you aren't going to accurately model anything, and certainly not well enough to create an AGI that makes intelligent decisions.

Words in sentences (and, indeed, proteins in a sequence) are causally predictive of each other - the grammar and semantics of one word tends to dictate what words are likely to surround it. So LLM's are very good at writing, and that is certainly useful! But that's just not the same as human intelligence.

When someone makes an AGI out of an LLM then I'll be proven wrong, I suppose. I'm just sharing my personal view on things.

Re: Weak-to-Strong Generalization

#70

Post by davecap1 » Thu, Dec 14, 2023, 7:42 PM UTC

spookie wrote: post #58

famouswaffles wrote: post #52

Earlier quoted context omitted.

>You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.) >You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company funda…

I'm sorry in advance, but aren't proteins glorified Lego?

There's a lot more to protein sequences than legos. I think the argument is that you don't need to train a model on fundamental organic chemistry/biochemistry, electrostatic protein interaction, hydrogen bonding, hydrophobic interaction, quantum mechanics, etc... in order for it to accurately predict protein sequences.