Unpredictable abilities emerging from large AI models • Hacker News Board

Re: Unpredictable abilities emerging from large AI models

#61

Post by danielbln » Fri, Mar 17, 2023, 12:51 PM UTC

macrolime wrote: post #51

maxlamb wrote: post #34

Earlier quoted context omitted.

I’m guessing one is data. The limit would be once you’ve trained a LLM on all public (or even private) data. Sure you can still make some improvements or try to find some additional private data but still, a fundamental limit has been reached.

Is it even feasible any time soon to train an LLM on all of YouTube?

Napkin math, assuming around 156 million hours of video on all of Youtube:

    156 million hours of YouTube videos
    9,000 words/hour
    6 characters/word (including space)
    First, let's find out the total number of characters:

    9,000 words/hour \* 6 characters/word = 54,000 characters/hour

    Now, let's calculate the total number of characters for 156 million hours of YouTube videos:

    54,000 characters/hour \* 156,000,000 hours = 8,424,000,000,000 characters

    Since 1 character is typically 1 byte, we can convert this to gigabytes:

    8,424,000,000,000 bytes / (1024 \* 1024 \* 1024) ≈ 7,842.11 GB

So, 8TB of text? Seems doable.

Re: Unpredictable abilities emerging from large AI models

#62

Post by PeterisP » Fri, Mar 17, 2023, 1:12 PM UTC

bryanrasmussen wrote: post #4

>“That language models can do these sort of things was never discussed in any literature that I’m aware of," I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system? Based on layman's readings in chaos and complexity theory.

> I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system?

Yes and no.

It's reasonable to expect that some unpredictable emergent behavior would exist in any sufficiently complex system, but it's not warranted to expect whether a particular capability will or will not emerge out of that system (that's the unexpected emergent behavior part); and it also was debated whether language models are a "sufficiently complex system", as many thinkers asserted that they should encounter fundamental limits in capabilities.

Re: Unpredictable abilities emerging from large AI models

#63

Post by oldgradstudent » Fri, Mar 17, 2023, 1:21 PM UTC

ishvanl wrote: post #33

There is an idea bouncing around my brain for a pipe-dream science fiction book I could write, about the rise of the first truly sentient (or sentient-appearing) AI & its effects on the world. It is beginning to get to the point where I am considering whether I need to actually start writing this book before this "sci-fi" concept becomes a mundane reality.

> It is beginning to get to the point where I am considering whether I need to actually start writing this book before this "sci-fi" concept becomes a mundane reality.

No need to rush. If that moment arrives, you could just change the Amazon category from fiction to nonfiction.

Re: Unpredictable abilities emerging from large AI models

#64

Post by glenstein » Fri, Mar 17, 2023, 2:06 PM UTC

bryanrasmussen wrote: post #4

>“That language models can do these sort of things was never discussed in any literature that I’m aware of," I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system? Based on layman's readings in chaos and complexity theory.

Same here. I don't think it's surprising, but depending on where you say it, you'll find people insisting that this can't be possible.

I think a lot of our widely held cultural beliefs on this front have been informed by academic philosophy from the 60s, 70s, and 80s. In particular, I would go do far as to say that Hubert Dreyfus, author of the book "What computers can't do" and frequent friendly adversary of Daniel Dennett, has a lot of responsibility here. He is famous for claiming computers would never become competitive at chess because chess required "insight", and always differentiated computer capabilities from complex behavior or human-like behavior, because the latter had something special that could only be represented with vague and underdefined terms. Even now I think Dreyfus is approvingly cited, while his core thesis is getting closer to being refuted every day if it hasn't been already.

Re: Unpredictable abilities emerging from large AI models

#65

Post by throwbadubadu » Fri, Mar 17, 2023, 2:45 PM UTC

tarruda wrote: post #44

curiousgal wrote: post #15

Earlier quoted context omitted.

I guess you perception of society is severely limited if you think a fancy autocomplete is capable of changing every aspect of it.

That was what I thought until a few months ago when ChatGPT was released. I never cared much about LLMs because it always felt like a brute force method to solving problems. What I'm seeing now is that some kind of intelligence seem to emerge from these models, even though under the hoods it is just a bunch o matrix multiplications. Who's can say for sure that our own brains doesn't work similarly? Maybe human intell…

Oh man... if it will be as stupid as us humans we are completely doomed. For now lets assume what it most likely is: not emerging anything, just a very good simulation of that emergeance.

I mean we already did stupid chat bots in the 90s with convoluted if-else-spaghettis that imitated Eliza. That could already hold for some sentences... GPT-4 now holds much longer, but still you can easily drive it to the point where it starts "looping" and not provide much more value, especially if you are more expert => makes me so far hope that this length will grow more, but still.. in my "sessions" did not (gladly) find anything emerging beyond the first wow and surprising answers... not even close to that.

Re: Unpredictable abilities emerging from large AI models

#66

Post by macrolime » Fri, Mar 17, 2023, 2:45 PM UTC

macrolime wrote: post #54

orwin wrote: post #50

Disclaimer : I did not test GPT 4 myself. I think those emerging abilities are really interesting from a philosophical point of view, especially on the matter of consciousness. It seems it really reinforce the illusionist point of view on the hard problem of consciousness, eg it doesn't really exist. It seems to reinforce most functionalist theories in fact. I would be really interested in what Chalmers think of thos…

The illusionist point of view is false by definition, although their definition of "illusion" or "exist" sounds like it might not be how I'd define those words.

Here's some things must be true if consciousness is an illusion:

Qualia does not exist

Subjective experience does not exist

Consciousness does not exist

There's nothing wrong with acting unethically or immoral because nobody actually experiences any harm because nobody experiences anything

Does anyone really think that's true?

Re: Unpredictable abilities emerging from large AI models

#67

Post by throwbadubadu » Fri, Mar 17, 2023, 2:50 PM UTC

mpalmer wrote: post #59

I'd like to see posts on LLMs written from a different perspective. For me, the surprise comes not from the sudden emergent capability of language models, but that the understanding (and synthesis!) of ideas encoded in language has succumbed to literally nothing more than statistical analysis. Or at least come that much closer to doing so. That it bears so close a resemblance to actual thinking says more about the im…

Interesting take.. especially as I often wondered, watching and loving animals, if not our ""intelligence separation"" from them is not to a large degree because of the language.. (and the wisdom storage and transfer that comes with it).

Re: Unpredictable abilities emerging from large AI models

#68

Post by nirav72 » Fri, Mar 17, 2023, 2:56 PM UTC

DeathArrow wrote: post #2

I used ChatGPT to generate erotic stories. Now I want a model which can produce porn videos from prompts.

I want ChatGPT style generative output to be used in games. For example - instead of standard flow based dialogs that are used for interacting with in-game NPCs - it would be neat to see them use a dynamically generated dialog based on player input. Albeit, maybe limited in scope. Imagine in a online RPG, a player interacts with a NPC. Then goes away. Another player comes along talks to the same NPC, the NPC can then recall and use the previous interaction with another the other player as context in the current interaction. Eventually, I think we would start seeing some weird emergent behavior in games.

Re: Unpredictable abilities emerging from large AI models

#69

Post by lamp987 » Fri, Mar 17, 2023, 3:22 PM UTC

danielbln wrote: post #21

legulere wrote: post #19

Earlier quoted context omitted.

The same was said 10 years ago. It's astonishing what can be done, but you can already see fundamental limits. I think it will raise productivity for some tasks, but not fundamentally change society.

Can you elaborate on fundamental limits?

there is no channel for uncertainty.

LLMs of this type will just start making up shit when they dont know something. because they simply generate the most probable next token based on previous x tokens. this is not fixable.

this alone makes these LLMs practically unusable in vast majority of real-world applications where you would otherwise imagine this tech to be used.

Re: Unpredictable abilities emerging from large AI models

#70

Post by EForEndeavour » Fri, Mar 17, 2023, 3:24 PM UTC

tarruda wrote: post #44

curiousgal wrote: post #15

Earlier quoted context omitted.

I guess you perception of society is severely limited if you think a fancy autocomplete is capable of changing every aspect of it.

That was what I thought until a few months ago when ChatGPT was released. I never cared much about LLMs because it always felt like a brute force method to solving problems. What I'm seeing now is that some kind of intelligence seem to emerge from these models, even though under the hoods it is just a bunch o matrix multiplications. Who's can say for sure that our own brains doesn't work similarly? Maybe human intell…

It doesn't matter to me whether the intelligence is "really emergent" or "just a simulation." Two things are true:

1. Solving all kinds of nontrivial problems posed in text format is extremely useful, no matter how it works under the hood. This means lots of people will use it, and it will change how people work

2. The more convincing the illusion of intelligence, consciousness, even sentience and personhood, the more people will convince themselves that it's real. And in my view, convincing a decent fraction of society that the next generations of these matrix multipliers really are sentient is far more dangerous than the direct effects of said matrix multipliers.