Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

111–120 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#111

Post by bemmu »

michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#112

Post by Retric »

tbrownaw wrote: post #75
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

> don't actually understand anything (as greater concepts), but rather operate as complex probability machines? But, things are defined by how they interact with the world around them. A concept is its relations to other concepts. Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.

It’s using tokens not concepts. ‘Apple’ the token could apply to a company or fruit, but tokens aren’t concepts. It’s a fairly fundamental limitation on the approach that’s most noticeable while feeding it data it can’t basically copy from it’s vast training data.

Thus the sharp drop off where it can for example guess the correct answer to some math problems yet get others that are conceptually identical completely wrong.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#114

Post by thereare5lights »

bemmu wrote: post #111
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

There's no evidence for nondeterminism in the human brain

Re: Many in the AI field think the bigger-is-better approach is running out of road

#115

Post by [deleted] »

sgt101 wrote: post #41
nextworddev wrote: post #14

Earlier quoted context omitted.

Yep. We are in the very early innings of capital being deployed to all this.

Ok - what's the ROI on the $10bn (++) that OpenAI have had? So far I reckon This isn't what VC's (or microsoft) dream of.

[deleted]

Re: Many in the AI field think the bigger-is-better approach is running out of road

#116

Post by chongli »

tbrownaw wrote: post #75
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

> don't actually understand anything (as greater concepts), but rather operate as complex probability machines? But, things are defined by how they interact with the world around them. A concept is its relations to other concepts. Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.

A concept is its relations to other concepts.

No, that's an a priori concept. A posteriori concepts comprise empirical knowledge which necessitates experience of the world [1].

Example: you can know a priori that "all bachelors are unmarried." If I tell you "Tom is a bachelor" then you know that Tom is unmarried (assuming I tell the truth about Tom being a bachelor).

But if I say "all bachelors are unhappy" then you haven't learned anything because knowledge about the happiness/unhappiness of bachelors is an empirical question. To know whether or not I was telling the truth, you would need to conduct research about the real world, for example by conducting a survey of bachelors.

[1] https://en.wikipedia.org/wiki/A_priori_and_a_posteriori

Re: Many in the AI field think the bigger-is-better approach is running out of road

#117

Post by anikan_vader »

thereare5lights wrote: post #114
bemmu wrote: post #111

Earlier quoted context omitted.

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

There's no evidence for nondeterminism in the human brain

There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#118

Post by SgtBastard »

famouswaffles wrote: post #33
woeirua wrote: post #10

Earlier quoted context omitted.

It seems consistent with the behavior we see when using GPT4 in chat mode. Every once in a while it will change its answer as it’s generating it, as though it’s switched which model it favors to produce the response. GPT3.5 doesn’t do that.

MoE models don't work like that though

Token-at-time MoE models could result in mid-output-stream directional change.

I agree with you that whole-output MoE models don’t.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#119

Post by 01100011 »

michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I still think LLMs are a game changer. They are seemingly amazing at deducing meaning and intent from natural language. That's enough to be a game changer. Is it a game changer like the internet is/will-be? No. Is it on the level of the television or the telephone? Probably not. But you can still leverage it in many ways to add value or reduce human work.

If we're talking about LLMs as general purpose AI or a tool to replace programmers... mostly a fail so far. LLMs seem like they could eventually be a component of something bigger, but I don't see a line between what they are and general purpose AI.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#120

Post by goldenshale »

As if anyone is good at predicting the future. Please can we stop acting like expertise equates to fortune telling capabilities?! Nobody has any clue what a 1000x sized GPT model could do, and anybody who makes strong claims is a charlatan. In this age of paranoid AI risk cultists we need to cultivate humility and calm, a willingness to follow data rather than beliefs and predictions.
Post reply on HN