Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

131–140 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#131

Post by hooande »

ftxbro wrote: post #94
croes wrote: post #83

Earlier quoted context omitted.

Better result from less data? I doubt that.

I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs. I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some wa…

Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is Yes, the correct thing to do is get more data. Much more.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#132

Post by koboll »

semiquaver wrote: post #46
satellite2 wrote: post #23

How is the economist qualified to answer this question?

Not sure where I heard this but it’s apparently a common trope that many of the politicians and leaders that treat The Economist as close to holy writ are often horrified to learn that most the staff is actually a bunch of very precocious 20-somethings that are good at research and writing in an authoritative tone. Actually, now that I think of it, not so different from LLMs… (Full disclosure, I’ve been a subscriber…

Also true about every single consulting firm

Re: Many in the AI field think the bigger-is-better approach is running out of road

#133

Post by kedean »

Legend2440 wrote: post #122
anikan_vader wrote: post #117

Earlier quoted context omitted.

There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.

Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see. Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.

If it's not actually possible to simulate a system within the confines of physics, does it being deterministic actually matter outside of thought experiments?

I feel like a random system and a deterministic system which cannot be simulated are effectively the same thing.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#134

Post by aoeusnth1 »

Legend2440 wrote: post #122
anikan_vader wrote: post #117

Earlier quoted context omitted.

There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.

Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see. Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.

And how is a seeded RNG that an LLM uses any logically different from a deterministic brain? I’m not sure why any of this physics would be relevant to the functional behavior of the brain vs an LLM.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#135

Post by Animats »

rco8786 wrote: post #19
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data. Data is still king.

You need sort of a "primary education" data set which gets the model up to roughly a high school education level. Then augment it with special-purpose models for specific areas.

But until "I don't know" comes out, rather than hallucinations, we're in trouble.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#136

Post by koochi10 »

Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

I disagree with this slightly, modern computer chips follow a general purpose architecture not special purpose ones. The reason for this is building a computer chip is expensive and difficult to do. Similarly building any useful language model requires tons of compute power, and very smart ML researchers. Most of the smaller open source one's are just trained on GPT output.

By "cramming all of the web" on a model what is really going on is the hidden layers of that network are getting better at understanding language and logic. Imagine trying to teach a kid who doesn't know how to read to learn about a Science by only giving them science textbooks. Chances are they won't get very far.

Building little specialist model's don't really work either. It's like trying to train a parrot to do science, sure it can repeat some of the phrases that you give it, but at the end of the day it's not really making any new connections for you.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#137

Post by csomar »

michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I'm guessing that's how people figured out it was getting "nerfed". For Copilot, I kind of developed a relationship with the "AI". I kind of expect what it is going to generate, and I use it as a shortcut to (find function names, write variable names, complete unit tests, etc...). On its own, it's really bad to come up with the whole picture but as something that auto-completes you on steroid, it's really perfect.

When it got nerfed, its perspective got completely out of whack and it's making very different and inconsistent generation.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#138

Post by seydor »

TeMPOraL wrote: post #42
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and…

They can't reason yet, but they can extrapolate, but it cant check if its extrapolation is reasonable. Reasoning is not baked into the architecture of a GPT model. It seems it would be an entirely different type of model

Re: Many in the AI field think the bigger-is-better approach is running out of road

#139

Post by Solvency »

koochi10 wrote: post #136
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

I disagree with this slightly, modern computer chips follow a general purpose architecture not special purpose ones. The reason for this is building a computer chip is expensive and difficult to do. Similarly building any useful language model requires tons of compute power, and very smart ML researchers. Most of the smaller open source one's are just trained on GPT output. By "cramming all of the web" on a model wha…

Since when did the world decide to shed the grammatically correct "computing power" for this weird tech bro "compute power" phrase?

Re: Many in the AI field think the bigger-is-better approach is running out of road

#140

Post by goatlover »

Legend2440 wrote: post #122
anikan_vader wrote: post #117

Earlier quoted context omitted.

There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.

Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see. Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.

Or the Many Worlds Interpretation is the correct understanding of quantum mechanics. The MWI people will say that indeterminism comes from the Copenhagen idea of there being a random collapse. But since measuring devices and human brains are also quantum systems, there's no reason to propose a collapse. Decoherence would be the reason we only see one result.
Post reply on HN