Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unpredictable abilities emerging from large AI models

quantamagazine.org

Post reply on HN
Print view View original

281–290 of 326 posts

Re: Unpredictable abilities emerging from large AI models

#281

Post by ogogmad »

bbor wrote: post #270
hgsgm wrote: post #227

Earlier quoted context omitted.

What's even more bizarre is the ChatGPT is proving him right , that a neural net can build logical grammar, and he is denying it!

This is a hot take, would love a link/explanation on why you think neural nets are human-like. For example, from the op Ed you’re likely referencing: “ Their deepest flaw is the absence of the most critical capacity of any intelligence: to say not only what is the case, what was the case and what will be the case — that’s description and prediction — but also what is not the case and what could and could not be the c…

Why do you think this is impossible? Have you tried making an account on chat.openai.com? Because it can in fact do that right now; and even people who are otherwise skeptical of LLMs have panned Chomsky's article.

Re: Unpredictable abilities emerging from large AI models

#282

Post by datpiff »

saiojd wrote: post #258
datpiff wrote: post #93

Earlier quoted context omitted.

> According to NVidia, LLM sizes have been increasing 10X per year for the last few years. Clearly this cannot continue, as the training costs will exceed all the compute capacity in existence. The other limit is training data, eventually you run out of cheap sources.

I would not be so sure about compute capacity? Neural network architectures are still in their infancy, it is very likely that more efficient approaches exist.

Sure, but the small & efficient networks so far have been pretty poor. Edge AI was hugely hyped a few years ago and it petered out. All of the big tech companies seems to be chasing the biggest networks with the largest training sets, that seems to be the direction for now.

There could be huge efficiency savings within the implementations but when training costs are already this high it seems naive to think the low-hanging fruit is still there.

Re: Unpredictable abilities emerging from large AI models

#283

Post by coldtea »

Karrot_Kream wrote: post #236
coldtea wrote: post #203

Earlier quoted context omitted.

> Everywhere I've worked there's been significantly more stuff we wanted to do than we could actually do. Speeding that up looks awesome to me. Does it? Slowing our breakneck anti-human business pace and stopping mindless and environmentally and culturally harmful consumption and production of trinkets is surely better!

Huh? Have you attended a local city council meeting recently? I can guarantee you they're running on shoestring budgets with an endless backlog of requests that they are only treading water on because of anemic tax revenue and low prestige for government jobs. "Productivity" isn't just some high tech, consumer bauble producing phenomenon. It helps everyone out. Highly profitable, headcount-bloated big companies are t…

I can guarantee you that the reason they do so, is an explosion of BS requests, extreme bureucracy, and needless requirements, with a simultaneous drop in actual useful basic work - on city instrastructure and such.

Re: Unpredictable abilities emerging from large AI models

#284

Post by zzleeper »

mark_l_watson wrote: post #144

Nice write up! I have been using classic back-prop neural networks since the 1980s, and deep learning for the last 8 years. This tech feels like a rocket ship that is accelerating exponentially! I am in my 70s and I don't work much anymore. That said, I find myself spending many hours in a typical day doing what I call "gentleman scientist" activities around Large Language Models. I was walking this morning with a no…

Hi Mark,

One question out of academic curiosity: I'm exploring ways to use these tools for research projects in econ and am struggling to see a good angle.

For instance, suppose I have lots of PDF reports on how firms have evolved on each quarter (10,000 reports or any other number beyond what I can read).

Can LLMs be used to spit out variables based on these reports? EG: indicator variables (optimistic-vs-pesimistic), categories, etc. that I can then use as data to test economic models?

I tried simple cases with ChatGPT3.5 and got the feeling it's great for outputting narrative text, but felt mediocre for when the output was more narrowly defined into categories (which was surprising).

Re: Unpredictable abilities emerging from large AI models

#285

Post by HervalFreire »

coldtea wrote: post #206
HervalFreire wrote: post #92

Earlier quoted context omitted.

What they said 10 years ago was correct. It did hit society like a sledge hammer. Machine learning basically took over the AI space and penetrated the consumer space with applications that were all but impossible in the previous decade. There's AI chips in smart phones now. What you're seeing here with LLMs is sledge hammer number 2. It's understandable how most people don't notice the sledge hammer. The decade prior…

> What they said 10 years ago was correct. It did hit society like a sledge hammer. Machine learning basically took over the AI space and penetrated the consumer space with applications that were all but impossible in the previous decade. There's AI chips in smart phones now. And still almost all of these applications are not really impactful or that important compared to actually society and life changing developmen…

Of course we hit many technological limits long ago. Most miracle technologies have already been discovered. Heck space travel has had zero actual progress other then Elon redoing what we did decades ago more "efficiently".

The AI sledge hammer is as good as we are going to get in paradigm shifts for a while given how we are at the top of an s-curve for development. Additionally we're only really seeing the first decade. When the computer was discovered was it ubiquitous in a decade? No. In fact it was largely a sort of useless academic research project for the longest time.

Re: Unpredictable abilities emerging from large AI models

#286

Post by roflyear »

HervalFreire wrote: post #194
roflyear wrote: post #169

Earlier quoted context omitted.

Gloating over people being wrong is rarely nice

Not if it's said in a nice way. The commenter called LLMs a freaking auto complete. That's not nice either, but nice enough for HN. So I figured a little subtle gloating is deserved and well within the bounds of etiquette here on HN.

Well insulting the capacity of an LLM is much different than gloating over someone being wrong about their livelihood.

Re: Unpredictable abilities emerging from large AI models

#287

Post by roflyear »

UncleOxidant wrote: post #177
roflyear wrote: post #165

Earlier quoted context omitted.

I think it'll end at using chatgpt as a great tool. And that seems fine. I think much of the anxiety here is in the unknown. You have to learn your IDE, but it doesn't make programming harder. It makes it more fun actually. Chatgpt is going to require that devs are even better IMO because you have to catch its errors and know how to work with it.

Yes, this stuff is fun to work with and use as an aid to programming. But I think what's dawning on a lot of us is that it seems to be able to facilitate a large increase productivity. And given that large productivity increase will industry need as many software developers (and other knowledge workers)?

I mean, people aren't going to be able to solve 100x as many problems because of this.

Re: Unpredictable abilities emerging from large AI models

#288

Post by roflyear »

Karrot_Kream wrote: post #236
coldtea wrote: post #203

Earlier quoted context omitted.

> Everywhere I've worked there's been significantly more stuff we wanted to do than we could actually do. Speeding that up looks awesome to me. Does it? Slowing our breakneck anti-human business pace and stopping mindless and environmentally and culturally harmful consumption and production of trinkets is surely better!

Huh? Have you attended a local city council meeting recently? I can guarantee you they're running on shoestring budgets with an endless backlog of requests that they are only treading water on because of anemic tax revenue and low prestige for government jobs. "Productivity" isn't just some high tech, consumer bauble producing phenomenon. It helps everyone out. Highly profitable, headcount-bloated big companies are t…

> anemic tax revenue

You mean, spending >1/2 their budget on police?

Re: Unpredictable abilities emerging from large AI models

#289

Post by YeGoblynQueenne »

Animats wrote: post #245
guns wrote: post #91

Earlier quoted context omitted.

This is what Stephen Wolfram concludes in a recent article about ChatGPT: > The specific engineering of ChatGPT has made it quite compelling. But ultimately (at least until it can use outside tools) ChatGPT is “merely” pulling out some “coherent thread of text” from the “statistics of conventional wisdom” that it’s accumulated. But it’s amazing how human-like the results are. And as I’ve discussed, this suggests some…

> (at least until it can use outside tools) This is key. ChatGPT/GPT-4 alone are limited to reformulating what they know from their training data. Linked to search engines, databases, and computational tools such as Wolfram Alpha, they acquire much more capability. We're already seeing that with Microsoft Bing. (Update: what happens as large language models learn Excel? Especially since Microsoft is already connectin…

>> Is the Singularity happening right now?

No, but it seems everyone loves to LARP it anyway.

Re: Unpredictable abilities emerging from large AI models

#290

Post by YeGoblynQueenne »

whimsicalism wrote: post #174
mpalmer wrote: post #59

I'd like to see posts on LLMs written from a different perspective. For me, the surprise comes not from the sudden emergent capability of language models, but that the understanding (and synthesis!) of ideas encoded in language has succumbed to literally nothing more than statistical analysis. Or at least come that much closer to doing so. That it bears so close a resemblance to actual thinking says more about the im…

I find the phrase "statistical analysis" a frustrating one nowadays as it seems to have become a signal for "I hold a particular philosophy of the mind". I don't understand this use of "statistical" as a diminutive to describe these models. Why can't incredibly complicated behavior be emergent from matrix multiplication subject to optimization in the same way that our biological matter has developed complicated emerg…

Perhaps that's because you haven't asked a single good question?

What do matrix multiplication and optimisation have to do with the way the human mind, or the human brain work? That they have anything to do at all, is your assumption, that you seem absolutely convinced about- and then you go asking people why can't it be true? You say why it is true. It's your assumption.

Matrix multiplication and optimization are human mathematical techniques. They have about as good a chance of being something that exists in nature independently of humans as Magic: the Gathering and Call of Duty. They might be useful models to help us understand how things work, but to assume they are how things work is a huge leap of faith.

Ask the right questions and then an answer may even suggest itself. Ask questions that follow from your preconceived answers and you're in a world of fantasy.

Post reply on HN