Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT turned generative AI into an “anything tool”

arstechnica.com

Post reply on HN
Print view View original

31–40 of 245 posts

Re: ChatGPT turned generative AI into an “anything tool”

#31

Post by danielbln »

Sohcahtoa82 wrote: post #23
atemerev wrote: post #3

This is what we used to call an "AGI" back in 2013-2014. Generalized artificial intelligence. An AI which is general enough to be used for diverse tasks in multiple fields, unlike specialized AIs of that time (e.g. for image recognition or self-driving cars). Nothing more than that. Now the goalposts are moved way forward, and now "AGI" is something that have conscious, with superhuman intelligence, and probably evil…

> But back in a day, AGI was just what we have today -- an universal chat bot. I don't know anybody that used AGI to mean that. I've always thought of AGI as an AI that can learn how to perform tasks with very simple instructions, rather than spending weeks training a neural network on a data center full of GPGPUs. Basically, my idea of AGI has always been what you're calling the "new" definition of AGI.

Aren't you effectively describing few-shot learning in LLMs? Teach the model a new task by describing it and maybe give a few examples?

Re: ChatGPT turned generative AI into an “anything tool”

#32

Post by theRealMe »

wongarsu wrote: post #7
Someone1234 wrote: post #2

Although let's see if it is sustainable. I pay for ChatGPT Plus but most people aren't, and from what I've read they're losing money like crazy. We may wind up seeing this as the technical high point before the various competitors cut back on expensive hardware/energy usage for the free tier, and the quality of the responses takes a dive. This might sound a bit conspiratorial so apologies for that but back when the c…

So far OpenAI's modus operandi was "does it get better if we make it bigger". But now we've reached "good enough" (arguably already with GPT3.5, as evidenced by free ChatGPT running on GPT3.5-turbo). There has been promising work about making models smaller without losing performance (e.g. by training them longer), or quantizing the weights (llama.cpp etc) to make the model cheaper to run. I believe we will see a lot…

I don’t want to make unsubstantiated guesses, but we should remember that GPT3.5 was originally just GPT3.5 and when they came out with turbo it not only sped it up but also reduced the cost (to us) by like 10x of what it used to be if I remember right. So they are (or at the very least “were”) working on and succeeding on reducing its cost to run.

Re: ChatGPT turned generative AI into an “anything tool”

#33

Post by viscanti »

malwrar wrote: post #27
Someone1234 wrote: post #2

Although let's see if it is sustainable. I pay for ChatGPT Plus but most people aren't, and from what I've read they're losing money like crazy. We may wind up seeing this as the technical high point before the various competitors cut back on expensive hardware/energy usage for the free tier, and the quality of the responses takes a dive. This might sound a bit conspiratorial so apologies for that but back when the c…

The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. I think at this point there are enough people knowledgeable and interested in open LM technology that, if openai begins to degrade in quality, we won’t lose this technology forever. I’ve personally been obsessively reading papers about new models…

> The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down.

Do we know that there's zero benefit from the user generated data they're getting now? They know when someone clicks the button to regenerate, and in theory they can have GPT4 review all the responses and classify if they're good or not. I don't know how beneficial that data is, so I'm curious if it's been proven to be completely worthless or not. If it's valuable at all, and openai is using it to continually improve the performance of ChatGPT, then maybe it will be difficult for a competitor to ever get traffic and data to their alternative.

Re: ChatGPT turned generative AI into an “anything tool”

#34

Post by dandellion »

JimtheCoder wrote: post #16
unshavedyak wrote: post #8

Earlier quoted context omitted.

On the plus side i think at-home LLMs will continue to get a lot better. ChatGPT 3.5/4 both seem in reach for Llama2 and seeing all the amazing fine tunings that were available for Llama1, i suspect we have a very functional future of at home LLMs. I say functional because i don't think we've seen the larger reasoning skills scale well for smaller deployments, so that may be out of reach. But i think a future were we…

"I look forward to hooking all my at-home cameras up to a voice recognition, image recognition, and LLM detection suite and just speak aloud and have the AI-suite "do stuff". It sounds really, really fun to me." Alternatively, this sounds like the starting plot to a crossover sci-fi/horror movie...

The door refused to open. It said, “Five cents, please.”

He searched his pockets. No more coins; nothing. “I’ll pay you tomorrow,” he told the door. Again he tried the knob. Again it remained locked tight. “What I pay you,” he informed it, “is in the nature of a gratuity; I don’t have to pay you.”

“I think otherwise,” the door said. “Look in the purchase contract you signed when you bought this conapt.”

In his desk drawer he found the contract; since signing it he had found it necessary to refer to the document many times. Sure enough; payment to his door for opening and shutting constituted a mandatory fee. Not a tip.

“You discover I’m right,” the door said. It sounded smug.

From the drawer beside the sink Joe Chip got a stainless steel knife; with it he began systematically to unscrew the bolt assembly of his apt’s money-gulping door.

“I’ll sue you,” the door said as the first screw fell out.

Joe Chip said, “I’ve never been sued by a door. But I guess I can live through it.

-- Philip K. Dick, Ubik

Re: ChatGPT turned generative AI into an “anything tool”

#35

Post by assistantist »

spott wrote: post #28
Someone1234 wrote: post #2

Although let's see if it is sustainable. I pay for ChatGPT Plus but most people aren't, and from what I've read they're losing money like crazy. We may wind up seeing this as the technical high point before the various competitors cut back on expensive hardware/energy usage for the free tier, and the quality of the responses takes a dive. This might sound a bit conspiratorial so apologies for that but back when the c…

If it was just computer that caused the various assistants to get bad, then why haven’t they gotten good again? Google assistant came out in 2016! That is 7 years ago! Since then, GPU compute in flops has 5xed… for single precision. If you include half-floats (not a thing back then), it is 75x the single precision flops. I think the reason that assistants have “gotten bad” is a combination of edge case controls (avoi…

If it was just a matter of flops then AI would already be driving cars and making door to door food deliveries. The problem is neither the energy expenditure nor the compute capacity. No one knows what the proper architecture should be for general purpose intelligence assistants so everyone has to fine-tune the existing models on their specialized use cases and even then it's a hit or miss depending on how much training data you have available.

No one currently has any ideas on how to deploy auto-adaptive ML software that continuously adapts to new data so the industry has settled on deploying snapshots and then hoping the data set that was used for the snapshot is going to be good enough for most use cases. It seems to work well enough for now and since Facebook has decided to open source most of their work someone in the open source community might figure out how to continously update these models without deterioration in output quality.

Re: ChatGPT turned generative AI into an “anything tool”

#36

Post by thfuran »

assistantist wrote: post #35
spott wrote: post #28

Earlier quoted context omitted.

If it was just computer that caused the various assistants to get bad, then why haven’t they gotten good again? Google assistant came out in 2016! That is 7 years ago! Since then, GPU compute in flops has 5xed… for single precision. If you include half-floats (not a thing back then), it is 75x the single precision flops. I think the reason that assistants have “gotten bad” is a combination of edge case controls (avoi…

If it was just a matter of flops then AI would already be driving cars and making door to door food deliveries. The problem is neither the energy expenditure nor the compute capacity. No one knows what the proper architecture should be for general purpose intelligence assistants so everyone has to fine-tune the existing models on their specialized use cases and even then it's a hit or miss depending on how much train…

The difficulty of improving AI capability towards AGI isn't really related to a purported worsening of an already-implemented system for financial reasons.

Re: ChatGPT turned generative AI into an “anything tool”

#37

Post by malwrar »

viscanti wrote: post #33
malwrar wrote: post #27

Earlier quoted context omitted.

The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. I think at this point there are enough people knowledgeable and interested in open LM technology that, if openai begins to degrade in quality, we won’t lose this technology forever. I’ve personally been obsessively reading papers about new models…

> The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. Do we know that there's zero benefit from the user generated data they're getting now? They know when someone clicks the button to regenerate, and in theory they can have GPT4 review all the responses and classify if they're good or not. I don'…

I don’t know about “zero benefit”—certainly their access to customer queries gives them a gods eye view into how people are using their LM implementation and to some degree could probably be used to fine-tune their model—but I doubt it gives them a leg up that couldn’t be matched through the sheer scale of academics and volunteers who contribute to open solutions & research. Maybe we invent a method for directly fine-tuning based on user interactions, or we discover that there’s some shelf where further fine-tuning only confuses the model, or discover some magical architectural modification or dataset that enables high-quality interactions. We’re still at the stage where basic modifications to the model breaks seemingly insurmountable problems. ALiBi for example swaps in a new attention mechanism, literally just a matrix addition operation, and found that it enables inference far past the context limit.

I’m ultimately optimistic, their data is valuable but I don’t think it’s insurmountable.

Re: ChatGPT turned generative AI into an “anything tool”

#38

Post by financialisty »

thfuran wrote: post #36
assistantist wrote: post #35

Earlier quoted context omitted.

If it was just a matter of flops then AI would already be driving cars and making door to door food deliveries. The problem is neither the energy expenditure nor the compute capacity. No one knows what the proper architecture should be for general purpose intelligence assistants so everyone has to fine-tune the existing models on their specialized use cases and even then it's a hit or miss depending on how much train…

The difficulty of improving AI capability towards AGI isn't really related to a purported worsening of an already-implemented system for financial reasons.

If money is the bottleneck then we will never get AGI. A superior machine intelligence would have no use for money.

Re: ChatGPT turned generative AI into an “anything tool”

#39

Post by atemerev »

danpalmer wrote: post #26
atemerev wrote: post #12

Earlier quoted context omitted.

Games are solved since AlphaZero. Just set up an adversarial model, and voila. Any game, any conditions. Proactive models are simple — just generate the “internal monologue” periodically from the system prompt, mix it with the compressed state, then feed it to another model (the outputs are used as state updates). I think this is how human agency works as well. Movement is a different story (LLMs do not work here, ex…

AlphaZero is amazing, but it only plays 2D games with well defined win conditions.

Yes, like military strategy.

Re: ChatGPT turned generative AI into an “anything tool”

#40

Post by danpalmer »

atemerev wrote: post #39
danpalmer wrote: post #26

Earlier quoted context omitted.

AlphaZero is amazing, but it only plays 2D games with well defined win conditions.

Yes, like military strategy.

Indeed, this is something that neither AlphaZero nor anything else has managed to do at anything approaching human level.
Post reply on HN