Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is ChatGPT's behavior changing over time?

arxiv.org

Post reply on HN
Print view View original

71–80 of 187 posts

Re: How is ChatGPT's behavior changing over time?

#71

Post by flangola7 »

littlestymaar wrote: post #56
flangola7 wrote: post #19

Earlier quoted context omitted.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval. Furthermore, Llama remains well below GPT-3 on human rated tes…

> Furthermore, Llama remains well below GPT-3 on human rated tests such as programming, and GPT-3 is already over three years old. It's just not part of the entended use-case, and hasn't been trained to do so. It's almost like complaining that StableDiffusion isn't good at text generation… > It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are e…

>It's just not part of the entended use-case, and hasn't been trained to do so. It's almost like complaining that StableDiffusion isn't good at text generation…

For the most part none of these systems were trained to do anything. The capabilities are emergent. My statement about the benchmarks remains unfazed.

>The biggest models will be abble to run on the CPU just fine with llama.cpp as long as you have enough (cheap) RAM. Sure it's slow, but you can run it.

Yes, but what products/clients are you pitching where that kind of wait will be acceptable? Time is money. Standing up racks of maxed out RAM server slots is still far from inexpensive too.

>What is that “per inference session” doing here? You pay the hardware only once you know… (and the number of kilowatt isn't per inference session either, the number of Watt•hour is)

Good job at completely missing the point. I know what a kilowatt is and it is the unit I meant, not kilowatt-hour. I'm referring to the hardware necessary to run one user session of inference. While user 1's tokens are generating, users 2 and up are in the queue waiting. If you want to serve multiple users simultaneously you will need to invest in multiple $xx,xxx units of hardware each requiring a multi-kW electrical circuit.

There is a reason ChatGPT incurs electrical bills on the order of a million dollars per day.

Re: How is ChatGPT's behavior changing over time?

#72

Post by deelly »

dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

No, we definitely should continue to quiz LLMs on mathematics and absolutely any other topics. Otherwise how do we know and understand limitation of the system?

Re: How is ChatGPT's behavior changing over time?

#73

Post by ThrowawayTestr »

lumost wrote: post #55
xp84 wrote: post #43

Earlier quoted context omitted.

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

I used GPT-4 to generate a non-cryptographic random 64 character string. It was faster to ask GPT-4 for the string than ask GPT-4 for the instructions to generate the string from my terminal. GPT-4 was faster than google.

Surely navigating to random.org is faster than typing a prompt.

Re: How is ChatGPT's behavior changing over time?

#74

Post by oefnak »

lumost wrote: post #55
xp84 wrote: post #43

Earlier quoted context omitted.

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

I used GPT-4 to generate a non-cryptographic random 64 character string. It was faster to ask GPT-4 for the string than ask GPT-4 for the instructions to generate the string from my terminal. GPT-4 was faster than google.

That definitely won't be random.

Re: How is ChatGPT's behavior changing over time?

#76

Post by dontupvoteme »

dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Could not agree more. It doesn't understand what a number is, why is everyone trying to quiz it on maths instead of, perhaps, seeing how good it is at language tasks, or even foreign languages? I suspect it has gotten a lot worse in non-english since launch.

Re: How is ChatGPT's behavior changing over time?

#77

Post by dontupvoteme »

esperent wrote: post #68
blowski wrote: post #61

Earlier quoted context omitted.

(Tweet contents) > No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. > Current hypothesis: When you use it more heavily, you start noticing issues you didn't see before. I don’t see how it supports your argument. Your comment says “they deny making changes to GPT-4”, and the tweet says “we are making incremental improvements to GPT-4”.

It's important to read company marketing statements as if you were a lawyer. We haven't made GPT-4 dumber != We haven't made ChatGPT(4) dumber. Personal hypothesis is that they have made a few changes to ChatGPT recently - possibly quantization, and almost certainly some tweaks to make it give shorter/less detailed answers. But by the nature of a probibalistic tool being run by a secretive company, it's hard to say f…

They're also going to claim "that's alignment, we're making it less evil, not dumber"

Re: How is ChatGPT's behavior changing over time?

#78

Post by delusional »

dontupvoteme wrote: post #76
dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Could not agree more. It doesn't understand what a number is, why is everyone trying to quiz it on maths instead of, perhaps, seeing how good it is at language tasks, or even foreign languages? I suspect it has gotten a lot worse in non-english since launch.

it "understands" numbers the exact same amount that it understands words.

Re: How is ChatGPT's behavior changing over time?

#79

Post by blobbers »

ineedasername wrote: post #33
blobbers wrote: post #24

Earlier quoted context omitted.

Not to seem sarcastic, but 500 randomly selected primes doesn't change my mind. I read the paper, and dumbed it down to dogs and cats. Do you really feel like the efforts were significant or meaningful?

>I read the paper, and dumbed it down to dogs and cats. I could read the Pricicipia, or anything, and dumb it down to dogs and cats, but that reduction to the dumb would be a failing all my own. As for the test-- a classification task benchmarked with 500 examples is a fairly decent test of a system's capabilities, be it LLM or a traditional machine learning model. And again, this was only one of a variety of tasks.…

Well, for starters, you take a model with 100 trillion parameters, and then you test it with... 500 examples. WAIT did you say hundred? 500 hundred? Not 500 hundred thousand right?

Okay okay hold on back to the drawing board.

...dumbed down to cats and dogs.

Re: How is ChatGPT's behavior changing over time?

#80

Post by vermilingua »

delusional wrote: post #78
dontupvoteme wrote: post #76

Earlier quoted context omitted.

Could not agree more. It doesn't understand what a number is, why is everyone trying to quiz it on maths instead of, perhaps, seeing how good it is at language tasks, or even foreign languages? I suspect it has gotten a lot worse in non-english since launch.

it "understands" numbers the exact same amount that it understands words.

Well, no, because numbers are “words” that represent something that behaves in a very different way to words in a sentance.
Post reply on HN