Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is ChatGPT's behavior changing over time?

arxiv.org

Post reply on HN
Print view View original

141–150 of 187 posts

Re: How is ChatGPT's behavior changing over time?

#141

Post by barrkel »

blowski wrote: post #61
barrkel wrote: post #59

Earlier quoted context omitted.

https://twitter.com/npew/status/1679538687854661637?t=3CLXlT...

(Tweet contents) > No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. > Current hypothesis: When you use it more heavily, you start noticing issues you didn't see before. I don’t see how it supports your argument. Your comment says “they deny making changes to GPT-4”, and the tweet says “we are making incremental improvements to GPT-4”.

I did not say that they deny making changes. I said that they denied changes in capabilities; implied, in this context, is changes for the worse, since changes for the better should not affect people's experiences negatively.

Re: How is ChatGPT's behavior changing over time?

#143

Post by tough »

dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Triple quotes sound like how you do markdown code blocks to me

``` code ```

Re: How is ChatGPT's behavior changing over time?

#144

Post by Sosh101 »

ramesh31 wrote: post #91
Sosh101 wrote: post #36

Earlier quoted context omitted.

> and $10,000+ of compute hardware per inference session. What hardware would you need to run it at home?

>What hardware would you need to run it at home? Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma... Step 2: https://github.com/ggerganov/llama.cpp Step 3: you're welcome

That's very helpful, thank you.

Re: How is ChatGPT's behavior changing over time?

#145

Post by edgyquant »

zeusk wrote: post #136
hammock wrote: post #133

Earlier quoted context omitted.

>"We" are never going to stop trying to use LLMs for math. They are obviously mimicking a smart person. A lot of words to make a big deal out of nothing. All that is needed is some new abstracted layer that identifies a math question and then proxies it over to the wolfram plugin. That’s it We don’t have crazy debates over whether a polygon should be rendered by the cpu or a gpu. We solved this problem

So a Mixture of Experts model; but then isn't ChatGPT using that already? Why're the models so bad at math despite being trained on academic papers and books clearly - or why do they hallucinate and make up non-existing citations?

Because they don’t actually reason about things they just map words based on probability. If they’ve seen a math problem enough they may get it right due to probability, but using that knowledge to solve a new problem isn’t likely.

Re: How is ChatGPT's behavior changing over time?

#146

Post by vicentwu »

xp84 wrote: post #43
dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

Asking models to do math is kind of an effecitve way to measure their capabilities, especially in reasoning and abstraction, which are quite important for problem solving.

Re: How is ChatGPT's behavior changing over time?

#147

Post by viraptor »

vicentwu wrote: post #146
xp84 wrote: post #43

Earlier quoted context omitted.

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

Asking models to do math is kind of an effecitve way to measure their capabilities, especially in reasoning and abstraction, which are quite important for problem solving.

You don't need to reason and abstraction to do basic calculation. ChatGPT will however happily give you some decent answers about not-too -hard math that requires reasoning. It just won't operate on digits.

Those are completely different ideas.

Re: How is ChatGPT's behavior changing over time?

#148

Post by viraptor »

Closi wrote: post #104
xp84 wrote: post #43

Earlier quoted context omitted.

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

> Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion. Using GPT to do maths is probably like using a 737 to drive around on the ground. Teaching GPT to do maths mi…

Well that's... a claim. What's your reasoning for operations on digits within the model itself improving any current metrics?

Re: How is ChatGPT's behavior changing over time?

#149

Post by avion23 »

Sosh101 wrote: post #36
flangola7 wrote: post #19

Earlier quoted context omitted.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval. Furthermore, Llama remains well below GPT-3 on human rated tes…

> and $10,000+ of compute hardware per inference session. What hardware would you need to run it at home?

> and $10,000+ of compute hardware per inference session.

That is not true. A common macbook with lots of RAM (>32GB) is enough. Or any x86 computer with lots of RAM. llama.cpp is CPU only and quite fast

Re: How is ChatGPT's behavior changing over time?

#150

Post by firewolf34 »

wizofaus wrote: post #111
SanderNL wrote: post #98

Earlier quoted context omitted.

It’s not about the results, it’s about its ability to “reason”. Math is about as close to pure reasoning we get so I don’t get the pessimism. If it is bad at math and can’t be taught, then you have a fundamental problem. It’s a matter of time before this limit gets hit in other domains.

This seems fair enough to me. That ChatGPT currently struggles with certain types of maths problems points to reasonably fundamental shortcomings in what otherwise appears to have the beginnings of a general purpose reasoning engine (whether you consider it AGI or not), and I'm willing to bet extraordinarily clever minds are working hard on trying to address those shortcomings.

I don't think that trying to shoehorn LLM into being AGI is the right path. Maybe some are trying to achieve this, or are hoping for this... but IMO this is trying to fit a square peg into a round hole. It's definitely an element in the overall puzzle, I think we can all agree, though. Even so, rather than warping this hammer to also screw in a bolt, why not combine it with another tool more fit for the job?
Post reply on HN