Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is ChatGPT's behavior changing over time?

arxiv.org

Post reply on HN
Print view View original

11–20 of 187 posts

Re: How is ChatGPT's behavior changing over time?

#11

Post by sanxiyn »

dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Or OpenAI can stop being stupid and adopt LLaMA-like tokenization, which special cases numbers and tokenize them into individual digits.

Re: How is ChatGPT's behavior changing over time?

#12

Post by ramesh31 »

nomel wrote: post #10
ramesh31 wrote: post #7

OpenAI is going to be left in the dust by (actual) open models. Llama 2 is already reaching GPT-3 levels, and can run inference on consumer hardware. Crazy how fast that flipped.

What enables this? There's a huge gap between GPT-3.5 and 4, put there by a massive amount of money, from my understanding. To compete, with open source projects being less well funded, I would assume that orders of magnitude improvements in training cost would be required. What do you see driving this, and who do you see paying for it? If Meta, or anyone else, gets something that beats GPT-4, I would naively assume…

>Or, are you suggesting that GPT-4 is the pinnacle of OpenAI, or that their funding will run dry?

My bet is that Meta has pivoted almost entirely to this space with their R&D in the last six months. Llama 2 is spectacular. And with its' success, there will undoubtedly be more. They also happen to have access to limitless amounts of compute, cash, and engineering that puts OpenAI to shame. This could finally be their chance to create a platform for real. And the open source community and startups will benefit off of that.

Re: How is ChatGPT's behavior changing over time?

#13

Post by ttul »

I tried the prime number problem and GPT-4 nailed it. I’m not sure whether they are testing things correctly…

“ Sure, let's go step by step.

A prime number is a number greater than 1 that has no positive divisors other than 1 and itself. This means if we can find any other number (excluding 1 and the number itself) that divides 17077, then it is not a prime number.

Let's start by checking divisibility by 2. Since 17077 is an odd number (its last digit is 7), it is not divisible by 2.

For the next step, we can look at divisibility by 3. We add up the digits: 1+7+0+7+7 = 22. Because 22 is not divisible by 3, 17077 is also not divisible by 3.

The smallest prime number greater than 3 is 5. Checking divisibility by 5, we see that 17077 does not end in 0 or 5, so it is not divisible by 5.

The smallest prime number greater than 5 is 7. To check divisibility by 7, there is no simple rule like with 2, 3, and 5, so we would perform the actual division. But we can stop here because I just recalled a rule that we can only check up to the square root of the number and the square root of 17077 is approximately 130.68. So if there were factors other than 1 and 17077, we would have found them by now.

Therefore, 17077 is a prime number.”

Re: How is ChatGPT's behavior changing over time?

#14

Post by blobbers »

A little surprised this sort of thing passes for a publish-able paper.

Isn't it the equivalent to saying "Here's the top 10 results for google searching golden retrievers March 2023, and here's the top 10 results from June 2023. We see that google is returning even cuter animals today. Unfortunately though, one of the results linked to a page full of cats."

I'm sure openai has a list of standard questions that it tracks the responses it is getting over a period of time with perfect knowledge / version timing of their own releases.

This does not seem like valid research / publication.

Re: How is ChatGPT's behavior changing over time?

#15

Post by jinay »

sanxiyn wrote: post #11
dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Or OpenAI can stop being stupid and adopt LLaMA-like tokenization, which special cases numbers and tokenize them into individual digits.

Are LLaMA-based models better at math because of this digit-based tokenization?

Re: How is ChatGPT's behavior changing over time?

#16

Post by bugglebeetle »

sanxiyn wrote: post #11
dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Or OpenAI can stop being stupid and adopt LLaMA-like tokenization, which special cases numbers and tokenize them into individual digits.

Isn’t the tokenization tied to the training of the model?

Re: How is ChatGPT's behavior changing over time?

#17

Post by swyx »

Previous commentary I know of from OpenAI staff:

Logan: The API does not just change without us telling you. The models are static there. https://twitter.com/OfficialLoganK/status/166393494793189785... may 31

Peter: No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. https://twitter.com/jlowin/status/1679660938415177731 july 14

either the models are static, or they are being improved continuously and there have been unforeseen regressions. only one can be true at any point in time. was this policy changed in the last 1.5 months?

Re: How is ChatGPT's behavior changing over time?

#18

Post by [deleted] »

ramesh31 wrote: post #12
nomel wrote: post #10

Earlier quoted context omitted.

What enables this? There's a huge gap between GPT-3.5 and 4, put there by a massive amount of money, from my understanding. To compete, with open source projects being less well funded, I would assume that orders of magnitude improvements in training cost would be required. What do you see driving this, and who do you see paying for it? If Meta, or anyone else, gets something that beats GPT-4, I would naively assume…

>Or, are you suggesting that GPT-4 is the pinnacle of OpenAI, or that their funding will run dry? My bet is that Meta has pivoted almost entirely to this space with their R&D in the last six months. Llama 2 is spectacular. And with its' success, there will undoubtedly be more. They also happen to have access to limitless amounts of compute, cash, and engineering that puts OpenAI to shame. This could finally be their…

[deleted]

Re: How is ChatGPT's behavior changing over time?

#19

Post by flangola7 »

ramesh31 wrote: post #7

OpenAI is going to be left in the dust by (actual) open models. Llama 2 is already reaching GPT-3 levels, and can run inference on consumer hardware. Crazy how fast that flipped.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval.

Furthermore, Llama remains well below GPT-3 on human rated tests such as programming, and GPT-3 is already over three years old. It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are even more lacking in capability. Full power Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.

OpenAI does not have a moat, but they do have a very high wall.

Re: How is ChatGPT's behavior changing over time?

#20

Post by flangola7 »

swyx wrote: post #17

Previous commentary I know of from OpenAI staff: Logan: The API does not just change without us telling you. The models are static there. https://twitter.com/OfficialLoganK/status/166393494793189785... may 31 Peter: No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. https://twitter.com/jlowin/status/1679660938415177731 july 14 either the models are static, or…

You can call a specific version of the model. It's ones of the API values. The latter person is referring to the "gpt-4" which the documentation states will update and change without warning.
Post reply on HN