Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is ChatGPT's behavior changing over time?

arxiv.org

Post reply on HN
Print view View original

51–60 of 187 posts

Re: How is ChatGPT's behavior changing over time?

#51

Post by ramesh31 »

nomel wrote: post #37
ramesh31 wrote: post #12

Earlier quoted context omitted.

>Or, are you suggesting that GPT-4 is the pinnacle of OpenAI, or that their funding will run dry? My bet is that Meta has pivoted almost entirely to this space with their R&D in the last six months. Llama 2 is spectacular. And with its' success, there will undoubtedly be more. They also happen to have access to limitless amounts of compute, cash, and engineering that puts OpenAI to shame. This could finally be their…

What motive do you see for them releasing these models for free, especially with the massive increase in cost associated with catching up?

> What motive do you see for them releasing these models for free, especially with the massive increase in cost associated with catching up?

Owning a platform. Zuck's dream.

They put tons of money into these models, nurture an ecosystem of companies built around them, and then start gradually figuring out a licensing model for the ones that take off.

Re: How is ChatGPT's behavior changing over time?

#52

Post by littlestymaar »

YZF wrote: post #40
ttul wrote: post #13

I tried the prime number problem and GPT-4 nailed it. I’m not sure whether they are testing things correctly… “ Sure, let's go step by step. A prime number is a number greater than 1 that has no positive divisors other than 1 and itself. This means if we can find any other number (excluding 1 and the number itself) that divides 17077, then it is not a prime number. Let's start by checking divisibility by 2. Since 170…

Aren't there more potential primes between 130 and 7?

Between 5 and 130 actually, it didn't even checked 7!

Re: How is ChatGPT's behavior changing over time?

#53

Post by barrkel »

Irritatingly, OpenAI reps deny any change in model capabilities over time. It's more likely that as the models are optimized for cost and performance, their in-house evaluation metrics don't cover everything customers are interested in. Meanwhile, the probabilistic nature of LLM output means there is plausible deniability.

Re: How is ChatGPT's behavior changing over time?

#54

Post by littlestymaar »

nomel wrote: post #37
ramesh31 wrote: post #12

Earlier quoted context omitted.

>Or, are you suggesting that GPT-4 is the pinnacle of OpenAI, or that their funding will run dry? My bet is that Meta has pivoted almost entirely to this space with their R&D in the last six months. Llama 2 is spectacular. And with its' success, there will undoubtedly be more. They also happen to have access to limitless amounts of compute, cash, and engineering that puts OpenAI to shame. This could finally be their…

What motive do you see for them releasing these models for free, especially with the massive increase in cost associated with catching up?

Kill Op*nAI, for starter. If they see it as a threat, commoditizing the tech is a great way to get rid of them at a reasonable cost.

Re: How is ChatGPT's behavior changing over time?

#55

Post by lumost »

xp84 wrote: post #43
dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

I used GPT-4 to generate a non-cryptographic random 64 character string. It was faster to ask GPT-4 for the string than ask GPT-4 for the instructions to generate the string from my terminal. GPT-4 was faster than google.

Re: How is ChatGPT's behavior changing over time?

#56

Post by littlestymaar »

flangola7 wrote: post #19
ramesh31 wrote: post #7

OpenAI is going to be left in the dust by (actual) open models. Llama 2 is already reaching GPT-3 levels, and can run inference on consumer hardware. Crazy how fast that flipped.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval. Furthermore, Llama remains well below GPT-3 on human rated tes…

> Furthermore, Llama remains well below GPT-3 on human rated tests such as programming, and GPT-3 is already over three years old.

It's just not part of the entended use-case, and hasn't been trained to do so. It's almost like complaining that StableDiffusion isn't good at text generation…

> It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are even more lacking in capability

The biggest models will be abble to run on the CPU just fine with llama.cpp as long as you have enough (cheap) RAM. Sure it's slow, but you can run it.

> Full power Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.

What is that “per inference session” doing here? You pay the hardware only once you know… (and the number of kilowatt isn't per inference session either, the number of Watt•hour is)

Re: How is ChatGPT's behavior changing over time?

#57

Post by gyrovagueGeist »

ramesh31 wrote: post #47
flangola7 wrote: post #19

Earlier quoted context omitted.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval. Furthermore, Llama remains well below GPT-3 on human rated tes…

>Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. Note that I never said "open source" just "open models". As in, I can now actually build things with GPT3 capability that run locally. I couldn't care less about the model code. >Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session. I'm running llama-2-7b-chat on my 8…

If you compiled llama.cpp with Metal I wouldn’t say its ~all CPU based. But it is still incredible! I need to set this up myself this weekend :)

Re: How is ChatGPT's behavior changing over time?

#58

Post by NLPaep »

barrkel wrote: post #53

Irritatingly, OpenAI reps deny any change in model capabilities over time. It's more likely that as the models are optimized for cost and performance, their in-house evaluation metrics don't cover everything customers are interested in. Meanwhile, the probabilistic nature of LLM output means there is plausible deniability.

Source? They’ve only denied the API model changing. Not the website.

Re: How is ChatGPT's behavior changing over time?

#59

Post by barrkel »

NLPaep wrote: post #58
barrkel wrote: post #53

Irritatingly, OpenAI reps deny any change in model capabilities over time. It's more likely that as the models are optimized for cost and performance, their in-house evaluation metrics don't cover everything customers are interested in. Meanwhile, the probabilistic nature of LLM output means there is plausible deniability.

Source? They’ve only denied the API model changing. Not the website.

https://twitter.com/npew/status/1679538687854661637?t=3CLXlT...
Post reply on HN