Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is ChatGPT's behavior changing over time?

arxiv.org

Post reply on HN
Print view View original

41–50 of 187 posts

Re: How is ChatGPT's behavior changing over time?

#41

Post by airgapstopgap »

BoorishBears wrote: post #32
airgapstopgap wrote: post #25

Earlier quoted context omitted.

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is goi…

I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works. At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The web app they host isn't a simple API wrapper, it does summarization, has some sort of system p…

No, it makes sense to secure engagement with the most expensive implementation and then cut costs, this kind of stuff is pervasive in the industry. Besides, we have Brockman on record saying that they do "a lot of quantization"[1][2] so it's not paranoia to suspect other optimization schemes when there's a clear performance drop, which they have also denied a few times.

1. https://chat.openai.com/share/44a0c5b6-c629-470a-992f-8cdbbe...

2. https://www.youtube.com/watch?v=_hpuPi7YZX8

Re: How is ChatGPT's behavior changing over time?

#42

Post by s3p »

Commented this on a submission that was a duplicate of this one, but:

When GPT's details were allegedly leaked [0] I read about the mixture of experts and wondered if this explains my recent distaste for GPT. Lately I have been using Anthropic's offering (mainly since it is free and has a 100k context window) and I've been surprised at just how well it reasons and understands what I'm asking. It responds like GPT 4 used to, and speaks to me with nuance I haven't seen since Bing released their original chatbot. I still find GPT 4 better if I want to fine-tune the model and make it adopt a persona-- Claude will almost always refuse.

Either way, I'm a bit confused with the way GPT 4 has been changing over time. It seems the team made significant changes to the model quality in favor of performance. Whether accuracy or performance is more important is up for debate, but something is clearly changing.

[0] https://archive.ph/2RQ8X

Re: How is ChatGPT's behavior changing over time?

#43

Post by xp84 »

dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

Re: How is ChatGPT's behavior changing over time?

#44

Post by visarga »

BoorishBears wrote: post #32
airgapstopgap wrote: post #25

Earlier quoted context omitted.

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is goi…

I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works. At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The web app they host isn't a simple API wrapper, it does summarization, has some sort of system p…

> OpenAI would not kneecap their commercial offering by randomly changing how it works.

Have you seen the 25 messages/3 hours limitation for GPT-4? Why do you think they did that? Of course they would make more money scaling up the volume, but how to do that when compute is so limited? Of course, by using some kind of approximation - quantised model or speculative sampling come to mind. It's hard to pinpoint model regressions, but scaling up volume is great, one more incentive to do it.

Re: How is ChatGPT's behavior changing over time?

#45

Post by BoorishBears »

visarga wrote: post #44
BoorishBears wrote: post #32

Earlier quoted context omitted.

I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works. At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The web app they host isn't a simple API wrapper, it does summarization, has some sort of system p…

> OpenAI would not kneecap their commercial offering by randomly changing how it works. Have you seen the 25 messages/3 hours limitation for GPT-4? Why do you think they did that? Of course they would make more money scaling up the volume, but how to do that when compute is so limited? Of course, by using some kind of approximation - quantised model or speculative sampling come to mind. It's hard to pinpoint model re…

You realize that's a limitation in the web application right?

The web app is a consumer app (B2C) the api is commercial (B2B). They tinker with the B2C app because it's already a lossy approximation of using the model between the summarization and system prompt.

They cannot mess with the commercial offering willy-nilly: People are building businesses predicated on it behaving a certain way. That's why there are dated version that you can pin to with the API. The web app changes whenever they feel like it.

Re: How is ChatGPT's behavior changing over time?

#46

Post by sheepscreek »

xp84 wrote: post #43
dudeinhawaii wrote: post #6

I think we should stop trying to quiz LLMs on mathematics, something for which they are explicitly not designed to do with their tokenized view of the world. Ask GPT-4 to use its Wolfram plugin and it returns the answers quickly and correctly. Second, I think the code generation bit of this paper is blown out of proportion. The code can't be immediately injected into a codebase due to a formatting change (triple quot…

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

A better example would be - calling your guitar lessons teacher for help on a statistics problem.

Re: How is ChatGPT's behavior changing over time?

#47

Post by ramesh31 »

flangola7 wrote: post #19
ramesh31 wrote: post #7

OpenAI is going to be left in the dust by (actual) open models. Llama 2 is already reaching GPT-3 levels, and can run inference on consumer hardware. Crazy how fast that flipped.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval. Furthermore, Llama remains well below GPT-3 on human rated tes…

>Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT.

Note that I never said "open source" just "open models". As in, I can now actually build things with GPT3 capability that run locally. I couldn't care less about the model code.

>Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.

I'm running llama-2-7b-chat on my 8GB M1 Mac right now with llama.cpp. Completions are instant, and essentially at GPT3 levels of accuracy.

The higher param models require up to 64GB RAM, but it's all CPU based.

Re: How is ChatGPT's behavior changing over time?

#48

Post by BoorishBears »

airgapstopgap wrote: post #41
BoorishBears wrote: post #32

Earlier quoted context omitted.

I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works. At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The web app they host isn't a simple API wrapper, it does summarization, has some sort of system p…

No, it makes sense to secure engagement with the most expensive implementation and then cut costs, this kind of stuff is pervasive in the industry. Besides, we have Brockman on record saying that they do "a lot of quantization"[1][2] so it's not paranoia to suspect other optimization schemes when there's a clear performance drop, which they have also denied a few times. 1. https://chat.openai.com/share/44a0c5b6-c629-…

Paranoia would be charitable: it's FUD.

If you intentionally smear the line between their web app which is chock full of optimizations to even let it function as it does (the web app's max conversation length exceeds the context window) and the API which is versioned and iterated on in the open... it's either a lack of understanding or FUD.

Re: How is ChatGPT's behavior changing over time?

#49

Post by pedrosorio »

flangola7 wrote: post #19
ramesh31 wrote: post #7

OpenAI is going to be left in the dust by (actual) open models. Llama 2 is already reaching GPT-3 levels, and can run inference on consumer hardware. Crazy how fast that flipped.

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval. Furthermore, Llama remains well below GPT-3 on human rated tes…

> They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval.

This is not what the Llama 2 license says [0]. There is a cap on the number of active users of products (any products, not just ones that may make use of Llama) by the company who plans to use Llama 2 as of Llama 2 release date.

Not a “future cap on number of users of your Llama2-based product”. Also, the cap is 700 million users.

> If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee's affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta

[0] https://github.com/facebookresearch/llama/blob/main/LICENSE

Re: How is ChatGPT's behavior changing over time?

#50

Post by colobas »

bugglebeetle wrote: post #16
sanxiyn wrote: post #11

Earlier quoted context omitted.

Or OpenAI can stop being stupid and adopt LLaMA-like tokenization, which special cases numbers and tokenize them into individual digits.

Isn’t the tokenization tied to the training of the model?

The model learns an embedding table, where (roughly) each row is used as the model’s internal representation of each token. The numbers in that table are learned. What isn’t learned is the map from token (i.e. combination of characters/byte-pairs) to row-index in the embedding table. That is given by tokenization

EDIT: removed redundant bit

Post reply on HN