Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

131–140 of 315 posts

Re: GPT-4 is getting worse over time, not better

#131

Post by smolder »

never_inline wrote: post #23
superkuh wrote: post #7

Earlier quoted context omitted.

Lets not conflate a model's smartness with a model's ability to be used reliably and predictably. They're very different things. Yes, fine-tuning, for instruction, or whatever you chose, will make the models easier to work with with less going off the rails. But they'll also take a big hit on perplexity and other more robust dataset completion tests. They're objectively stupider.

I would like to read on this. Do you have any sauce?

Can I recommend you use the actual word "source", and let the pointless "sauce" meme die its overdue death?

Re: GPT-4 is getting worse over time, not better

#132

Post by hospitalJail »

bufferoverflow wrote: post #108
hospitalJail wrote: post #88

Earlier quoted context omitted.

Are we talking GPT4 or chatgpt4? Its not disputed that chatGPT4 has degraded in quality as its been 'aligned'.

There's no such thing called chatgpt4. There's GPT-4 and ChatGPT.

Yes, these are different. Ask each how to do insert illegal activity, let me know if you find a difference.

Re: GPT-4 is getting worse over time, not better

#133

Post by JohnFen »

insanitybit wrote: post #100
JohnFen wrote: post #93

Earlier quoted context omitted.

If you're paying for it, it's a released product, not a beta. No matter how much a company wants to claim it's a "beta", it's simply not. The company just wants to be able to adhere to lower standards. OpenAI is hardly the only company that pulls these shenanigans, too.

No, beta has nothing to do with if it's paid or not. Lots of products are released, and you pay for them, with early access/ beta. No one lied to you. No one tricked you. It is in beta. You chose to pay for it.

I never said anyone lied to or tricked anyone. I'm saying that "beta" is a term referring to pre-release testing. If people are paying for it, it's been released and is therefore no longer a beta.

Calling it a "beta" at that point is just pure PR.

Re: GPT-4 is getting worse over time, not better

#134

Post by andai »

j45 wrote: post #83
brandall10 wrote: post #13

Earlier quoted context omitted.

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models. * https://arxiv.org/pdf/2101.03961.pdf

I wonder if additional layers of factories of factories approach can continue to improve it. I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?

I think the experts part implies that each "GPT-3" has been specialized in some way?

Re: GPT-4 is getting worse over time, not better

#135

Post by JohnFen »

somsak2 wrote: post #114
JohnFen wrote: post #93

Earlier quoted context omitted.

If you're paying for it, it's a released product, not a beta. No matter how much a company wants to claim it's a "beta", it's simply not. The company just wants to be able to adhere to lower standards. OpenAI is hardly the only company that pulls these shenanigans, too.

this is just not at all true -- companies have private betas that they monetize all the time. yes, the company provides lower guarantees on the product being supported. but the customer gets to use a product far before they normally would if they waited for general availability. that's the core of the idea behind a beta, not related to if it's paid or not

As I said, doing this is not rare and is certainly not limited to OpenAI. But it's misusing the term "beta" pretty significantly.

Re: GPT-4 is getting worse over time, not better

#136

Post by andai »

rogerkirkness wrote: post #36
andai wrote: post #6

Earlier quoted context omitted.

I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.

It's actually 16 GPT 3.5s in a trenchcoat where each one is slightly different, like the minions.

Can anyone elaborate on the specialization? Does it have to do with segmenting the dataset, or with training on different kinds of tasks?

Re: GPT-4 is getting worse over time, not better

#137

Post by reportgunner »

londons_explore wrote: post #129

Has anyone experimented with mixing outputs from LLm's on a per-token basis? Ie. easy tokens can be provided by a cheap-to-run model, and hard tokens are given by an expensive to run model? A model could be used to decide when it is worth running the expensive model, based on the inputs, output so far, and probability distribution of the output of the cheap model. For example, "Q: If I have 3 bananas and eat none, th…

So you run a model to avoid running a model? Math doesn't seem to add up.

Re: GPT-4 is getting worse over time, not better

#138

Post by tanaros »

gwd wrote: post #72
majkinetor wrote: post #63

Earlier quoted context omitted.

So the original paper from Stanford and Berkley is also linked to this AI influencer? I am really amazed by this kind of dismissal. Its totally irrelevant who posted the info and how framed it is, as long as you have access to the source.

This comment has an interesting take on it, haven't read the paper to verify the take: https://news.ycombinator.com/item?id=36781968 EDIT: FWIW I haven't noticed any such regression. I don't generally use it to find prime numbers, but I do use it for coding, and have been really impressed with what it's able to do. 8 This paper is being misinterpreted. The degradations reported are somewhat peculiar to the authors' t…

> Code generation: the change they report is that the newer GPT-4 adds non-code text to its output. They don't evaluate the correctness of the code. They merely check if the code is directly executable. So the newer model's attempt to be more helpful counted against it.

In the prompt they specifically request only the Python code, no other output. An “attempt to be helpful” that directly contradicts the user’s request seems like it should count against it.

Re: GPT-4 is getting worse over time, not better

#139

Post by dudeinjapan »

JCharante wrote: post #127
dudeinjapan wrote: post #41

Earlier quoted context omitted.

Life is good, Japan is a wonderful place to live. Was just in Kyoto last weekend for the Gion Matsuri, a big parade festival. If you're interested to come, check out my company TableCheck: https://careers.tablecheck.com/ We have a really talented and motivated team, great clients, and have a lot of fun.

a parade festival? A festival of parades?

A parade and a festival. It's a multi-day party, for much of the time they close the main streets for cars. At night there are food stalls and performances (traditional dance/drama/music/geishas) and during the day there is a parade of giant floats on wheels called "yama" and "hoko". Gion is the famous geisha district of Kyoto however much of the festival takes place more in the center of the city.

https://www.japan-guide.com/e/e3942.html

Re: GPT-4 is getting worse over time, not better

#140

Post by csmpltn »

Vvector wrote: post #70
jeremyjh wrote: post #56

Earlier quoted context omitted.

So you are just going to ignore the data (not anecdotes) presented in the SP?

ChatGPT isn't the right tool to use for checking if numbers are prime. It is tuned for conversations. I'd like to see a MathGPT or WolframGPT. The real question is if ChatGPT is worse on math and better elsewhere, or just worse overall. That is still unknown

> "ChatGPT isn't the right tool to use for checking if numbers are prime. It is tuned for conversations."

Two months ago they were telling me ChatGPT is coming for everyone - programmers, accountants, technical writers, lawyers, etc.

Now we're slowly back to "so here's the thing about LLMs"...

Post reply on HN