Experiencing decreased performance with ChatGPT-4 • Hacker News Board

Re: Experiencing decreased performance with ChatGPT-4

#181

Post by fleischhauf » Sat, Jul 08, 2023, 1:51 AM UTC

kccqzy wrote: post #141

Wowfunhappy wrote: post #108

Earlier quoted context omitted.

> Notice how you never hear anyone saying that GPT-4 is better since the launch. You'd expect to hear something like that as people gain more experience with prompting it. I'd expect the opposite. The first time you use ChatGPT (or GPT-4), you're in awe of what it can do, and more willing to overlook failures. As you use it, it becomes more mundane, and the instances where it messes up become more obvious.

I've noticed the same thing. People also like to complain the quality of Google Search has gone down for much of the same reason: if you first do a Google search that returned a good result and then repeat it, you are going to notice the absence. But if you first do a Google search that didn't return the thing you expect you might think such a thing just doesn't exist on the Internet. Ergo, quality decrease is simply…

I don't know man, it used to give me results where my query would appear in the text. this is not happening anymore. even if I use quotes around a term it will just be ignored. I feel like my searches are too specific, it might have gotten better for the average person, but worse for people looking for some more special information.

Re: Experiencing decreased performance with ChatGPT-4

#182

Post by flir » Sat, Jul 08, 2023, 2:21 AM UTC

gregw134 wrote: post #87

cheeko1234 wrote: post #38

I use the following test to ensure I'm on GPT4 and not 3.5. (I noticed that it did fail at this test temporarily and then got it. Not sure why. Maybe it reverts back to 3.5 when under load?) I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it? GPT4: You actually don't need to do anything because one of your jugs is already a 6-liter jug. If you fill it up to the top, you'll have exactl…

It figures it out once you let it reflect on its answer: Consider the following situation: You have a 12 liter jug and a 6 liter jug, and you want to measure out exactly 6 liters of water. First, generate an initial solution for this problem. Then, think about the solution you've generated, considering if there might be a simpler or more straightforward way to achieve the goal. If there is, please provide the more ac…

Interesting approach. 3.5 gave me this as its second answer:

1. Fill the 6 liter jug to its maximum capacity. 2. Pour the contents of the 6 liter jug into the 12 liter jug, which will leave 6 liters in the 6 liter jug. 3. You have now measured out exactly 6 liters of water using the 6 liter jug.

Re: Experiencing decreased performance with ChatGPT-4

#183

Post by fwungy » Sat, Jul 08, 2023, 2:36 AM UTC

afro88 wrote: post #169

fwungy wrote: post #165

Earlier quoted context omitted.

Reduce the parameter count. Increase the acceptable error. The more parameters on a curve fit the better the fit will be, but the compute power increases too.

Is that possible? I thought parameter counts were fixed in the model.

There are always ways to trade precision for speed in computer statistics models.

Re: Experiencing decreased performance with ChatGPT-4

#184

Post by SkyPuncher » Sat, Jul 08, 2023, 3:10 AM UTC

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

Definitely not. We’re pinning to an old version right now because the new one is worse across the board.

Re: Experiencing decreased performance with ChatGPT-4

#185

Post by SkyPuncher » Sat, Jul 08, 2023, 3:14 AM UTC

dannyobrien wrote: post #4

Without concrete examples, I do wonder if much of this is perceptual. I love using ChatGPT, but once the amazement that it works as well as it does has worn off, one ends up spotting the flaws more than before. I feel that advocates and critics of ChatGPT are both right, to a degree, but looking at the models responses from slightly different angles: it wouldn't be surprising if users' angles shift over time.

imo, You won’t see concrete examples because it’s proprietary information at most companies.

My experience is the new versions perform much worse on the same prompts.

Re: Experiencing decreased performance with ChatGPT-4

#186

Post by SkyPuncher » Sat, Jul 08, 2023, 3:17 AM UTC

facu17y wrote: post #24

azhenley wrote: post #5

Wonder if this is the same as the discussion from 35 days ago on "OpenAI Employee: GPT-4 has been static since March" https://news.ycombinator.com/item?id=36155267

the base model may have been but not necessarily the RLHF fine-tuned layers they might have added or the shortcut they're taking during inference due to such fine tuning (or for perf optimization unrelated to fine tuning.)

In “legacy” software development, this would be the equivalent to saying “our database schema is the same” while completely ignoring all of the business logic and UI that gets placed in front of that database.

Re: Experiencing decreased performance with ChatGPT-4

#187

Post by imranq » Sat, Jul 08, 2023, 3:52 AM UTC

stavros wrote: post #178

psychphysic wrote: post #129

Earlier quoted context omitted.

It'd be insane if OpenAI wasn't changing GPT-4. That kind of flat footedness would cost them their entire first mover advantage.

In that case, I'd hope they're changing it for the better, rather than making it more of an anodyne prude.

Maybe this increased prudishness is coming from the kinds of queries they are seeing come in...

Re: Experiencing decreased performance with ChatGPT-4

#188

Post by reissbaker » Sat, Jul 08, 2023, 4:50 AM UTC

TeMPOraL wrote: post #172

rrrrrrrrrrrryan wrote: post #166

Earlier quoted context omitted.

Can I ask how you use this function calling in your workflow? Any examples?

Seconded. In particular, how does function calling help restore performance in general prompts like: "Here's roughly what I'm trying to achieve: Could you please write me such function/script/whatever?". Maybe I lack the imagination, but what function should I give to the LLM? "insert(text: string)"?

For sure! Here's one example where I have it generate SQL (scroll through the thread, the function API is the second tweet): https://twitter.com/reissbaker/status/1671361372092010497

For generating arbitrary code, I imagine you could do the same thing but swap `query_db` with the name `exec_javascript` or something similar based on your preferred language.

Re: Experiencing decreased performance with ChatGPT-4

#189

Post by jiggawatts » Sat, Jul 08, 2023, 5:07 AM UTC

llamaLord wrote: post #170

TeMPOraL wrote: post #167

Earlier quoted context omitted.

Indeed. I have a sinking feeling they realized (or were otherwise convinced) those models are too disruptive to existing businesses and whole market segments, in particular (but not limited to) when it comes to writing code. Or at least that's where it's most obvious to me just how many different classes of companies could grow and capture value[0] that GPT-4 has been providing, pay-as-you-go, for a dozen cents per u…

I've smelt the sweet scent of anticompetitive back-room dealing around OpenAI ever since they and Microsoft started forcing people to apply for access to the APIs and including telling them what the use case they were going to use it was. It just seemed obvious that if anyone suggested a use case that was actually really high value MS would just take the idea, run with it for a month or two to see if it has legs, and…

Meanwhile Sam Altman was on a worldwide press tour repeatedly saying that their mission is to “democratise” AI. They’re actually doing the exact opposite: gatekeeping, building moats, and seeking legislation to entrench a monopoly position.

Re: Experiencing decreased performance with ChatGPT-4

#190

Post by gregw134 » Sat, Jul 08, 2023, 5:13 AM UTC

flir wrote: post #182

gregw134 wrote: post #87

Earlier quoted context omitted.

It figures it out once you let it reflect on its answer: Consider the following situation: You have a 12 liter jug and a 6 liter jug, and you want to measure out exactly 6 liters of water. First, generate an initial solution for this problem. Then, think about the solution you've generated, considering if there might be a simpler or more straightforward way to achieve the goal. If there is, please provide the more ac…

Interesting approach. 3.5 gave me this as its second answer: 1. Fill the 6 liter jug to its maximum capacity. 2. Pour the contents of the 6 liter jug into the 12 liter jug, which will leave 6 liters in the 6 liter jug. 3. You have now measured out exactly 6 liters of water using the 6 liter jug.

Yeah 3.5 still can't figure it out, 4 does though. Hilariously 3.5 gave me this:

Fill the 6-liter jug completely with water. Pour the water from the 6-liter jug into the 12-liter jug. Fill the 6-liter jug again with water. Pour water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. This leaves 4 liters of water in the 6-liter jug. Empty the 12-liter jug. Pour the 4 liters of water from the 6-liter jug into the empty 12-liter jug. Fill the 6-liter jug with water again. Pour water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. This leaves exactly 6 liters of water in the 12-liter jug.