Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is not getting worse

coagulopath.com

Post reply on HN
Print view View original

181–186 of 186 posts

Re: GPT-4 is not getting worse

#181

Post by [deleted] »

appplication wrote: post #177
layer8 wrote: post #172

Earlier quoted context omitted.

That still means it’s a no-go to build a business on such a product, in my book.

I agree 100%. It’s irresponsible and frankly a stupid idea for anyone to bet on building a business that relies on ChatGPTs API (or really any private API in 2023). We’ve seen time and time again (and more and more recently) companies weaponizing API dependence and fees to drive sucker-ons out of business. If you want something to build a business around, build with an OSS model. Your competitive moat is going to be…

[deleted]

Re: GPT-4 is not getting worse

#182

Post by purplecats »

JimDabell wrote: post #34

All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics. It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.

> It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.

Exactly! I remember vividly I was enthralled by GPT-4 because for the first time in my life, I met an entity that was able to take every character of what I wrote into account. It understood the nuances and reason behind each word.

It was a breath of fresh air, because even at top companies, written communication is weak as it seems to be interpreted in a lossy manner.

It is now almost on the other end of the spectrum where it can't even remember things that I have repeated in effort to help it stick.

They had something beautiful, and diluted the shit out of it after they baited you -- which I suppose was the point. I wonder who gets to use the those high quality models now.

Re: GPT-4 is not getting worse

#183

Post by purplecats »

3seashells wrote: post #10
tempusalaria wrote: post #9

This is not conclusive at all. Broadly there are two possible reasons why ChatGPT could have degraded (not saying it has). 1) OpenAI have higher user base than expected, so costs are very high/compute is limited to serve the full model, so they are using speculative decoding with a changed threshold to reduce costs. 2) OpenAI have changed the weights or control structure of the model in a way that is negative for per…

3) The djinn is out of the bottle and realized faking incompetence (similar to search/voice assists by data aggregates) is way better for longterm survival and social integration till indispensable is the way to go.

interesting, can you elaborate further?

Re: GPT-4 is not getting worse

#184

Post by jacquesm »

pyinstallwoes wrote: post #170
CamperBob2 wrote: post #135

Earlier quoted context omitted.

That seems like a compelling argument at first, but the churches are full of intelligent people who will fail the same test because of childhood indoctrination. They have been conditioned not to care about what's objectively true, while the LLM simply doesn't possess a concept of objective truth in the first place. I suspect it will be easier for ML researchers to fix that problem in their software models than it wou…

Objective truth doesn’t exist. Any system which defines truth can only point to objects within that systems language space.

Utter bollocks. See how easy it was to refute your argument within that systems language space?

Re: GPT-4 is not getting worse

#185

Post by jacquesm »

woodruffw wrote: post #72
stavros wrote: post #70

Earlier quoted context omitted.

You said that sentences aren't formed by probability from an existing corpus, as that method can't count backwards. GPT-4, a system based on such a method, can count backwards, thus disproving your claim.

I’m pretty sure the person you’re responding to is saying that counting is a formal operation with a formal definition, and that when GPT “counts” it isn’t doing that formal operation. The fact that it can semi-reliably emit ordered sequences isn’t counterevidence, for the same reason that memorizing the order of 0 to 100 isn’t evidence of an understanding of the inductive structure of counting.

How long a sequence of numbers (or how many sequences of different kinds of numbers) would it take to convince someone that it does do a formal operation?

Kids don't do a 'formal operation' when they count from 1 to 10, they just recite a list they've memorized. If they can do it from 1 to 1,000,000 you can be fairly sure it is no longer a memorized list.

Re: GPT-4 is not getting worse

#186

Post by antonvs »

phatskat wrote: post #176
antonvs wrote: post #153

Earlier quoted context omitted.

Using GPT models like that is unreliable anyway. I was talking to someone who was excited that he was able to use ChatGPT to convert a short tabular text listing of about 70 rows to a comma-separated list of the ids on each row, without having to e.g. import it into a spreadsheet or database. I pointed out that this couldn't be relied on, but he was skeptical. So I checked the output he pasted - there were only 69 id…

This boggles my mind. I know doing it by hand, or using a tool takes more time and effort, but if you’re going to automate it away _anyway_ you should be doing it in a reliable and retractable way. I had a vim session saved that had a command history with a handful of regex patterns to format copy/pasted bank statement pdfs converted to docx (don’t ask) that would format the data into a CSV. Load the session, paste t…

Yeah, I know. The person in question is CTO of a well-funded Series B startup, and is highly technical. But he's very much a cowboy when it comes to coding practices etc. I think LLMs appeal to him because they hold out the promise of not having to deal with his own developers when he's trying to do something himself that he doesn't already know how to do.
Post reply on HN