Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is ChatGPT's behavior changing over time?

arxiv.org

Post reply on HN
Print view View original

121–130 of 187 posts

Re: How is ChatGPT's behavior changing over time?

#121

Post by ineedasername »

0x000xca0xfe wrote: post #102
lumost wrote: post #55

Earlier quoted context omitted.

I used GPT-4 to generate a non-cryptographic random 64 character string. It was faster to ask GPT-4 for the string than ask GPT-4 for the instructions to generate the string from my terminal. GPT-4 was faster than google.

Obligatory bash solution: $ random_bytes() { xxd -plain -c 0 -l "$1" /dev/urandom; } $ random_bytes 32 e6a4a7bbea69a0164cbb66c89f8f528af93c6d2459fd28d2640e2952c031b618

>/dev/urandom

Hmm, I’ve been using /usr/games/fortune

Is this not a best practice?

Re: How is ChatGPT's behavior changing over time?

#122

Post by ineedasername »

muzani wrote: post #96
deelly wrote: post #72

Earlier quoted context omitted.

No, we definitely should continue to quiz LLMs on mathematics and absolutely any other topics. Otherwise how do we know and understand limitation of the system?

We should also test its capabilities on cooking steak, flying rockets, and making love. Only then will we know if AI can be superior to humans on all things.

Well, if there was an API hooked up to a webcam & 6-dof arm that would be an interesting task. (The steak)

Re: How is ChatGPT's behavior changing over time?

#123

Post by ineedasername »

staunton wrote: post #66
ehnto wrote: post #28

Earlier quoted context omitted.

I think knowing if the code can be used verbatim is actually the more important part practically speaking. That is the actually useful part. Quality is important to humans, because humans have to read it, but correctness is what people using ChatGPT for code actually need. So long as the quality and performance is good enough, then it will be useful. Performance is such a nuanced topic that you need very context awar…

> never going to have that kind of awareness Be careful with that goalpost, it might make sudden movements.

Yes, when ChatGPT went public using 3 I was like, “Hah, yeah, that can’t come close to writing the narrative portions in my work.” Then 4 came along and it was more like, “ooh, a basic first draft in 15 seconds? That I can work with.”

Now, at least for some projects, I can give it a few rapid bullet points in incomplete sentences and have that first draft in seconds, after which I just need to tweak, add in tables and more detailed stats & results, etc. quite useful.

Re: How is ChatGPT's behavior changing over time?

#124

Post by AndroidKitKat »

amelius wrote: post #118

How is Copilot's behavior changing over time? And has anyone successfully used Copilot to do simple but tedious things like generate library bindings for various languages? I suspect that Copilot would be good at this kind of thing because it is a relatively simple task and there is a lot of training-code available.

I use Copilot primarily as a tab-complete bot, essentially only accepting its input if that's what I was going to type. It is correct mayb ~95% of the time, but depending on the programming language it will try to suggest a large block of code that I don't really trust (in my usage, it loves to spit out Python but in my Java codebase it sticks to boilerplate or finishing my line for me).

Re: How is ChatGPT's behavior changing over time?

#125

Post by brushfoot »

remus wrote: post #119
xp84 wrote: post #43

Earlier quoted context omitted.

Seriously. GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. Wtf is the point. We have computer things to do every math problem. It’s a waste of energy to use LLMs for it in my opinion.

> GPT doing math is like using a 737 to drive around on the ground, or if you had the phone number of a prominent astrophysicist and you call him to do long division for you. I don't think this is a great analogy. if your 737 couldn't drive on the ground and your astrophysicist couldn't answer basic maths questions I wouldn't want to fly in that plane or put much faith in the astrophysicists answers to more complex q…

> asking questions where it is easy to judge the factual accuracy of the responses seems a pretty reasonable test to be running.

It isn't reasonable if that isn't what the system was designed to do.

It would be a poor test of my general practitioner's competence to ask him calculus questions and conclude he doesn't know what he's talking about because he can't answer them.

Re: How is ChatGPT's behavior changing over time?

#126

Post by taneq »

oefnak wrote: post #74
lumost wrote: post #55

Earlier quoted context omitted.

I used GPT-4 to generate a non-cryptographic random 64 character string. It was faster to ask GPT-4 for the string than ask GPT-4 for the instructions to generate the string from my terminal. GPT-4 was faster than google.

That definitely won't be random.

It's probably 4, that's pretty random.

Re: How is ChatGPT's behavior changing over time?

#129

Post by ineedasername »

metanonsense wrote: post #62

With those GPT4 architecture infos leaked a few weeks ago, my personal theory is that they use mixture-of-experts routing for scaling the system. For instance, they could say: from 0 to 50% system load use a mixture of top-6 models for inference, from 60 to 80% use top-4 models, above 80% use top-2 models. This could make ChatGPT look dumber or smarter depending on the time of the day and system usage. Naturally, the…

I’m guessing that some tasks relegated to a single one of those experts are tasks that previously benefited from those experts’ capabilities being integrated in a single model.

Sort of like having a problem and then asking both a biologist & a chemist for their assessment. But if asked a single biochemist they could more readily provide an answer that synthesized both more comprehensively, even if you got both the biologist and the chemist in the same room with each other.

Re: How is ChatGPT's behavior changing over time?

#130

Post by jdshaffer »

JanSt wrote: post #84
barrkel wrote: post #53

Irritatingly, OpenAI reps deny any change in model capabilities over time. It's more likely that as the models are optimized for cost and performance, their in-house evaluation metrics don't cover everything customers are interested in. Meanwhile, the probabilistic nature of LLM output means there is plausible deniability.

I'm now sometimes using GPT3.5 because it produces better results than GPT4. GPT4 was amazing when it first came out for me. Either it has deteriorated or I was lucky in the beginning and am now lucky with GPT3.5 from time to time.

I have the same experience. GPT4 seemed surprisingly better at first, but now I can't tell a big difference between the two. Admittedly I'm bantering back and forth about history, literature, and languages...
Post reply on HN