Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

241–250 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#241

Post by PUSH_AX »

raincole wrote: post #236
shepherdjerred wrote: post #114

Earlier quoted context omitted.

I simply asked it "what are you" and it responded that it was GPT-4 based. > I'm ChatGPT, a virtual assistant powered by artificial intelligence, specifically designed by OpenAI based on the GPT-4 model. I can help answer questions, provide explanations, generate text based on prompts, and assist with a wide range of topics. Whether you need help with information, learning something new, solving problems, or just loo…

It means its training data set has GPT4-generated text in it. Yes, that's it.

I think using chat gpt output to train other models is against the TOS and something they crack down on hard.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#242

Post by hackerlight »

kromem wrote: post #175

I certainly hope it's not GPT-5. This model struggles with reasoning tasks Opus does wonderfully with. A cheaper GPT-4 that's this good? Neat, I guess. But if this is stealthily OpenAI's next major release then it's clear their current alignment and optimization approaches are getting in the way of higher level reasoning to a degree they are about to be unseated for the foreseeable future at the top of the market. (T…

I'm seeing a big leap in performance for coding problems. Same feeling as GPT-3.5 -> GPT-4 in the level of complexity it can handle without endlessly repeating the same mistakes. Inference is slow. Would not be surprised if this was GPT-4.5 or GPT-5.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#243

Post by zarzavat »

somenameforme wrote: post #234
harryp_peng wrote: post #153

Earlier quoted context omitted.

You know at one point we wouldn't be able to benchmark them, due to the sheer complexity of the test required. I.e. if you are testing a model on maths, the problem will have to be extremely difficult to even consider a 'hustle' for the LLM; it would then take you a day to work out the solution yourself. See where it's getting at? When humans are no longer on the same spectrum as LLMs, that's probably the definition…

You know, people often complain about goal shifting in AI. We hit some target that was supposed to be AI (or even AGI), kind of go meh - and then change to a new goal. But the problem isn't goal shifting, the problem is that the goals were set to a level that had nothing whatsoever to do where we "really" want to go, precisely in order to make them achievable. So it's no surprise that when we hit these neutered goals…

There are different shades of AGI, but we don’t know if they will happen all at once or not.

For example, if an AI can replace the average white collar worker and therefore cause massive economic disruption, that would be a shade of AGI.

Another shade of AGI would be an AI that can effectively do research level mathematics and theoretical physics and is therefore capable of very high-level logical reasoning.

We don’t know if shades A and B will happen at the same time, or if there will be a delay between developing one and other.

AGI doesn’t imply simulation of a human mind or possessing all of human capabilities. It simply refers to an entity that possesses General Intelligence on par with a human. If it can prove the Riemann hypothesis but it can’t play the cello, it’s still an AGI.

One notable shade of AGI is the singularity: an AI that can create new AIs better than humans can create new AIs. When we reach shades A and B then a singularity AGI is probably quite close, if not before. Note that a singularity AGI doesn’t require simulation of the human mind either. It’s entirely possible that a cello-playing AI is chronologically after a self-improving AI.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#244

Post by imtringued »

harryp_peng wrote: post #153
throwup238 wrote: post #46

Earlier quoted context omitted.

I'm asking it about how to make turbine blades for a high bypass turbofan engine and it's giving very good answers, including math and some very esoteric material science knowledge. Way past the point where the knowledge can be easily checked for hallucinations without digging into literature including journal papers and using the math to build some simulations. I don't even have to prompt it much, I just keep saying…

You know at one point we wouldn't be able to benchmark them, due to the sheer complexity of the test required. I.e. if you are testing a model on maths, the problem will have to be extremely difficult to even consider a 'hustle' for the LLM; it would then take you a day to work out the solution yourself. See where it's getting at? When humans are no longer on the same spectrum as LLMs, that's probably the definition…

Me: 478700000000+99000000+580000+7000?

GPT4: 478799650000

Me: Well?

GPT4: Apologies for the confusion. The sum of 478700000000, 99000000, 580000 and 7000 is 478799058000.

I will be patient.

The answer is 478799587000 by the way. You just put the digits side by side.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#245

Post by imtringued »

sfkgtbor wrote: post #240
somenameforme wrote: post #226

Earlier quoted context omitted.

*Assuming you don't mean mathematically prove.* I can't test the bot right now, because it seems to have been hugged to death. But there's quite a lot of simple tests LLMs fail. Basically anything where the answer is both precise/discrete and unlikely to be directly in its training set. There's lots of examples in this [1] post, which oddly enough ended up flagged. In fact this guy [2] is offering $10k to anybody tha…

Multiple people found prompts to make LLM solve the problem, and the $10k has been awarded: https://twitter.com/VictorTaelin/status/1777049193489572064

The entire problem with LLMs is that you don't want to prompt them into solving specific problems. The reason why instruction finetuning is so popular is that it makes it easier to just write whatever you want. Text completion on the other hand requires you to conform to the style of the previously written text.

In a sense, LLMs need an affordance model so that it can estimate the difficulty of a task and plan a longer sequence of iterations automatically according to its perceived difficulty.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#246

Post by stavros »

graboid wrote: post #221
kromem wrote: post #175

I certainly hope it's not GPT-5. This model struggles with reasoning tasks Opus does wonderfully with. A cheaper GPT-4 that's this good? Neat, I guess. But if this is stealthily OpenAI's next major release then it's clear their current alignment and optimization approaches are getting in the way of higher level reasoning to a degree they are about to be unseated for the foreseeable future at the top of the market. (T…

To me, it seemed a bit better than GPT-4 at some coding task, or at least less inclined to just give the skeleton and leave out all the gnarly details, like GPT-4 likes to do these days. What frustrates me a bit is that I cannot really say if GPT-4, as it was in the very beginning when it happily executed even complicated and/or large requests for code, wasn't on the same level as this model actually, maybe not in te…

You can still use the original gpt-4 model via the API, no? Or have they shut that down? I haven't checked lately.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#247

Post by Tiberium »

dorkwood wrote: post #188
redox99 wrote: post #180

This is without a doubt the most advanced LLM out there, easily beating GPT4 and everything else. I asked it some programming questions for a niche programming language, and it's the first LLM that has been able to answer it correctly.

I've been told by several users here that GPT4 provides perfect programming answers for all languages, as long as you prompt it correctly. Maybe you need to work on your prompts?

That's just not true, GPT-4 is pretty bad for languages that are relatively niche, e.g. for Nim, Zig, Crystal as some random examples (and those do have some popularity at least).

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#248

Post by qeternity »

hehdhdjehehegwv wrote: post #165
harryp_peng wrote: post #150

Earlier quoted context omitted.

This really is the fastest growing technology of all time. Do you feel the curve? I remember Mixtral8x7b dominating for months; I expected data bricks to do the same! but it was washed out of existence in days, with 8x22b, llama3, gemini1.5... WOW.

Funny thing is I’m still in love with Mistral 7B as it absolutely shreds on a nice GPU. For simple tasks it’s totally sufficient.

Llama3 8B is for all intents and purposes just as fast.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#249

Post by Attummm »

stavros wrote: post #246
graboid wrote: post #221

Earlier quoted context omitted.

To me, it seemed a bit better than GPT-4 at some coding task, or at least less inclined to just give the skeleton and leave out all the gnarly details, like GPT-4 likes to do these days. What frustrates me a bit is that I cannot really say if GPT-4, as it was in the very beginning when it happily executed even complicated and/or large requests for code, wasn't on the same level as this model actually, maybe not in te…

You can still use the original gpt-4 model via the API, no? Or have they shut that down? I haven't checked lately.

If you had used GPT-4 from the beginning, the quality of the responses would have been incredibly high. It also took 3 minutes to receive a full response.

And prompt engineering tricks could get you wildly different outputs to a prompt.

Using the 3xx ChatGPT4 model from the API doesn't hold a candle to the responses from back then.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#250

Post by Lockal »

brainwad wrote: post #44

For translation, this thing is pretty amazing. "Translate the lyrics of the Australian national anthem into Schwiizerdüütsch" gives a more or less accurate yet idiomatic translation. ChatGPT 3.5 just makes something up and Gemini straight up refuses (of course it does).

ChatGPT was configured to not output lyrics and recipes[1]. Later on they removed this from the prompt (seemingly moved anywhere else), but it's highly unlikely that such censorship will ever be removed: any citation of songs is a direct step towards a DMCA complaint, and it would be extremely difficult for OpenAI to "delete" text scattered over the model.

Other simpler models have no such censorship and can easily output both songs and their translations without specifying authors and translators.

[1] https://medium.com/@adrian.punga_29809/chatgpt-system-prompt...

Post reply on HN