Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

271–280 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#271

Post by zaptrem »

atemerev wrote: post #220
swyx wrote: post #193

Earlier quoted context omitted.

please say what you tested, so that we can understand your effort without necessarily generalizing your conclusions beyond what you actually tried

Asking for the information about non-public individuals, including myself. RAG-assisted GPT-4 easily provides such information. GPT2 output is consistent with a good model without RAG (it tries to speculate, but says it doesn’t have such information ultimately). I liked that it doesn’t try to hallucinate things.

You could also test this more easily with current events.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#272

Post by nrvn »

MyFirstSass wrote: post #196

An interesting thing i've been trying is to ask for a route from A to B in some city. Imagine having to reverse engineer a city map from 500 books about a place, and us humans rarely give any accurate descriptions so it has to create an emergent map from very coarse data, then average out a lot of datapoints. I tried for various scandinavian capitals and it seems to be able to, very crudely traverse various neighbour…

test it with small cities and preferably outside of US. Hallucinating with street names and directions not worse than other models. Had a good laugh.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#273

Post by versteegen »

MyFirstSass wrote: post #203
ryanwaggoner wrote: post #200

Earlier quoted context omitted.

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

That's wild. I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things? How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

> How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo?

Because Transformers are 'AI-complete'. Much is made of (decoder-only) transformers being next token predictors which misses the truth that large transformers can "think" before they speak: there are many layers in-between input and output. They can form a primitive high-level plan by a certain layer of a certain token such as the last input token of the prompt, e.g. go from A to B via approximate midpoint C, and then refer back to that on every following token, while expanding upon it with details (A to C via D): their working memory grows with the number of input+output tokens, and with each additional layer they can elaborate details of an earlier representation such as a 'plan'.

However the number of sequential steps of any internal computation (not 'saved' as an output token) is limited by the number of layers. This limit can be worked around by using chain-of-thought, which is why I call them AI-complete.

I write this all hypothetically, not based on mechanistic interpretability experiments.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#274

Post by jskherman »

93po wrote: post #94
hombre_fatal wrote: post #37

Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…

i just tested this too, really cool. i own a yaris have used an online forum for yaris cars for the past decade and had a vague memory of a user who deleted some of the most helpful guides. i asked about it and sure enough it knew exactly who i meant: who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?: One notable user from the Yaris forums who deleted many of their helpful guid…

This answer looks eerily similar to the llama-3-sonar-large-32k-online model by Perplexity on labs.perplexity.ai

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#275

Post by dorkwood »

Tiberium wrote: post #247
dorkwood wrote: post #188

Earlier quoted context omitted.

I've been told by several users here that GPT4 provides perfect programming answers for all languages, as long as you prompt it correctly. Maybe you need to work on your prompts?

That's just not true, GPT-4 is pretty bad for languages that are relatively niche, e.g. for Nim, Zig, Crystal as some random examples (and those do have some popularity at least).

Yes, I've experienced the same thing with Vex. But I've been told it's because my prompts aren't good enough. I need to keep working on them.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#276

Post by nickfromseattle »

thorum wrote: post #7

It’s an impressive model, but why would OpenAI need to do that?

Altman said in the latest Lex Friedman podcast that OAI has consistently received feedback their releases "shock the world", and that they'd like to fix that.

I think releasing to this 3rd party so the internet can start chattering about it and discovering new functionality several months before an official release aligns with that goal of drip-feeding society incremental updates instead of big new releases.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#277

Post by joshstrange »

7734128 wrote: post #267
qingcharles wrote: post #237

Earlier quoted context omitted.

GPT regularly gives me A/B responses and asks me which one is better.

I often get this with code, but when I try to select the text to copy the code to test it that is immediately treated as a vote.

This frustrates me, just because I'm copying code doesn't mean it's the better choice. I actuallly want to try both and then vote but instead I accidentially vote each time.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#278

Post by jacobsimon »

versteegen wrote: post #273
MyFirstSass wrote: post #203

Earlier quoted context omitted.

That's wild. I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things? How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

> How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Because Transformers are 'AI-complete'. Much is made of (decoder-only) transformers being next token predictors which misses the truth that large transformers can "think" before they speak: there are many layers in-between input and output. They can form a primitive high-level plan by a certain layer o…

I like your interpretation, but how would they refer back to a plan if it isn’t stored in the input/output? Wouldn’t this be lost/recalculated with each token?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#279

Post by Kuinox »

swyx wrote: post #193
atemerev wrote: post #87

Earlier quoted context omitted.

No, I tested it, this is a pure model without RAG (at least no internet search)

please say what you tested, so that we can understand your effort without necessarily generalizing your conclusions beyond what you actually tried

You can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#280

Post by SOLAR_FIELDS »

Attummm wrote: post #249
stavros wrote: post #246

Earlier quoted context omitted.

You can still use the original gpt-4 model via the API, no? Or have they shut that down? I haven't checked lately.

If you had used GPT-4 from the beginning, the quality of the responses would have been incredibly high. It also took 3 minutes to receive a full response. And prompt engineering tricks could get you wildly different outputs to a prompt. Using the 3xx ChatGPT4 model from the API doesn't hold a candle to the responses from back then.

I hear this somewhat often from people (less so nowadays) but before and after prompt examples are never provided. Do you have some example responses saved from the olden days, by chance? It would be quite easy to demonstrate your point if you did.
Post reply on HN