Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

351–360 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#351

Post by sensanaty »

ryanwaggoner wrote: post #200
MyFirstSass wrote: post #196

An interesting thing i've been trying is to ask for a route from A to B in some city. Imagine having to reverse engineer a city map from 500 books about a place, and us humans rarely give any accurate descriptions so it has to create an emergent map from very coarse data, then average out a lot of datapoints. I tried for various scandinavian capitals and it seems to be able to, very crudely traverse various neighbour…

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

So it got the whole thing pretty much correct... Except for stuff like taking you down opposite directions?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#352

Post by jiggawatts »

thund wrote: post #229

It still takes the goat first, doesn't seem smarter than GPT4. > Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?

Err... isn't that the right answer?

Take the goat.

Return by yourself.

Take the lion.

Return with the goat.

Take the cabbage.

Return by yourself.

Take the goat.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#353

Post by Aeolun »

parineum wrote: post #324
Workaccount2 wrote: post #298

Earlier quoted context omitted.

Usually when I encounter sentiment like this it is because they only have used 3.5 (evidently not the case here) or that their prompting is terrible/misguided. When I show a lot of people GPT4 or Claude, some percentage of them jump right to "What year did Nixon get elected?" or "How tall is Barack Obama?" and then kind of shrug with a "Yeah, Siri could do that ten years ago" take. Beyond that you have people who pro…

> or that their prompting is terrible/misguided. This is the "You're not using it right" defense. It's an LLM, it's supposed to understand human language queries. I shouldn't have to speak LLM to speak to an LLM.

I mean, you need to speak German to talk to a German. It’s not really much different for LLM, just because the language they speak has a root in English doesn’t mean it actually is English.

And even if it was, there’s plenty of people completely unintelligible in English too…

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#354

Post by tedunangst »

jiggawatts wrote: post #352
thund wrote: post #229

It still takes the goat first, doesn't seem smarter than GPT4. > Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?

Err... isn't that the right answer? Take the goat. Return by yourself. Take the lion. Return with the goat. Take the cabbage. Return by yourself. Take the goat.

You just left the cabbage and lion alone together.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#355

Post by jiggawatts »

tedunangst wrote: post #354
jiggawatts wrote: post #352

Earlier quoted context omitted.

Err... isn't that the right answer? Take the goat. Return by yourself. Take the lion. Return with the goat. Take the cabbage. Return by yourself. Take the goat.

You just left the cabbage and lion alone together.

Ah... heh. I get it now.

That's a nice test of LLM capabilites and also, human inattention!

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#359

Post by versteegen »

jacobsimon wrote: post #278
versteegen wrote: post #273

Earlier quoted context omitted.

> How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Because Transformers are 'AI-complete'. Much is made of (decoder-only) transformers being next token predictors which misses the truth that large transformers can "think" before they speak: there are many layers in-between input and output. They can form a primitive high-level plan by a certain layer o…

I like your interpretation, but how would they refer back to a plan if it isn’t stored in the input/output? Wouldn’t this be lost/recalculated with each token?

The internal state at layer M of token N is available at every following token > N and layer > M via attention heads. Transformed by a matrix but a very direct lookup mechanism. The state after the final attention layer is not addressable in this way, but it immediately becomes the output token which is of course accessible.

Note also that sequential computations such as loops translate nicely to parallel ones, e.g. k layers can search the paths of length k in a graph, if each token represents one node. But since each token can only look backwards, unless you're searching a DAG you'd also have to feed in the graph multiple times so the nodes can see each other. Hmm... that might be a useful LLM prompting technique.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#360

Post by acchow »

losvedir wrote: post #341
smallnamespace wrote: post #225

Earlier quoted context omitted.

Have you ever heard the term NP-complete ?

Yeah, I mean, that's the joke. The comment I replied to, "a huge class of problems that's extremely difficult to solve but very easy to check", sounded to me like an assertion that P != NP, which everyone takes for granted but actually hasn't been proved. If, contrary to all expectations, P = NP, then that huge class of problems wouldn't exist, right? Since they'd be in P, they'd actually be easy to solve as well.

We could end up with a non-constructive proof of P=NP. That is, a proof that the classes are equal but no algorithm to convert a problem in one into the other (or construct a solution of one into a solution of the other).
Post reply on HN