Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

201–210 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#201

Post by concurrentsquar »

MyFirstSass wrote: post #178
hombre_fatal wrote: post #37

Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…

Weird, it doesn't seem to have any info on reddit users or their writings. I tried asking about a bunch, also just about general "legendary users" from various subreddits and it seemingly just hallucinated.

Reddit may have told OpenAI to pay (probably a lot of) money to legally use Reddit content for training, which is something Reddit is doing with other AI labs (https://www.cbsnews.com/news/google-reddit-60-million-deal-a... ); but GPTBot is not banned under the Reddit robots.txt (https://www.reddit.com/robots.txt).

This is assuming that lmsys' GPT-2 is retained GPT-4t or a new GPT-4.5/5 though; I doubt that (one obvious issue: why name it GPT-2 and not something like 'openhermes-llama-3-70b-oai-tokenizer-test' (for maximum discreetness) or even 'test language model (please ignore)' (which would work well for marketing); GPT-2 (as a name) doesn't really work well for marketing or privacy (at least compared to the other options)).

Lmsys has tested models with weird names for testing before: https://news.ycombinator.com/item?id=40205935

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#202

Post by bigger_cheese »

realharo wrote: post #59

It failed on this math problem: how much alcohol volume is there in 16 grams of a 40% ABV drink, with the rest being water? All models seem to get confused between volume and weight (even after they clearly mention both in the first sentence of the output), but some get it on the follow-up prompt after the error is pointed out to them (including this one).

Math and Science questions in general chat bots seem to struggle with.

I've asked the chat bot that Microsoft Bing uses (not sure what brand or version it is) Chemistry questions - In the format "Is the reaction between X and Y Exothermic under Z Conditions."

It will give a confident answer including things like chemical equations and citations that is blatantly wrong.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#203

Post by MyFirstSass »

ryanwaggoner wrote: post #200
MyFirstSass wrote: post #196

An interesting thing i've been trying is to ask for a route from A to B in some city. Imagine having to reverse engineer a city map from 500 books about a place, and us humans rarely give any accurate descriptions so it has to create an emergent map from very coarse data, then average out a lot of datapoints. I tried for various scandinavian capitals and it seems to be able to, very crudely traverse various neighbour…

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

That's wild.

I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things?

How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#204

Post by concurrentsquar »

skissane wrote: post #140

gpt2-chatbot is not the only "mystery model" on LMSYS. Another is "deluxe-chat". When asked about it in October last year, LMSYS replied [0] "It is an experiment we are running currently. More details will be revealed later" One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors [0]…

> One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors

Beam search or weird attention/non-transformer architecture?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#205

Post by losvedir »

coffeemug wrote: post #187
harryp_peng wrote: post #153

Earlier quoted context omitted.

You know at one point we wouldn't be able to benchmark them, due to the sheer complexity of the test required. I.e. if you are testing a model on maths, the problem will have to be extremely difficult to even consider a 'hustle' for the LLM; it would then take you a day to work out the solution yourself. See where it's getting at? When humans are no longer on the same spectrum as LLMs, that's probably the definition…

There is a huge class of problems that's extremely difficult to solve but very easy to check.

Prove it...

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#206

Post by the_shivers »

Alas, it still fails at my favorite music theory question. I've yet to see a chatbot get this right, even though it isn't a trick question at all.

I ask:

"Consider a tenor ukulele in standard tuning. If we tune it down by a half step, how might we then finger a Gmaj7?"

It initially reasons correctly that this must be G-B-D-F# and it understands what the resulting strings will be tuned to if we drop them a half step. It just can't work out a valid fingering, and ends up getting confused and suggesting a nonsense chord.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#207

Post by concurrentsquar »

numlocked wrote: post #30
xennygrimmato wrote: post #11

What's interesting it seems to also edit text in between for me, not just add text at the end.

I'm seeing this as well. I don't quite understand how it's doing that in the context of LLMs to date being a "next token predictor". It is writing code, then adding more code in the middle.

Is it something similar to beam search (https://huggingface.co/blog/how-to-generate#beam-search) or completely different (probably is not beam search if it's changing code in the middle of a block)?

(I can't try right now because of API rate limits)

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#208

Post by ec109685 »

MyFirstSass wrote: post #203
ryanwaggoner wrote: post #200

Earlier quoted context omitted.

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

That's wild. I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things? How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

There must be some point a to point c training data that it had to learn how to complete via tons of point a to b data, which it then learned to generalize?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#209

Post by hnfong »

MyFirstSass wrote: post #203
ryanwaggoner wrote: post #200

Earlier quoted context omitted.

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

That's wild. I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things? How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

A couple hundred billion floating point numbers is enough to store quite a few things.

Also, algorithms you learn in CS are for scaling problems to arbitrary sizes, but you don't strictly need those algos to handle problems of a small size. In a sense, you could say the "next token predictor" can simulate some very crude algorithms, eg. at every token, greedily find the next location by looking at the current location, and output the neighboring location that's to the direction of the destination.

The next token predictor is a built in for loop, and if you have a bunch of stored data on where the current location is roughly, its neighboring locations, and the relative direction of the destination... then you got a crude algo that kinda works.

PS: but yeah despite the above, I still think the emergence is "magic".

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#210

Post by flawsofar »

MyFirstSass wrote: post #203
ryanwaggoner wrote: post #200

Earlier quoted context omitted.

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

That's wild. I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things? How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

maybe: embedding vector contains coordinates, vectors are really good at coordinates
Post reply on HN