Live data from Hacker News

GTP Blind Voting: GPT-5 vs. 4o

gptblindvoting.vercel.app

31–40 of 54 posts

Re: GTP Blind Voting: GPT-5 vs. 4o

#31

I don't like this test, because the very first question I was present with, had both answers looked equivalently good. Actually they were almost the same, just with different phrasing. So my choice would be absolute random. It means, that end score will be polluted by random. They should have added things like "both answers good" and "both answers bad".

I have a lot of experience with pairwise testing so I can explain this.

The reason there isn't an "equal" option is because it's impossible to calibrate. How close do the two options have to be before the average person considers them "equal"? You can't really say.

The other problem is when two things are very close, if you provide an "equal" option you lose the very slight preference information. One test I did was getting people to say which of two greyscale colours is lighter. With enough comparisons you can easily get the correct ordering even down to 8 bits (i.e. people can distinguish 0x808080 and 0x818181), but they really look the same if you just look at a pair of them (unless they are directly adjacent, which wasn't the case in my test).

The "polluted by randomness" issue isn't a problem with sufficient comparisons because you show the things in a random order so it eventually gets cancelled out. Imagine throwing a very slightly weighted coin; it's mostly random but with enough throws you can see the bias.

...

On the other hand, 16 comparisons isn't very many at all, and also I did implement an ad-hoc "they look the same" option for my tests and it did actually perform significantly better, even if it isn't quite as mathematically rigorous.

Also player skill ranking systems like Elo or TrueSkill have to deal with draws (in games that allow them), and really most of these ranking algorithms are totally ad-hoc anyway (e.g. why does Bradley-Terry use a sigmoid model?), so it's not really a big deal to add more ad-hocness into your model.

Re: GTP Blind Voting: GPT-5 vs. 4o

#32
post #27

Does anyone ever get answers this short? What's the system prompt here? That may bias things a little. Also it's GPT not GTP

> Does anyone ever get answers this short?

If you converse with it, yes

> What's the system prompt here?

You don't need a specific one. If you talk to it, it turns into that.

Re: GTP Blind Voting: GPT-5 vs. 4o

#33

Earlier quoted context omitted.

Before GPT-5, I've used almost exclusively o3 and sometimes o3-pro. Now I'm using GPT 5 Thinking and sometimes GPT 5 Pro. So I think that I have some control over quality. At least it thinks for few dozens of seconds every time.

> GPT 5 Pro Have you noticed either of these things: (1) If your first prompt is too long (50k+ tokens) but just below the limit (like 80k tokens or whatever), it cannot see the right-side of your prompt. (2) By the second prompt, if the first prompt was long-ish, the context from the first prompt is no longer visible to the model.

definitely 1!

it seems to truncate your prompt even under the "maximum message length" and yeah around 55k is where it starts to happen.

extremely annoying. o1 pro worked up until 115k or so. both o3 and gpt5 have the issue. (it happens on all models for me not just the pro variations)

with the new 400k context length in api i would expect atleast 128k message lengths and maybe 200k context in chat.

Re: GTP Blind Voting: GPT-5 vs. 4o

#34

Earlier quoted context omitted.

Before GPT-5, I've used almost exclusively o3 and sometimes o3-pro. Now I'm using GPT 5 Thinking and sometimes GPT 5 Pro. So I think that I have some control over quality. At least it thinks for few dozens of seconds every time.

> GPT 5 Pro Have you noticed either of these things: (1) If your first prompt is too long (50k+ tokens) but just below the limit (like 80k tokens or whatever), it cannot see the right-side of your prompt. (2) By the second prompt, if the first prompt was long-ish, the context from the first prompt is no longer visible to the model.

Sorry, can't really answer to it, as I very rarely using any long context. I prefer to either edit previous question or just start new chat to keep context short. And even when I need to dump code, I prefer to choose relevant snippets. I'm aware that LLM quality degrades with long contexts, so I've trained myself to avoid it.

Re: GTP Blind Voting: GPT-5 vs. 4o

#36
post #33

Earlier quoted context omitted.

> GPT 5 Pro Have you noticed either of these things: (1) If your first prompt is too long (50k+ tokens) but just below the limit (like 80k tokens or whatever), it cannot see the right-side of your prompt. (2) By the second prompt, if the first prompt was long-ish, the context from the first prompt is no longer visible to the model.

definitely 1! it seems to truncate your prompt even under the "maximum message length" and yeah around 55k is where it starts to happen. extremely annoying. o1 pro worked up until 115k or so. both o3 and gpt5 have the issue. (it happens on all models for me not just the pro variations) with the new 400k context length in api i would expect atleast 128k message lengths and maybe 200k context in chat.

Do you have a workaround?

I'm putting the highest quality context into the 50k tokens, and attaching the rest for RAG. But maybe there is a better way.

Post reply on HN