Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

51–60 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#51

Post by gnyman »

msoad wrote: post #40

Very impressive Prompt: > there are 3 black blocks on top of an block that we don't know the color of and beneath them there is a blue block. We remove all blocks and shuffle the blocks with one additional green block, then put them back on top of each other. the yellow block is on top of blue block. What color is the block we don't know the color of? only answer in one word. the color of block we didn't know the col…

I don't think this is a good test. If I prefix it with "a riddle" then GPT 4 got it right for me

"Yellow"

I think the "temperature" (randomness) of a LLM makes it so you'd need to run a lot of these to know if it's actually getting it right or just being lucky and selecting the right color randomly

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#52

Post by vincelt »

Still fails to solve this one:

Prompt:

> Complete the following Python program:

```

len, print = print, len

def print_len(x):

     "Print the length of x"
```

gpt2-chatbot:

```

len, print = print, len # Swap the functionalities of print and len

def print_len(x): "Print the length of x"

    len(x)  # This 'len' is actually 'print' because of the swap above
print_len("Hello, world!") # This will output: 13

```

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#53

Post by msoad »

msoad wrote: post #49

Prompt: my mother's sister has two brothers. each of her siblings have at least one child except for the sister that has 3 children. I have four siblings. How many grandchildren my grandfather has? Answer only with the result (the number) ChatGPT4: 13 Claude3 Opus: 10 (correct) GPT2-Chatbot: 15

By removing "Answer only with the result" all models can answer this correctly by doing "chain of thoughts"

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#56

Post by nicklecompte »

xdennis wrote: post #48

I'm surprised by people's impression. I tried it in my own language and much worse than GPT-4. Of the open source LLMs I've tried, all suck in non-English. I imagine it's difficult to make an LLM work in tens of languages on a consumer computer.

There's a core problem with LLMs: they learn sentences, not facts.

So an LLM may learn a ton of English-language sentences about cats, and much fewer Spanish sentences about gatos. And it even learns that cat-gato is a correct translation. But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish. So the LLM might be a genius if you ask it about cats in English, but in Spanish it might tell you "gatos tienen tres patas" simply because OpenAI didn't include enough Spanish biology books. These machines are just unfathomably dumb.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#58

Post by SeanAnderson »

msoad wrote: post #43
sebzim4500 wrote: post #32

Earlier quoted context omitted.

To be fair, this is pretty hard. Imagine you had to do to sit down and write this without being able to test it.

Styling I understand but to tick the clock when time actually has changed you need to use animation frames APIs and check for clock change more often than "every 1000 ms" because setTimeout will eventually drift even if you start exactly on first second time change. This is a test for depth of knowledge of a programmer I used to use in the past in interviews.

This is platform-specific, right? Chromium corrects for drift (https://source.chromium.org/chromium/chromium/src/+/main:thi...)

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#59

Post by realharo »

It failed on this math problem:

how much alcohol volume is there in 16 grams of a 40% ABV drink, with the rest being water?

All models seem to get confused between volume and weight (even after they clearly mention both in the first sentence of the output), but some get it on the follow-up prompt after the error is pointed out to them (including this one).

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#60

Post by famouswaffles »

nicklecompte wrote: post #56
xdennis wrote: post #48

I'm surprised by people's impression. I tried it in my own language and much worse than GPT-4. Of the open source LLMs I've tried, all suck in non-English. I imagine it's difficult to make an LLM work in tens of languages on a consumer computer.

There's a core problem with LLMs: they learn sentences, not facts. So an LLM may learn a ton of English-language sentences about cats, and much fewer Spanish sentences about gatos. And it even learns that cat-gato is a correct translation. But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish. S…

> but in Spanish it might tell you "gatos tienen tres patas"

Have you actually had a State of the art LLM do something like this?

Because this

>But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish

is just untrue. You can definitely query knowledge only learnt in one language in other languages.

Post reply on HN