Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

61–70 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#61

Post by staticman2 »

msoad wrote: post #19

Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers. ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output None is correct.…

When I first read this prompt I thought "Okay but what time do you want the clock to be set to?"

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#62

Post by msoad »

SeanAnderson wrote: post #58
msoad wrote: post #43

Earlier quoted context omitted.

Styling I understand but to tick the clock when time actually has changed you need to use animation frames APIs and check for clock change more often than "every 1000 ms" because setTimeout will eventually drift even if you start exactly on first second time change. This is a test for depth of knowledge of a programmer I used to use in the past in interviews.

This is platform-specific, right? Chromium corrects for drift ( https://source.chromium.org/chromium/chromium/src/+/main:thi... )

I wish that was true but you can easily see it drifting in Chrome

        let lastms 
        function tick() {
            if (lastms === undefined)
                lastms = new Date().getMilliseconds()
            else if (lastms !== new Date().getMilliseconds()) 
                throw new Error('Drifted')
        }
        setInterval(tick, 1000)

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#64

Post by acchow »

msoad wrote: post #19

Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers. ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output None is correct.…

Have you tried this with "chain of thought"?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#67

Post by [deleted] »

msoad wrote: post #62
SeanAnderson wrote: post #58

Earlier quoted context omitted.

This is platform-specific, right? Chromium corrects for drift ( https://source.chromium.org/chromium/chromium/src/+/main:thi... )

I wish that was true but you can easily see it drifting in Chrome let lastms function tick() { if (lastms === undefined) lastms = new Date().getMilliseconds() else if (lastms !== new Date().getMilliseconds()) throw new Error('Drifted') } setInterval(tick, 1000)

[deleted]

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#68

Post by gs17 »

rany_ wrote: post #21

If you ask it about the model name/cutoff date it claims to be "ChatGPT, based on GPT-4" and that the cutoff date is "Nov 2023." It claims that consistently so I think it might be accurate.

That could be down to the system prompt, FastChat has a default prompt for GPT with that information in it.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#69

Post by nicklecompte »

famouswaffles wrote: post #60
nicklecompte wrote: post #56

Earlier quoted context omitted.

There's a core problem with LLMs: they learn sentences, not facts. So an LLM may learn a ton of English-language sentences about cats, and much fewer Spanish sentences about gatos. And it even learns that cat-gato is a correct translation. But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish. S…

> but in Spanish it might tell you "gatos tienen tres patas" Have you actually had a State of the art LLM do something like this? Because this >But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish is just untrue. You can definitely query knowledge only learnt in one language in other languages.

I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili.

> You can definitely query knowledge only learnt in one language in other languages.

Do you have a source on that? I believe this is simply not true, unless maybe the pretraining data has enough context-specific "bridge translations." And I am not sure how on earth you would verify that any major LLM only learnt something in one language. What if the pretraining data includes machine translations?

Frustratingly, just a few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it. So I can't really criticize you if you don't have a source :) The argument was essentially what I said above: since LLMs associate tokens by related tokens, not ideas by related ideas, the emergent conceptual relations formed around the token "cat" do not have any means of transferring to conceptual relations around the token "gato."

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#70

Post by realharo »

msoad wrote: post #62
SeanAnderson wrote: post #58

Earlier quoted context omitted.

This is platform-specific, right? Chromium corrects for drift ( https://source.chromium.org/chromium/chromium/src/+/main:thi... )

I wish that was true but you can easily see it drifting in Chrome let lastms function tick() { if (lastms === undefined) lastms = new Date().getMilliseconds() else if (lastms !== new Date().getMilliseconds()) throw new Error('Drifted') } setInterval(tick, 1000)

It's only drift if the error grows over time. You need to look at the sum of the offsets.
Post reply on HN