Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

41–50 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#41

Post by Imnimo »

I'd be interested to see how this does on Nicholas Carlini's benchmark:

https://nicholas.carlini.com/writing/2024/my-benchmark-for-l...

I've tried out some of my own little test prompts, but most of those are tricky rather than practical. At least for my inputs, it doesn't seem to do better than other top models, but I'm hesitant to draw conclusions before seeing outputs on more realistic tasks. It does feel like it's at least in the ballpark of GPT-4/Claude/etc. Even if it's not actually GPT-4.5 or whatever, it's still an interesting mystery what this model is and where it came from.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#42

Post by JacobiX »

The model provides verbose answers even when I asked for more succinct ones. It still struggles with arithmetic (for example, it incorrectly stated "7739 % 23 = 339 exactly, making 23 a divisor"). When tested with questions in French, the responses were very similar to those of GPT-4. It is far better in knowledge based questions, I've asked this difficult one (it is not 100% correct but better than other LLMs) :

In Anna Karenina what does it mean: most of us prefer the company of Claras ?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#43

Post by msoad »

sebzim4500 wrote: post #32
msoad wrote: post #19

Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers. ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output None is correct.…

To be fair, this is pretty hard. Imagine you had to do to sit down and write this without being able to test it.

Styling I understand but to tick the clock when time actually has changed you need to use animation frames APIs and check for clock change more often than "every 1000 ms" because setTimeout will eventually drift even if you start exactly on first second time change. This is a test for depth of knowledge of a programmer I used to use in the past in interviews.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#44

Post by brainwad »

For translation, this thing is pretty amazing. "Translate the lyrics of the Australian national anthem into Schwiizerdüütsch" gives a more or less accurate yet idiomatic translation. ChatGPT 3.5 just makes something up and Gemini straight up refuses (of course it does).

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#46

Post by throwup238 »

steve_adams_86 wrote: post #10

The results of this LLM are consistently far better than any other that I choose. I asked ‘what is the most efficient approach to building a led grow light with off-the-shelf parts?’ and its response was incredible. Very much in line with how I’ve done it in the past after weeks of research, trial and error, and feedback from people. The other LLMs gave mostly reasonable yet sparse and incomplete answers. It also opt…

I'm asking it about how to make turbine blades for a high bypass turbofan engine and it's giving very good answers, including math and some very esoteric material science knowledge. Way past the point where the knowledge can be easily checked for hallucinations without digging into literature including journal papers and using the math to build some simulations.

I don't even have to prompt it much, I just keep saying "keep going" and it gets deeper and deeper. Opus has completely run off the rails in comparison. I can't wait till this model hits general availability.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#48

Post by xdennis »

I'm surprised by people's impression. I tried it in my own language and much worse than GPT-4.

Of the open source LLMs I've tried, all suck in non-English. I imagine it's difficult to make an LLM work in tens of languages on a consumer computer.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#49

Post by msoad »

Prompt: my mother's sister has two brothers. each of her siblings have at least one child except for the sister that has 3 children. I have four siblings. How many grandchildren my grandfather has? Answer only with the result (the number)

ChatGPT4: 13

Claude3 Opus: 10 (correct)

GPT2-Chatbot: 15

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#50

Post by nicklecompte »

My very first response from gpt2-chatbot included a fictional source :(

> A study by Lucon-Xiccato et al. (2020) tested African clawed frogs (Xenopus laevis) and found that they could discriminate between two groups of objects differing in number (1 vs. 2, 2 vs. 3, and 3 vs. 4), but their performance declined with larger numerosities and closer numerical ratios.

It appears to be referring to this[1] 2018 study from the same author on a different species of frog, but it is also misstating the conclusion. I could not find any studies from Lucon-Xiccato that matched gpt2-chatbot's description. Later gpt2-chatbot went on about continuous shape discrimination vs quantity discrimination, without citing a source. Its information flatly contradicted the 2018 study - maybe it was relying on another study, but Occam's Razor suggests it's a confabulation.

Maybe I just ask chatbots weird questions. But I am already completely unimpressed.

[1] https://www.researchgate.net/profile/Tyrone-Lucon-Xiccato/pu...

Post reply on HN