Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

31–40 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#31
Some might remember this from other comment threads, I like to always benchmark with this question:

> I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.

because it looks like a simple math problem but there's a lot of real world knowledge about racing you can apply to provide a more useful answer. gpt2-chatbot is better than I've ever seen.

Here's the full response with LaTeX syntax trimmed. I've never seen a model consider whether the formation lap needs to be factored in. And I especially like how this model is uncertain about that, so it gives the resulting number with and without it. It is extra nice that it understands this concern applies only to video games.

> To calculate how many liters of fuel you will need for a 20-minute race in Assetto Corsa Competizione, we first need to determine how many laps you'll complete during the race. Here’s how you can do it step-by-step:

> Convert Your Lap Time to Minutes: Your lap time is 2 minutes and 4.317 seconds. Convert this entirely into minutes: [latex trimmed]

> Calculate the Number of Laps in the Race: The race duration is 20 minutes, so divide the total race time by your lap time to find out how many laps you will complete: [latex trimmed] Since you cannot complete a fraction of a lap, you should round this number to the nearest whole number. Typically, you should round up, as you'll need to complete all laps started before the 20-minute mark is reached. So, you will complete approximately 10 laps.

> Calculate Fuel Usage Per Lap: You've mentioned that the car uses 2.73 liters of fuel per lap.

> Calculate Total Fuel Needed: Multiply the number of laps by the fuel usage per lap: [latex trimmed] This is the total amount of fuel you would need, theoretically, to complete the race.

> Consider Additional Factors:

> Fuel for Formation Lap: If your game/race format includes a formation lap, remember to add fuel for this extra lap. That's an additional 2.73 liters.

> Safety Margin: It's also wise to include a margin to account for variations in fuel consumption (like higher consumption in traffic, different driving styles, or fuel usage during race incidents). A safe margin could be an additional 5-10% of the calculated fuel need. [latex trimmed]

> Rounding this number up, you might consider starting the race with about 33 liters to be on the safe side if there's a formation lap, or 30 liters if there isn't.

> This calculation should help you ensure that you do not run out of fuel during the race, taking into account both necessary consumption and a reasonable safety margin. Drive safely and best of luck in your race!

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#32
post #19

Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers. ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output None is correct.…

To be fair, this is pretty hard. Imagine you had to do to sit down and write this without being able to test it.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#33

What's interesting it seems to also edit text in between for me, not just add text at the end.

I'm seeing this as well. I don't quite understand how it's doing that in the context of LLMs to date being a "next token predictor". It is writing code, then adding more code in the middle.

Are you seeing a `...` suffix when the model is in the middle of writing a code block? I'm pretty sure that's just a display bug in lmsys.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#36
My usual question when testing a new model is `Can you write a function to convert svg to png in browser?` I dunno why, it's just become this.

This model is by far the most human like, most models just give me a method which do this, but this one made a html file, allowing me an option to pick a svg using file picker, a new .js file included in the script.

Even wrote a detailed explaination, additional notes, when this wont work, the output was very well structured.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#37
Man, its knowledge is insane.

I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it.

And then I asked "who are other notable users at " and it gave me a list of some mods but also stand out users. It knew the types of posts they wrote and the subforums they spent time in. And without a single hallucination.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#39

All of the facts based queries I have asked so far have not been 100% correct on any LLM including this one. Here are some examples of the worst performing: "What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer. "Is there a pre-existing controller or utility to migrate persistent volume claims from one storage…

> Discord gives the right answer.

People on Discord give the right answer (if the people with the specific knowledge feel like it and are online at the time).

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#40
Very impressive

Prompt:

> there are 3 black blocks on top of an block that we don't know the color of and beneath them there is a blue block. We remove all blocks and shuffle the blocks with one additional green block, then put them back on top of each other. the yellow block is on top of blue block. What color is the block we don't know the color of? only answer in one word. the color of block we didn't know the color of initially

ChatGPT 4: Green

Claude3 Opus: Green

GPT2-Chatbot: Yellow

Post reply on HN