GPT-4.5 or GPT-5 being tested on LMSYS?

If you ask it about the model name/cutoff date it claims to be "ChatGPT, based on GPT-4" and that the cutoff date is "Nov 2023." It claims that consistently so I think it might be accurate.

Impressive! We’re approaching the “our model can rewrite stolen content word for word” territory.

The better it gets the more people will get frustrated that they get less website visitors to their content and instead ChatGPT is swallowing them whole.

Interesting to think about tho!

All of the facts based queries I have asked so far have not been 100% correct on any LLM including this one.

Here are some examples of the worst performing:

"What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer.

"Is there a pre-existing controller or utility to migrate persistent volume claims from one storage class to another in the open source Kubernetes ecosystem?" It said no (wrong) and then provided another approach that partially used Velero that wasn't correct, if you know what Velero does in those particular commands. Discord communities give the right answer, such as `pvmigrate` (https://github.com/replicatedhq/pvmigrate).

Here is something more representative:

"What alternatives to Gusto would you recommend? Create a table showing the payroll provider in a column, the base monthly subscription price, the monthly price per employee, and the total cost for 3 full time employees, considering that the employees live in two different states" This and Claude do a good job, but do not correctly retrieve all the prices. Claude omitted Square Payroll, which is really the "right answer" to this query. Google would never be able to answer this "correctly." Discord gives the right answer.

The takeaway is pretty obvious right? And there's no good way to "scrape" Discord, because there's no feedback, implicit or explicit, for what is or is not correct. So to a certain extend their data gathering approach - paying Kenyans - is sort of fucked for these long tail questions. Another interpretation is that for many queries, people are asking the wrong places.

Sadly, still fails my test of reproducing code that implements my thesis (Dropback Continuous Pruning), which I used because it's vaguely complicated and something I know very well. It totally misses the core concept of using an PRNG and instead implements some pretty standard pruning+regrowth algo.