Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

291–300 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#291

Post by david_allison »

Kuinox wrote: post #279
swyx wrote: post #193

Earlier quoted context omitted.

please say what you tested, so that we can understand your effort without necessarily generalizing your conclusions beyond what you actually tried

You can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.

It's close to live-ingesting GitHub.

I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#292

Post by HarHarVeryFunny »

zarzavat wrote: post #243
somenameforme wrote: post #234

Earlier quoted context omitted.

You know, people often complain about goal shifting in AI. We hit some target that was supposed to be AI (or even AGI), kind of go meh - and then change to a new goal. But the problem isn't goal shifting, the problem is that the goals were set to a level that had nothing whatsoever to do where we "really" want to go, precisely in order to make them achievable. So it's no surprise that when we hit these neutered goals…

There are different shades of AGI, but we don’t know if they will happen all at once or not. For example, if an AI can replace the average white collar worker and therefore cause massive economic disruption, that would be a shade of AGI. Another shade of AGI would be an AI that can effectively do research level mathematics and theoretical physics and is therefore capable of very high-level logical reasoning. We don’t…

The term "AGI" has been loosely used for so many years that it doesn't mean anything very specific. The meaning of words derives from their usage.

To me Shane Legg's (DeepMind) definition of AGI meaning human level across full spectrum of abilities makes sense.

Being human or super-human level at a small number of specialized things like math is the definition of narrow AI - the opposite of general/broad AI.

As long as the only form of AI we have is pre-trained transformers, then any notion of rapid self-improvement is not possible (the model can't just commandeer $1B of compute for a 3-month self-improvement run!). Self-improvement would only seem possible if we have an AI that is algorithmically limited and does not depend on slow/expensive pre-training.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#293

Post by simonw »

mdp2021 wrote: post #216
simonw wrote: post #113

Earlier quoted context omitted.

If your everyday work consists of looking up academic citations then yeah, LLMs are not going to be useful for that - you'll get hallucinations every time. That's absolutely not a task they are useful for. There are plenty of other tasks that they ARE useful for, but you have to actively seek those out.

(Hi Simon, I am laughing as I write - I just submitted an article from your blog minutes ago. Then stumbled into this submission, and just before writing this reply, I checked the profile of "simonw"... I did not know it was your username here.) Well, assuming one normally queries for information, if the server gives false information then you have failure and risk. If one were in search for supplemental reasoning (e…

Many of the most interesting uses of LLMs occur when you move away from using them as a source of information lookup - by which I mean pulling directly from information encoded into their opaque model weights.

Anything where you feed information into the model as part of your prompt is much less likely to produce hallucinations and mistakes - that's why RAG question answering works pretty well, see also summarization, fact extraction, structure data conversion and many forms of tool usage.

Uses that involve generating code are very effective too, because code has a form of fact checking built in: if the model hallucinates an API detail that doesn't exist you'll find out the moment you (or the model itself via tools like ChatGPT Code Interpreter) execute that code.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#294

Post by Workaccount2 »

hehdhdjehehegwv wrote: post #143
steve_adams_86 wrote: post #10

The results of this LLM are consistently far better than any other that I choose. I asked ‘what is the most efficient approach to building a led grow light with off-the-shelf parts?’ and its response was incredible. Very much in line with how I’ve done it in the past after weeks of research, trial and error, and feedback from people. The other LLMs gave mostly reasonable yet sparse and incomplete answers. It also opt…

The level of competition right now is so crazy, if it ISN’T a new version of ChatGPT then OpenAI is in trouble. GPT-4 isn’t significantly better than Llama 3. Sure, it’s better, but the margins are evaporating fast. (I’d mention Gemini…but why bother? IMHO, this is now OpenAI the company vs Open models)

Don't sleep on Gemini 1.5. The 1,000,000 token context window is crazy when you can dump everything from a single project (hundreds, even thousands of documents) into it and then inference. Sure it's not the strongest model, but it is still good, and its the best when you can basically train it on whatever you are working with.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#295

Post by HarHarVeryFunny »

PoignardAzur wrote: post #259
93po wrote: post #94

Earlier quoted context omitted.

i just tested this too, really cool. i own a yaris have used an online forum for yaris cars for the past decade and had a vague memory of a user who deleted some of the most helpful guides. i asked about it and sure enough it knew exactly who i meant: who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?: One notable user from the Yaris forums who deleted many of their helpful guid…

Holy crap. Even if this is RAG-based, this is insanely good.

Yes, but this is where current LLMs shine - transforming (in this case summarizing) text rather than generating anything factual from scratch or reasoning/planning.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#296

Post by ChildOfChaos »

aaroninsf wrote: post #55

I find it notable that it's not GPT-2. It's GPT2.

Sam seems to have referenced this in his tweets, were he tweeted:

"I have a soft spot for GPT-2" and then "I have a soft spot of GPT2"

Considering that it reports back it's GPT 4, I'm guessing the underlaying model is the same/slightly tweaked GPT4, but something else is different and it's that which is a 'v2' version, maybe agents, reasoning layer etc.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#297

Post by jprete »

Eisenstein wrote: post #284
concurrentsquar wrote: post #201

Earlier quoted context omitted.

Reddit may have told OpenAI to pay (probably a lot of) money to legally use Reddit content for training, which is something Reddit is doing with other AI labs ( https://www.cbsnews.com/news/google-reddit-60-million-deal-a... ); but GPTBot is not banned under the Reddit robots.txt ( https://www.reddit.com/robots.txt ). This is assuming that lmsys' GPT-2 is retained GPT-4t or a new GPT-4.5/5 though; I doubt that (one o…

Sam Altman was on the board of reddit until recently. I don't know how these things work in SV but I wouldn't think one would go from 'partly running a company' to 'being charged for something that is probably not enforceable'. It would maybe make sense if they did pay reddit for it, because it isn't Sam's money, anyway, but for reddit to demand payment and then OpenAI to just not use the text data from reddit -- one…

Maybe training on whatever this is started before the licensing deal?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#298

Post by Workaccount2 »

krainboltgreene wrote: post #181
cma wrote: post #160

Earlier quoted context omitted.

Are you trying the paid gpt or just free 3.5 chatgpt?

100% of the time when I post a critique someone replies with this. I tell them I've used literally every LLM under the sun quite a bit to find any use I can think of and then it's immediately crickets.

Usually when I encounter sentiment like this it is because they only have used 3.5 (evidently not the case here) or that their prompting is terrible/misguided.

When I show a lot of people GPT4 or Claude, some percentage of them jump right to "What year did Nixon get elected?" or "How tall is Barack Obama?" and then kind of shrug with a "Yeah, Siri could do that ten years ago" take.

Beyond that you have people who prompt things like "Make a stock market program that has tabs for stocks, and shows prices" or "How do you make web cookies". Prompts that even a human would struggle greatly with.

For the record, I use GPT4 and Claude, and both have dramatically boosted my output at work. They are powerful tools, you just have to get used to massaging good output from them.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#299

Post by ipython »

shepherdjerred wrote: post #114
humansareok1 wrote: post #78

Earlier quoted context omitted.

Unfortunately this is not reliable, many Non-GPT models happily claim to be GPT-4 e.g.

I simply asked it "what are you" and it responded that it was GPT-4 based. > I'm ChatGPT, a virtual assistant powered by artificial intelligence, specifically designed by OpenAI based on the GPT-4 model. I can help answer questions, provide explanations, generate text based on prompts, and assist with a wide range of topics. Whether you need help with information, learning something new, solving problems, or just loo…

Why would the model be self aware? There is no mechanism for the llm to know the answer to “what are you” other than training data it was fed. So it’s going to spit out whatever it was trained with, regardless of the “truth”

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#300

Post by SirMaster »

skerit wrote: post #269
hombre_fatal wrote: post #37

Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…

It does seem to have more data. I asked it about some of my Github projects that don't have any stars, and it responded correctly. Wasn't able to use direct-chat, so I always chose it as the winner in battle mode! OpenAI has been crawling the web for quite a while, but how much of that data have they actually used during training? It seems like this might include all that data?

Hmm, I asked it about my GitHub project that has been out for 4 years and it got everything completely wrong.
Post reply on HN