Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

301–310 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#301

Post by irthomasthomas »

Given that a few people mentioned it having better than average knowledge of obscure things from the internet, I asked it what the llm tools from Simon Willison is. Gpt2-chatbot said "Log Lady's Mug". Gpt-4-turbo said "Log Lady Mysteries".

https://imgur.com/LHg4m5V

I tried a few versions of the prompt, including asking first about shot-scraper. It knows shot-scraper was made by Simon Willison and mostly knows how to use it, but that didn't help it with llm. Long tail is still a problem.

If I had API access to this, I might be interested in trying to find if there exists a prompt that can improve this sort of thing.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#302

Post by vincefav »

the_shivers wrote: post #206

Alas, it still fails at my favorite music theory question. I've yet to see a chatbot get this right, even though it isn't a trick question at all. I ask: "Consider a tenor ukulele in standard tuning. If we tune it down by a half step, how might we then finger a Gmaj7?" It initially reasons correctly that this must be G-B-D-F# and it understands what the resulting strings will be tuned to if we drop them a half step.…

I feel like that’s more of a ukulele question than a music theory question, but I do wonder if it gets it wrong for the same reason diffusion models struggle with hands.

Music theory is a great topic to test though, since GPT-4 struggles a little with it.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#303

Post by bamboozled »

razodactyl wrote: post #170
orbital-decay wrote: post #139

Earlier quoted context omitted.

I don't think a magical prompt is suddenly going to make any current public model draw an ASCII unicorn like this thing does. (besides, it already leaks a system prompt which seems very basic)

https://imgur.com/a/z39k8xz Seems like it's not too hard an ask.

It’s curious you and the other guy got the almost exact same response for the unicorn.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#304

Post by minosu »

qeternity wrote: post #248
hehdhdjehehegwv wrote: post #165

Earlier quoted context omitted.

Funny thing is I’m still in love with Mistral 7B as it absolutely shreds on a nice GPU. For simple tasks it’s totally sufficient.

Llama3 8B is for all intents and purposes just as fast.

Mistral 7b inferences about 18% faster for me as a 4bit quantized version on an A100. Thats definitely relevant when running anything but chatbots.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#306

Post by svnt »

sanxiyn wrote: post #194
sanxiyn wrote: post #189

Earlier quoted context omitted.

Compare, this from real Nabokov (he wrote long sentences; sentences in imitation actually needs to be longer): Twelve years and some eight months later, two naked children, one dark-haired and tanned, the other dark-haired and milk-white, bending in a shaft of hot sunlight that slanted through the dormer window under which the dusty cartons stood, happened to collate that date (December 16, 1871) with another (August…

Compare, a Nabokov imitation about high speed rail written by Claude: The sleek, serpentine carriages slithered through the verdant landscape, their velocity a silver-streaked affront to the indolent clouds above. Inside, passengers sat ensconced in plush seats, their faces a palimpsest of boredom and anticipation, while the world beyond the tinted windows blurred into a smear of colors -- an impressionist painting i…

You can sure see the sentence length and structure limitations imposed, though

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#308

Post by Jensson »

Kuinox wrote: post #279
swyx wrote: post #193

Earlier quoted context omitted.

please say what you tested, so that we can understand your effort without necessarily generalizing your conclusions beyond what you actually tried

You can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.

> You can ask it's knowledge cutoff and it will respond November 2023

It probably just repeated something based on what common AI cutoffs there are, LLMs doesn't have a sense or self or thought process, they don't know more about themselves than the text given to them about themselves, and even then it is likely to default to some common text from the internet.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#309

Post by Kuinox »

david_allison wrote: post #291
Kuinox wrote: post #279

Earlier quoted context omitted.

You can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.

It's close to live-ingesting GitHub. I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.

It's unable to know anything about the project I work on, so I don't think so.
Post reply on HN