Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

151–160 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#151

Post by FrustratedMonky »

appplication wrote: post #102
civilitty wrote: post #86

Earlier quoted context omitted.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

Are you implying that to counter these logic puzzles that GPT4 was specifically trained on logic puzzles so it would know the answers?

In that case, just make new problems. If it is being 'patched' to pass specific known problems, then the new ones would fail.

If it is able to answer them, then maybe it is actually analyzing them and working out the solution.

Not sure how you can assume there was no underlying improvement, and these are cases of feeding it the answers.

Re: Asking 60 LLMs a set of 20 questions

#152

Post by moffkalast »

vincelt wrote: post #118
belter wrote: post #101

Earlier quoted context omitted.

I confirm GPT-4 solves this correctly. Makes me immediately doubt everything else in the article...

Have you tried replicating via the API with a temp of 0?

Using a temp of zero usually returns garbage results from most models, so it would likely do so in case of GPT 4 as well. Any other great ideas?

Re: Asking 60 LLMs a set of 20 questions

#153

Post by ruszki »

appplication wrote: post #102
civilitty wrote: post #86

Earlier quoted context omitted.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

I don’t think that it’s not an improvement. It’s not an improvement in context of finding new genuine solutions, sure.

But that’s definitely not needed most of the time in real life for an average person, just like it’s not needed for an average developer anymore.

Re: Asking 60 LLMs a set of 20 questions

#154

Post by FrustratedMonky »

jonwinstanley wrote: post #97
nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

I've been talking to GPT and asking it long multi part questions and it does seem to 'think through' them, and able to provide negative responses.

Specific logic questions like in this post, or long math formulas seem to still be a struggle. But I get the impression it is just a matter of time before it is scaled up to handle them.

You can argue whether it is 'reasoning' about them or not. But if it becomes indistinguishable from a human, then does the word 'reason' even really apply to the human anymore? Both are black boxes, giving answers.

Re: Asking 60 LLMs a set of 20 questions

#155

Post by CamperBob2 »

artursapek wrote: post #41

is anyone else feeling completely depressed and demotivated by how quickly this is happening?

No. When we were kids, my generation was promised flying cars, unlimited fusion power, and sentient computers. There's a good chance I'll live to see one out of three of those things happen, and that's better than the zero out of three I thought we'd get.

Re: Asking 60 LLMs a set of 20 questions

#157

Post by thisisthenewme »

artursapek wrote: post #41

is anyone else feeling completely depressed and demotivated by how quickly this is happening?

Not depressed but just makes me question the value of our existence. If we are truly able to create machines with our intellectual potential, where does that leave us? Do we compete with the machines in a rat-race for a chance at happy existence? Or will we create a utopia where the machines do the difficult work and allow for an enjoyable existence for all. The people who are rooting for LLM's are hopefully working towards the latter but not sure if that is just wishful thinking.

Re: Asking 60 LLMs a set of 20 questions

#158

Post by carlossouza »

appplication wrote: post #102
civilitty wrote: post #86

Earlier quoted context omitted.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

And what’s more data to a model if not patches that inflate performance?

The more data we use to train a model (or as you said, the more patches we use), the better it’s performance will be.

Re: Asking 60 LLMs a set of 20 questions

#159

Post by circuit10 »

moffkalast wrote: post #152
vincelt wrote: post #118

Earlier quoted context omitted.

Have you tried replicating via the API with a temp of 0?

Using a temp of zero usually returns garbage results from most models, so it would likely do so in case of GPT 4 as well. Any other great ideas?

What do you mean? It works fine for me when I’ve tried it

Re: Asking 60 LLMs a set of 20 questions

#160

Post by visarga »

NoraCodes wrote: post #52
smokel wrote: post #45

Earlier quoted context omitted.

What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?

Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…

> Given that people are already firing real human workers to replace them with worse but cheaper LLMs

That's impossible, LLMs are not that good. They might be firing people and crashing service quality.

Post reply on HN