Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

311–320 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#311

Post by Grimblewald »

moffkalast wrote: post #152
vincelt wrote: post #118

Earlier quoted context omitted.

Have you tried replicating via the API with a temp of 0?

Using a temp of zero usually returns garbage results from most models, so it would likely do so in case of GPT 4 as well. Any other great ideas?

Not in my experience, in fact I find that when I need precise, realistic, and reliable results temp 0 is needed. For example, here is a bunch of names, gather the names of specific plastics under headings matching their common acronym - if I don't use temp 0 I might get nonsense out. Temp 0? reliably correct.

Re: Asking 60 LLMs a set of 20 questions

#312

Post by moffkalast »

Grimblewald wrote: post #311
moffkalast wrote: post #152

Earlier quoted context omitted.

Using a temp of zero usually returns garbage results from most models, so it would likely do so in case of GPT 4 as well. Any other great ideas?

Not in my experience, in fact I find that when I need precise, realistic, and reliable results temp 0 is needed. For example, here is a bunch of names, gather the names of specific plastics under headings matching their common acronym - if I don't use temp 0 I might get nonsense out. Temp 0? reliably correct.

Interesting, that's the exact opposite of my experience.

Re: Asking 60 LLMs a set of 20 questions

#313

Post by glinkot »

belter wrote: post #101
nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I confirm GPT-4 solves this correctly. Makes me immediately doubt everything else in the article...

When calling the GPT-4 API I get this answer:

Sally has 2 sisters. Each brother counts Sally and two other girls as sisters, which means there are three girls in total. Since one of these girls is Sally herself, that leaves two others as her sisters.

When I said 'nope', it corrected itself and gave the right answer.

Re: Asking 60 LLMs a set of 20 questions

#314

Post by actualwitch »

pininja wrote: post #229

Spoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."

I'm pretty sure Luminous Base Control here stopped 1 step away from a great joke. I will just go ahead and assume the answer is "a vacation":

I'm not sure if this is the right category, but here goes: What do you call a vacation that's too short? A...

Re: Asking 60 LLMs a set of 20 questions

#315

Post by anaganisk »

jongjong wrote: post #306

I was playing around with GPT a while back and I found that it could come up with some good jokes if I started the joke with a subject. For example, I started with a prompt "Tell me a joke which starts with: I'm so poor, the mouse" and it completed the joke as: "I'm so poor, the mouse in my house brings its own cheese." Some other ones I still remember which cracked me up: "I'm so poor, after I stepped on a cockroach…

[flagged]

Re: Asking 60 LLMs a set of 20 questions

#316

Post by simonh »

ruszki wrote: post #153
appplication wrote: post #102

Earlier quoted context omitted.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

I don’t think that it’s not an improvement. It’s not an improvement in context of finding new genuine solutions, sure. But that’s definitely not needed most of the time in real life for an average person, just like it’s not needed for an average developer anymore.

It creates the impression that the tool can do something it actually can’t, or is good at something when it isn’t.

Re: Asking 60 LLMs a set of 20 questions

#317

Post by santiagobasulto »

civilitty wrote: post #86
facu17y wrote: post #75

Earlier quoted context omitted.

It might be trained on this question or a variant of it.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

I'm curious, what logic puzzles do you use for evaluation? I could use them for my own students.

Re: Asking 60 LLMs a set of 20 questions

#318

Post by cubacaban »

ftxbro wrote: post #222
jhbadger wrote: post #170

Earlier quoted context omitted.

People say that "they are literally Markov chains", but anyone who has looked at the code for LLMs knows that they are more complicated than that. I implemented Markov chains in BASIC in about ten lines of code in the 1980s on a 1 Mhz 64K Apple II after reading about the famous Mark V. Shaney hoax ( https://en.wikipedia.org/wiki/Mark_V._Shaney ). No neural nets or fancy GPUs required. It's one thing to stress that LL…

> People say that "they are literally Markov chains", but anyone who has looked at the code for LLMs knows that they are more complicated than that. They are literally Markov chains according to the mathematical definition. The code is complicated. Having complicated code doesn't mean it's not literally a Markov chain. > I implemented Markov chains in BASIC in about ten lines of code in the 1980s on a 1 Mhz 64K Apple…

How big is the state space of the Markov chain corresponding to a LLM generating a sequence of tokens? Wouldn't it be (size of the vocabulary)^(size of the context window), i.e. ~ (100k)^(4k)? How useful is it to conceptualize LLMs as Markov chains at that point? For example, is there a result about Markov chains with interesting implications for LLMs?

Re: Asking 60 LLMs a set of 20 questions

#319

Post by BurningFrog »

b112 wrote: post #141
BurningFrog wrote: post #139

Earlier quoted context omitted.

This assumes there are no half sisters/brothers in the family.

It's fair that a half sister is not a sister. At best, you'd expect two half sisters to be a full sister, otherwise, they're a different thing.

I have 1 full sibling and three half siblings, and sometimes like to confuse people by saying I have 2½ siblings.

Re: Asking 60 LLMs a set of 20 questions

#320

Post by Kiro »

anaganisk wrote: post #315
jongjong wrote: post #306

I was playing around with GPT a while back and I found that it could come up with some good jokes if I started the joke with a subject. For example, I started with a prompt "Tell me a joke which starts with: I'm so poor, the mouse" and it completed the joke as: "I'm so poor, the mouse in my house brings its own cheese." Some other ones I still remember which cracked me up: "I'm so poor, after I stepped on a cockroach…

[flagged]

When searching for them all I can find is this thread.
Post reply on HN