Leading through these feels like a quick explanation of why I feel like some of the excitement for these products has gotten a bit ahead of the results.
Asking 60 LLMs a set of 20 questions
231–240 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#232I've been laughing my ass out all my life thinking about non-sensical crap like this in my head and apparently now we have the tech for it!
Re: Asking 60 LLMs a set of 20 questions
#233Leading through these feels like a quick explanation of why I feel like some of the excitement for these products has gotten a bit ahead of the results.
Really, I thought Claud and GPT were both excellent, and it show how far ahead they are of the rest.
Re: Asking 60 LLMs a set of 20 questions
#234Earlier quoted context omitted.
Right. By logging all of this, then vectorizing, then evaluating using some type of transfer knowledge (classify this response and it's accuracy), one could build up a routing table of likely candidates for building an ensemble to evaluate the correct answer.
Maybe you can fine tune an llm for the routing thing ;)
Stuff that stuff into a semantic graph and call it done.
Re: Asking 60 LLMs a set of 20 questions
#235Earlier quoted context omitted.
I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?
I don't think so. Consider the following conversation from UndiMix-v2-13B (a local LLM running on my Mac M1 via llama.cpp). It gets it wrong at first, but then gets it right with a little hint. I don't see how the hint could work if it was merely repeating an example it knew. User: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Llama: Hello there! I'd be happy to help you…
She counts as her own sister :D I can be sister of myself?
Re: Asking 60 LLMs a set of 20 questions
#236Spoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."
I read a Time article which used an LLM to generate Onion-style headlines and the best one they got was “rural town up in arms over depiction in summer blockbuster ‘Cow Fuckers.’” Crude, admittedly.
Re: Asking 60 LLMs a set of 20 questions
#237Which LLM is best for coding? Specifically code architecture and system design. I can't really find any comparisons of this online they mostly focus on commonly solved coding problems.
Re: Asking 60 LLMs a set of 20 questions
#238The amazing thing this demonstrated to me is how many capable models there are out there now. The progress in the last year has been incredible!
Re: Asking 60 LLMs a set of 20 questions
#239Earlier quoted context omitted.
Humans. After all, LLMs are designed to reason equal to or better than humans.
Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.
The thing with humans is they will say “I don’t remember how many syllables a haiku has” and “what the hell is kubernetes?” No LLM can reliably produce a haiku because their lexing process deprives them of reliable information about syllable counts. They should all say “I’m sorry, I can’t count syllables, but I’ll try my best anyway.” But the current models don’t do that because they were trained on texts by humans, who can do haiku, and not properly taught their own limits by reinforcement learning. It’s Dunning Kruger gone berserk.
Re: Asking 60 LLMs a set of 20 questions
#240Earlier quoted context omitted.
I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?
I've been talking to GPT and asking it long multi part questions and it does seem to 'think through' them, and able to provide negative responses. Specific logic questions like in this post, or long math formulas seem to still be a struggle. But I get the impression it is just a matter of time before it is scaled up to handle them. You can argue whether it is 'reasoning' about them or not. But if it becomes indisting…
Indeed, anyone who says llm's cannot reason since it is just reformulating things it remembers, is unable to explain what reasoning is, let alone how it would not be reasoning.
In the end, if the result is indistinguishable from a treat people call reasoning, let's call the behavior reasoning.