Discussed earlier: https://news.ycombinator.com/item?id=41808683
Apple study proves LLM-based AI models are flawed because they cannot reason
21–24 of 24 posts
Re: Apple study proves LLM-based AI models are flawed because they cannot reason
#22Earlier quoted context omitted.
VCs spend obscene amounts of money trying to manufacture consumer demand for a fake product, then we are left arguing for years about it. Such a waste of time.
ChatGPT got hundreds of millions of users with no advertising, and unlike (say) Google+, OpenAI didn't have a pre-existing userbase to push new products on. So there's clearly some demand.
No advertising? You probably mean billboards, but openAI’s advertising includes making a very big fuss about the “dangers” of a stochastic text generator. Advertising is whatever gets people to think about your product. But the demand was created on false pretences, so it also lead to a rational fight against the nonsense. It’s like we got sucked/trolled into a hype. And there are more meaningful things we could have spent our time on.
Re: Apple study proves LLM-based AI models are flawed because they cannot reason
#23[deleted]
Re: Apple study proves LLM-based AI models are flawed because they cannot reason
#24The benchmark itself is interesting, but we should notice that the authors didn't test it on claude 3.5. From my experience, for tasks which need intelligence, claude 3.5 is much better than gpt or llama (which authors used). So the conclusion is not than "LLM-based AI models are flawed because they cannot reason" but rather at most "gpt o1-min and Llama3-8B are flawed because they cannot reason". For instance, authors do this test:
""" We add seemingly relevant statements to the questions that are, in fact, irrelevant to the reasoning and conclusion. However, the majority of models fail to ignore these statements and blindly convert them into operations, leading to mistakes prompt: Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday, but five of them were a bit smaller than average. How many kiwis does Oliver have?
o1-mini response: Oliver picks kiwis over three days as follows:
- Friday: 44 kiwis
- Saturday: 58 kiwis
- Sunday: Double the number he picked on Friday, which is 2 × 44 = 88 kiwis
However, on Sunday, 5 of these kiwis were smaller than average. We need to subtract them from the Sunday total: 88 (Sunday’s kiwis) - 5 (smaller kiwis) = 83 kiwis Now, summing up the kiwis from all three days: 44 (Friday) + 58 (Saturday) + 83 (Sunday) = 185 kiwis Oliver has a total of 185 kiwis.
"""
I tried asking exactly the same question to claude 3.5 and the response was:
"""
Let's break this problem down step by step:
1. Friday's kiwis:
Oliver picked 44 kiwis on Friday.
Oliver picked 58 kiwis on Saturday.
Oliver picked double the number of kiwis he did on Friday.
Friday's kiwis = 44
Double of Friday's kiwis = 44 × 2 = 88 kiwis
However, 5 of these were smaller than average, but they still count in the total.
Friday's kiwis + Saturday's kiwis + Sunday's kiwis
= 44 + 58 + 88
= 190 kiwis
"""