141–150 of 393 posts
Judging from the title, I was most interested in whether they tried to reason against some of the more logical phrases from Alice in Wonderland (e.g. "this bottle was not marked poison", "a grin without a cat", etc), and was disappointed to find that it was not the case.
As a non-coder I can get away with asking this: Why is it so hard to simulate reason? Logic and reason are based on rules. Then you add values to steer the conclusions based on the available data. Why not have separate systems for values and logic and memory working together as an AI brain to generate truly reasoned responses? You could even have adversarial parts that duke it out (left-wing vs right-wing, Jefferson…
LLMs operate on word co-occurances. Given an initial prompt (consisting of words), it tries to predict the most likely next word. And then it goes on to predict the next most likely word based on the previous output. There is also a bit of randomization involved, so that it doesn't always produce the same results. Some people call these LLMs "statistical parrots", because the LLMs simply repeat the word combinations…
It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data.
For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend).
In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.
Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper) I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64ca…
>Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong... Yes this is a common thing I see people who think LLMs are idiots do. The more an LLM talks the smarter it ge…
If you're among technologists discussing LLMs academically, as we are, that's a reasonable approach. However, I see a lot of people fail to distinguish that from LLM-powerd products sold to the general public as intelligent bots that can understand your plain english and output answers.
People use their existing mental models when interacting with something. If you have 3 different interfaces with a widget to trigger the same exact function, but one look like a music play button, one looks like a gas pedal, and one looks like mechanical pinball plunger, we interact with those things differently because we know how those things work. In this context, chatbots are designed to engage people's existing mental model for chatting with a person via text. The further you stray from people's expectations of human chat, the further you are from people's expectations, for better, or worse.
If you're selling someone a product claiming it understands plain language questions and gives plain language answers, then not getting the right answer to that question makes it idiotic. The subtleties aren't within most users' grasp, and the "FYI: this thing might be full of shit" disclaimer isn't helpful if you don't know enough about what you're asking to administer a proper smell test.
Your statements are obviously not wrong, but I see people saying these things like its reasonable for non-technical end users to reason about those subtleties. Considering how those things are marketed, I really don't think it is.
Earlier quoted context omitted.
Yeah, the overfitting is apparent with these kinds of edge cases, which makes you wonder where else it's over fitting. I can actually tell this when I'm working in a programming domain where I'm an expert and where a compiler holds the source of truth, but not in others, so I'm wary to use it there.
Interestingly you get a great answer if you change the prompt as follows which implies that it is not a puzzle: > I am on the shore of a river with a goat and a boat. How do I get to the other side? I work in commercial agriculture logistics. This gets a great answer even with GPT3.5
It’s an interesting paper, but my worry is that this is cherry-picking a surprising negative result. (The paper does not seem to discuss how many evals were considered to find this case.) The general expectation should be that some AI failure modes will be for things that seem obvious to humans. (Because they have a different architecture to humans and different strengths.) The important question is whether this is a…
For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong co…
Its not an AI hype. A hype is defined as something which gets oversold: "promote or publicize (a product or idea) intensively, often exaggerating its benefits." Just yesterday I visited a google cloud summit and one person from bosch told the audiance how they are now able to work with less external agencies like texting, graphicsdesigner and photographers for their materials. It already saves money, has real impacts…
A majority don’t deny that it’s good. The problem is that so many think it is actually reasoning, believing the answers can be trusted.
Earlier quoted context omitted.
If you really think about what an LLM is you would think there is no way that leads to general purpose AI. At the same time though they are already doing way more than we thought they could. Maybe people were surprised by what OpenAI achieved so now they are all just praying that with enough compute and the right model AGI will emerge.
> If you really think about what an LLM is you would think there is no way that leads to general purpose AI It is an autoregressive sequence predictor/generator. Explain to me how humans are fundamentally different
Many of the datasets for the "benchmarks" on which the major public LLMs are assessed are clearly present in their training data, making them basically useless for establishing reliability of the models. Its fairly obvious that at least some of the improved scores from later generations of models are that this benchmark data is increasingly represented in the training data. A better way of assessing LLMs is waiting a…
Simply picking answers at random should give you 25 points. Knowing 50% of the answers and picking the rest randomly gives you 62.5%, which is very close to the scores of SOTA LLMs. The benchmarks that supposedly show reasoning are pretty bad and have very little to do with reasoning. A lot of the questions can be answered through memorization.
I agree with you. The benchmarks are garbage. I thought about building my own benchmarks, but this would require building a complex benchmarking framework first and I just don't have the time for preparatory work like that.
Earlier quoted context omitted.
Its not an AI hype. A hype is defined as something which gets oversold: "promote or publicize (a product or idea) intensively, often exaggerating its benefits." Just yesterday I visited a google cloud summit and one person from bosch told the audiance how they are now able to work with less external agencies like texting, graphicsdesigner and photographers for their materials. It already saves money, has real impacts…
> AI doesn't need to be ready tomorrow, but its capabilities are already really good. A majority don’t deny that it’s good. The problem is that so many think it is actually reasoning, believing the answers can be trusted.
Earlier quoted context omitted.
>Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong... Yes this is a common thing I see people who think LLMs are idiots do. The more an LLM talks the smarter it ge…
Even if this were all true, it points to a fundamental risk of using LLM's for important tasks, which is that it is not at all clear to a user that this prompt would cause a problem. The LLM doesn't say "I'm sorry Dave, I just can't do that", it just complies with it and gets the wrong answer. You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or…
Good example: I sunk probably an hour into trying to get Gemini Advanced to help me integrate it with a personal Google Calendar account. I kept asking it things and going crazy because nothing lined up with the way things worked. Finally, it referred to itself as Bard and I realized it was giving me information for a different product. As soon as I asked "are you giving me instructions for Gemini Advanced or Bard?" it was like "OH LOL WOOPS!! YOU GOT ME BRO! XD I CAN'T DO ANY OF THAT! LOL." Which, honestly, is great. Being able to evaluate its answers to realize it's wrong is really neat. Unfortunately, it was neat too late and too manually to stop me from wasting a ton of time.
I have decades of experience working in software-- imagine some rando that didn't know what the hell Bard was or even imagine this thing with "Advanced" in the name couldn't even distinguish between its own and other products' documentation.