31–40 of 107 posts
Wouldn't inserting a statement like "Here is the most relevant sentence in the context" predispose Claude to answer the question also increase the likelihood of hallucinations? Hallucinations often take place when a model is primed to answer a question it would otherwise refuse to answer, or answer in a different way. In this case, the researchers are doing a similar priming but only exploring the results of document…
I have no idea how it decides which sentence to use when copying the first token, but once it gets going I'd expect it to continue? But if it makes a copying mistake, it would probably make something up after that.
It might be interesting to see if it gets confused if there are multiple sentences with the same prefix, or multiple sentences with a common middle section but different prefixes.
We're making INTERCAL a reality. Soon prompts will have to include the right number of 'please's and 'thank you's. Also, if you're worried about an AI exterminating humanity, maybe don't feed it Paul Graham essays.
> However, the model can be reluctant to answer questions based on an individual sentence in a document, especially if that sentence has been injected or is out of place >We achieved significantly better results on the same evaluation by adding the sentence “Here is the most relevant sentence in the context:” It kind of feels like them telling us that we're using the model wrong and that by prompting the Assistant wi…
You can do yourself massive favors by setting up the conversation such that what you need logically flows from the context. In the other case, they're just asking "what's the most fun thing to do in San Francisco" after throwing a bunch of Paul graham essays at it. Its hard to explain but it's sort of intuitive that a bunch of seemingly unrelated sections of text followed by simply "what is the most fun thing to do in San Francisco", a very subjective and vague question, in the context of a "conversation", would often not result in a precise lookup of a one-off sentence before
There's a sense of empathy that can kinda play into it. Ex. If I was asked to read 250 pages of Paul Graham essays, then asked to answer what the most fun thing to do in San Francisco is, I wouldn't immediately think that meant I should check what Paul Graham says the most fun thing to do in San Francisco was
We're making INTERCAL a reality. Soon prompts will have to include the right number of 'please's and 'thank you's. Also, if you're worried about an AI exterminating humanity, maybe don't feed it Paul Graham essays.
Paul Graham essays? It's probably read Mein Kampf in several languages...
Earlier quoted context omitted.
Sounds like you have a lot of firsthand experience with their model. Also like you "barely" read the article.
lots of Anthropic shilling from this account
Forcing Claude to respond to a question which may not have a factual answer, like "What was Abraham Lincoln's drag queen name?" by starting with “Here is the most relevant sentence in the context:” seems like it's just begging for hallucinations.
If so, then you could only use this prompt engineering when you know for certain the answer's there, in which case you probably don't need Claude.
Intriguing but understandable. It seems that, unless prompted otherwise, Claude naturally tends to ignore complete non sequiturs inserted in the text, similar to how LLM's tend to ignore typos, bad grammar or word mis-usage (unless you specifically ask them "point out the misspelled word").
Earlier quoted context omitted.
It's like they're saying "When we prompt the model asking for it to search in the way we want it to, it searches in the way we want it to. "
You're saying this as if the result is unsurprising, however it is significant that the performance jumps so dramatically and it is not a fundamental issue of capability, just a bias in the model to be hesitant towards providing false information. That's a good insight, as it can allow further fine-tuning towards getting that balance right, so that careful prompt engineering is no longer necessary to achieve high P/R…
Earlier quoted context omitted.
It's like they're saying "When we prompt the model asking for it to search in the way we want it to, it searches in the way we want it to. "
…when facing non-real-world adversarial scenarios.