That actually looks like a pretty good rebuttal of the original test.
I wonder if this also works on other 200k models like yi
Long context prompting for Claude 2.1
11–20 of 107 posts
Re: Long context prompting for Claude 2.1
#12Earlier quoted context omitted.
It’s not where , it’s how .
It's like they're saying "When we prompt the model asking for it to search in the way we want it to, it searches in the way we want it to. "
You're saying this as if the result is unsurprising, however it is significant that the performance jumps so dramatically and it is not a fundamental issue of capability, just a bias in the model to be hesitant towards providing false information. That's a good insight, as it can allow further fine-tuning towards getting that balance right, so that careful prompt engineering is no longer necessary to achieve high P/R on this task.
Re: Long context prompting for Claude 2.1
#13I wonder if you can preempt it but as part of the user message. For example: Human: {context} What is the most fun thing to do in San Francisco based on the context? Don't give in formation outside the document. Start with "Here is the most relevant sentence in the context:" Assistant: It just feels more natural to do it like that especially when constructing the prompt based on various factors.
[deleted]
Re: Long context prompting for Claude 2.1
#14That actually looks like a pretty good rebuttal of the original test. I wonder if this also works on other 200k models like yi
Yes, I think I agree if I am understanding correctly - the test is not a good fit for how it works, because it "wants" to weigh things based on surrounding context and to give a lower weight to things that it feels are out of place. That makes it likely a great candidate for certain kinds of work, like sentiment analysis and just overall literary understanding.
Re: Long context prompting for Claude 2.1
#15I wonder if you can preempt it but as part of the user message. For example: Human: {context} What is the most fun thing to do in San Francisco based on the context? Don't give in formation outside the document. Start with "Here is the most relevant sentence in the context:" Assistant: It just feels more natural to do it like that especially when constructing the prompt based on various factors.
I realize it's all just embeddings and probability blah blah blah... But this kind of meta prompting is really interesting to me. Can you ask a model about its weights?
Re: Long context prompting for Claude 2.1
#16I wonder if you can preempt it but as part of the user message. For example: Human: {context} What is the most fun thing to do in San Francisco based on the context? Don't give in formation outside the document. Start with "Here is the most relevant sentence in the context:" Assistant: It just feels more natural to do it like that especially when constructing the prompt based on various factors.
You can try, but in general, this is less reliable. Prompt-based instructions to start or end a response with certain strings or templates are not, for any models, 100% successful in producing the requested behavior.
Re: Long context prompting for Claude 2.1
#17Intriguing but understandable. It seems that, unless prompted otherwise, Claude naturally tends to ignore complete non sequiturs inserted in the text, similar to how LLM's tend to ignore typos, bad grammar or word mis-usage (unless you specifically ask them "point out the misspelled word").
Re: Long context prompting for Claude 2.1
#18Weird that a company releases an article about how it can barely control the output of its own model