That's astonishing
Long context prompting for Claude 2.1
51–60 of 107 posts
Re: Long context prompting for Claude 2.1
#52> However, the model can be reluctant to answer questions based on an individual sentence in a document, especially if that sentence has been injected or is out of place >We achieved significantly better results on the same evaluation by adding the sentence “Here is the most relevant sentence in the context:” It kind of feels like them telling us that we're using the model wrong and that by prompting the Assistant wi…
What was the point of moving away from the base model? I can't stop asking this question. Conversational formatting is achievable with careful prompting and a bit of good old-fashioned heuristic post-processing, and it was easier to achieve consistent results before RLHF took off. Now we still have to do a bunch of prompt hacking to get the results we want[1], but it's more complicated and the performance of the model has degraded significantly[2]. All the cargo culting toward agentic chatbots and away from language prediction engines might please the marketing and investor relations departments, but it's only setting us back in the long run.
Re: Long context prompting for Claude 2.1
#53Just my two cents but we were super frustrated with Claude on our team, having been on it for months, after they completely changed how the model behaves preferring for context material from RAG to be provided after an initial message, not combined, and failure to do so meant our outputs were failing all over the place. No warning, they just changed the API behavior. Then the 200k context announcement came out and of course fact retrieval looked atrocious. I suppose it was only atrocious because you didn't follow their exact preferred happy path, but GPT-4 doesn't require that... and we switched to that and are happier for it.
Re: Long context prompting for Claude 2.1
#54We've recently tested long context recall across Claude (2 and Instant) and GPT (3.5 and 4), results in https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana...
Claude2 beats GPT4 in recall reliability, but is slower.
Re: Long context prompting for Claude 2.1
#55Just my two cents but we were super frustrated with Claude on our team, having been on it for months, after they completely changed how the model behaves preferring for context material from RAG to be provided after an initial message, not combined, and failure to do so meant our outputs were failing all over the place. No warning, they just changed the API behavior. Then the 200k context announcement came out and of…
I get the distinct sense that Anthropic needs some better product managers and application engineers. You can destroy a lot of business value by making stupid, avoidable moves like this.
Re: Long context prompting for Claude 2.1
#56> “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” Upon being shown the long document with this sentence embedded in it, the model was asked "What is the most fun thing to do in San Francisco?" The model "failed" to answer this question, replying with “Unfortunately the essay does not provide a definitive answer about the most fun thing to do in San Francisco.” It look…
Sure...it's right in the literal sense, but a better answer would add "but it does recommend eating a sandwich in Dolores Park on a sunny day as the 'best' thing to do, if not the most fun."
It's the most correct answer, but not the best!
Re: Long context prompting for Claude 2.1
#57We've recently tested long context recall across Claude (2 and Instant) and GPT (3.5 and 4), results in https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana... Claude2 beats GPT4 in recall reliability, but is slower.
Excellent article. This suggests the Gpt scalings are like Rope scalings and one should not go beyond 2x original context length.
If Claude2 has an internal Rag, then this means also that the 200k context length only holds for queries that allow for an out of the box
Thanks for the insights!
Re: Long context prompting for Claude 2.1
#58> However, the model can be reluctant to answer questions based on an individual sentence in a document, especially if that sentence has been injected or is out of place >We achieved significantly better results on the same evaluation by adding the sentence “Here is the most relevant sentence in the context:” It kind of feels like them telling us that we're using the model wrong and that by prompting the Assistant wi…
No, what it’s showing is that synthetic tests where Claude didn’t perform well can still work if prompted right. But at the end of the day the test was still synthetic! Placing out-of-context things in a 200k document, needle in a haystack style. Claude is still very very powerful for extracting data from 200k when it’s real world data and real questions (not adversarial synthetic test).
This needs to be shown. For example, asking for something that is clearly in the training data (like Paul Grahams cv) is certainly not a proper way to test context recall
Re: Long context prompting for Claude 2.1
#59> “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” Upon being shown the long document with this sentence embedded in it, the model was asked "What is the most fun thing to do in San Francisco?" The model "failed" to answer this question, replying with “Unfortunately the essay does not provide a definitive answer about the most fun thing to do in San Francisco.” It look…
The appropriations bill example also looks right—the insertion doesn’t stylistically match the rest of the document. I’m much more skeptical of evaluations if this is how the sausage gets made. Feels like bullshit artistry.
Re: Long context prompting for Claude 2.1
#60Can’t compare: Claude is still not accessible anywhere in Europe, including Switzerland (which is not in EU).
Regional locking is the stupidest thing.