So prompt engineering is back.
Long context prompting for Claude 2.1
61–70 of 107 posts
Re: Long context prompting for Claude 2.1
#62We've recently tested long context recall across Claude (2 and Instant) and GPT (3.5 and 4), results in https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana... Claude2 beats GPT4 in recall reliability, but is slower.
One recurring problem I have with Claude 2 is that it sometimes "bugs out" and starts to repeat the same token ad infinitum (which I still have to pay for). This happens with longer prompts, say, 30k. Have you encountered this issue?
Re: Long context prompting for Claude 2.1
#63We've recently tested long context recall across Claude (2 and Instant) and GPT (3.5 and 4), results in https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana... Claude2 beats GPT4 in recall reliability, but is slower.
One recurring problem I have with Claude 2 is that it sometimes "bugs out" and starts to repeat the same token ad infinitum (which I still have to pay for). This happens with longer prompts, say, 30k. Have you encountered this issue?
I haven't, but tbh we work a lot more with GPT than Claude so it's possible I haven't encountered many warts there.
For what we do (AI code writing), GPT output seems qualitatively much better than Claude's, but we want to keep our options open.
Re: Long context prompting for Claude 2.1
#64> “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” Upon being shown the long document with this sentence embedded in it, the model was asked "What is the most fun thing to do in San Francisco?" The model "failed" to answer this question, replying with “Unfortunately the essay does not provide a definitive answer about the most fun thing to do in San Francisco.” It look…
Sure...it's right in the literal sense, but a better answer would add "but it does recommend eating a sandwich in Dolores Park on a sunny day as the 'best' thing to do, if not the most fun." It's the most correct answer, but not the best!
[deleted]
Re: Long context prompting for Claude 2.1
#65although usually LLMs don't care, I would have also tried fixing the typo “Francico” and see if Claude acts differently
Re: Long context prompting for Claude 2.1
#66Earlier quoted context omitted.
One recurring problem I have with Claude 2 is that it sometimes "bugs out" and starts to repeat the same token ad infinitum (which I still have to pay for). This happens with longer prompts, say, 30k. Have you encountered this issue?
I haven't, but tbh we work a lot more with GPT than Claude so it's possible I haven't encountered many warts there. For what we do (AI code writing), GPT output seems qualitatively much better than Claude's, but we want to keep our options open.
Thanks!
I use it for classification for a personal project (non-commercial) and, for me, they are both pretty close in terms of quality. GPT-4 is better, but has a shorter window. I was hoping to reduce costs by using Claude exclusively, but that bug makes it too unreliable, sadly.
Re: Long context prompting for Claude 2.1
#67Earlier quoted context omitted.
No, what it’s showing is that synthetic tests where Claude didn’t perform well can still work if prompted right. But at the end of the day the test was still synthetic! Placing out-of-context things in a 200k document, needle in a haystack style. Claude is still very very powerful for extracting data from 200k when it’s real world data and real questions (not adversarial synthetic test).
This needs to be shown. For example, asking for something that is clearly in the training data (like Paul Grahams cv) is certainly not a proper way to test context recall
Could we feed it Anna Karenina and ask it what is a difference between happy and unhappy families?
Re: Long context prompting for Claude 2.1
#68> However, the model can be reluctant to answer questions based on an individual sentence in a document, especially if that sentence has been injected or is out of place >We achieved significantly better results on the same evaluation by adding the sentence “Here is the most relevant sentence in the context:” It kind of feels like them telling us that we're using the model wrong and that by prompting the Assistant wi…
What was the point of moving away from the base model? I can't stop asking this question. Conversational formatting is achievable with careful prompting and a bit of good old-fashioned heuristic post-processing, and it was easier to achieve consistent results before RLHF took off. Now we still have to do a bunch of prompt hacking to get the results we want[1], but it's more complicated and the performance of the mode…
Are you asking why use RLHF? It's a way to improve step by step reasoning. They are training a reward model to understand problem solving step by step, instead of just training reward model on the outcome. They then tune the model based on this reward model. It's shown to greatly improve performance on reasoning.
The reward models are kind of forgotten by everyone, but they are substantial transformer models with billions of parameters themselves. I think companies are using RLHF because it really helps align preferences/steer/improve performance.
Re: Long context prompting for Claude 2.1
#69Did they also test it by asking for fake information? Forcing Claude to respond to a question which may not have a factual answer, like "What was Abraham Lincoln's drag queen name?" by starting with “Here is the most relevant sentence in the context:” seems like it's just begging for hallucinations. If so, then you could only use this prompt engineering when you know for certain the answer's there, in which case you…
To verify you could either do a simple text search through the source document or utilize a 2-shot approach to double check the answer. Just take the answer from the first step and then ask the model again:
Given the following document:
Does this document support the following statement:
Re: Long context prompting for Claude 2.1
#70> “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” Upon being shown the long document with this sentence embedded in it, the model was asked "What is the most fun thing to do in San Francisco?" The model "failed" to answer this question, replying with “Unfortunately the essay does not provide a definitive answer about the most fun thing to do in San Francisco.” It look…
this comment and comment section eerily reminds me of Reddit and i'm sad HN is turning into that.