LLMs seem to mechanize poor average human performance then. Not noticing a "mis-placed" clause in a long contract, for example.
Another point against use in high risk applications.
Long context prompting for Claude 2.1
71–80 of 107 posts
Re: Long context prompting for Claude 2.1
#72Did anyone stumble upon expansion plans regarding availability? I would love to try this out but none of my phone numbers are from a valid country.
Re: Long context prompting for Claude 2.1
#73Did anyone stumble upon expansion plans regarding availability? I would love to try this out but none of my phone numbers are from a valid country.
There are services online that can help you out. Google is your friend.
Re: Long context prompting for Claude 2.1
#74We've recently tested long context recall across Claude (2 and Instant) and GPT (3.5 and 4), results in https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana... Claude2 beats GPT4 in recall reliability, but is slower.
My experience matched this as well.
GPT-4 Turbo is more watered down on the details with long context
But also it’s a newer feature for OpenAI, so they might catch up with next version
Re: Long context prompting for Claude 2.1
#75Earlier quoted context omitted.
No, what it’s showing is that synthetic tests where Claude didn’t perform well can still work if prompted right. But at the end of the day the test was still synthetic! Placing out-of-context things in a 200k document, needle in a haystack style. Claude is still very very powerful for extracting data from 200k when it’s real world data and real questions (not adversarial synthetic test).
This needs to be shown. For example, asking for something that is clearly in the training data (like Paul Grahams cv) is certainly not a proper way to test context recall
Link from thread
https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana...
Re: Long context prompting for Claude 2.1
#76Earlier quoted context omitted.
On a constructive note, these things will trickle down into the models. Bing for example already does "thinking" step that is hidden from the user. Also see this quote from Ethan Mollick on twitter: > I have a strong suspicion that “prompt engineering” is not going to be a big deal in the long-term & prompt engineer is not the job of the future > AI gets easier. You can already see in Midjourney how basic prompts wen…
Gosh I think I'll be a little sad about that future? I'm reminded of how we used to know really fun tricks for squeezing another bit of performance out of our assembly code -- "The Story of Mel" -- and then compilers started doing all the work for us. The past year or so of published literature on LLMs has been kind of hilarious because there is a substantial chunk of stuff whose contribution is "putting this extra E…
Imagine an assembly that you didn’t make, but was passed down to you by aliens.
Now we have to tinker with it to learn instead of read Textbooks
Re: Long context prompting for Claude 2.1
#77Intriguing but understandable. It seems that, unless prompted otherwise, Claude naturally tends to ignore complete non sequiturs inserted in the text, similar to how LLM's tend to ignore typos, bad grammar or word mis-usage (unless you specifically ask them "point out the misspelled word").
Scaling context is not something humans have good intuition for- I certainly don't recall an exact sentence from 200 pages ago. This is an area where we actually want the models to not mimic us.
Interestingly human memory works the other way.
We tend to remember out of place things more often.
E.g. if there was a kid in a pink hat and blue mustache at a suit and tie business party, everybody is going to remember the outlier.
Re: Long context prompting for Claude 2.1
#78Earlier quoted context omitted.
We'll need some kind of hybrid system to deal with this. For example the LLM 'indexes' the text it reads and assigns importance weights to parts of it, then as it moves to new text it can check back to these more important parts to ensure its not forgetting things.
I would think there is some benefit to synthesizing, and compressing. Summarization is similar in that the heavier weighed text remains and the rest is pruned. If the same basic information is all over a text, combine it.
We already know LLMs are good at summarizing.
Question is how good they are are retaining minute details from extremely long context, say 200k tokens.
That’s the frontier Claude and now GPT-4 Turbo are pushing
Re: Long context prompting for Claude 2.1
#79> “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” Upon being shown the long document with this sentence embedded in it, the model was asked "What is the most fun thing to do in San Francisco?" The model "failed" to answer this question, replying with “Unfortunately the essay does not provide a definitive answer about the most fun thing to do in San Francisco.” It look…
Language can be ambiguous.
But these LLMs were fine tuned on realistic human question and answer pairs to make them user friendly.
I’m pretty sure the average person wouldn’t prefer an LLM whose output is always playing grammar Nazi or semantics tai chi on every word you said.
There has to be a reasonable “error correction” on the receiving end for language to work as a communication channel.
Re: Long context prompting for Claude 2.1
#80Can’t compare: Claude is still not accessible anywhere in Europe, including Switzerland (which is not in EU). Regional locking is the stupidest thing.
And in this case it's even enforced through country-specific phone number verification... They _really_ don't want us to use it.