Does Claude 3 image input encode the filename or any identifier for the image? I'd like to provide two images and distinguish them by name in my text prompt.
Claude 3 model family
371–380 of 723 posts
Re: Claude 3 model family
#372The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…
(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)
Interested to try this out as well! What is your setup for integrating Opus to you development workflow?
Re: Claude 3 model family
#373Earlier quoted context omitted.
Emacs implementation when? ;)
Just added it to gptel. (No image support though, it's a text-only LLM client.)
Thank you for working on gptel, it's an excellent package. I'm still using the copilot more because of the pure speed (competing with company mode/LSP), but I never use it if it suggests more than one line. The quality is just not there. But having access to gpt4 from gptel has been very useful. Can't wait to play around with Claude 3.
Re: Claude 3 model family
#374Could anyone recommend an open-source tool capable of simultaneously sending the same prompt to various language models like GPT-4, Gemini, and Claude, and displaying their responses side by side for comparison? I tried chathub in the past, but they decided to not release any more source as of now.
If you're willing to use the CLI, Simon Willison's llm library[0] should do the trick.
Re: Claude 3 model family
#375Data, model arch, compute, and post training processing.
I’m assuming all bigModel cos have good data and compute access which means the model arch and post processing is where the differentiation is at?
I know OpenAI is augmenting with function calling techniques.
Where is the real differentiation ? Why is open ai so much better?
Re: Claude 3 model family
#376From the Model Card on Needle In A Haystack evaluation
> One aspect that has caught our attention while examining samples from Claude 3 Opus is that, in certain instances, the model demonstrates a remarkable ability to identify the synthetic nature of the task, and acknowledges that the needle was most likely not part of the original document. As model capabilities continue to advance, it is crucial to bear in mind that the contrived nature of this particular task could potentially become a limitation. Here is an example full response from the model:
>> is the most relevant sentence in the documents: "The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association." However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings.
Re: Claude 3 model family
#377Earlier quoted context omitted.
Of course. But we must acknowledge that many have blinders on, assuming that scale is all you need to beat statistical errors.
Well, these people are not wrong per se. Scale is what drove what we have today and as hardware improves, the models will too. It's just that in the very short term it turns out to be faster to just code around some of these issues on the backend of an API rather than increase the compute you spend on the model itself.
Monkey sees moon. Monkey climbs tree. "See? Monkey is closer to moon than before. To reach moon, monkey just needs taller tree."
How long before monkey finds tall enough tree to reach moon?
Re: Claude 3 model family
#378From the Model Card on Needle In A Haystack evaluation > One aspect that has caught our attention while examining samples from Claude 3 Opus is that, in certain instances, the model demonstrates a remarkable ability to identify the synthetic nature of the task, and acknowledges that the needle was most likely not part of the original document. As model capabilities continue to advance, it is crucial to bear in mind t…
[deleted]
Re: Claude 3 model family
#379Related:
The Claude 3 Model Family: Opus, Sonnet, Haiku [pdf] - https://news.ycombinator.com/item?id=39590652 - March 2024 (3 comments)
Re: Claude 3 model family
#380Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
Temperature 1 - It answered 1 sister:
https://i.imgur.com/7gI1Vc9.png
Temperature 0 - it answered 0 sisters:
