361–370 of 723 posts
Opus just crushed Gemini Pro and GPT4 on a pretty complex question I have asked all of them, including Claude 2. It involved taking a 43 page life insurance investment pdf and identifying various figures in it. No other model has gotten close. Except for Claude 3 sonnet, which just missed one question.
They claim that the new models "are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models", looks like about a third of "incorrect refusals" compared to Claude 2.1. Given that Claude 2 was completely useless because of this, this still feels like a big limitation.
The guard rails on the models make the llm-market a complete train wreck. Wish we could just collectively grow up and accept that if a computer says something bad that doesn't have any negative real world impact - unless we let it - just like literally any other tool.
Opus just crushed Gemini Pro and GPT4 on a pretty complex question I have asked all of them, including Claude 2. It involved taking a 43 page life insurance investment pdf and identifying various figures in it. No other model has gotten close. Except for Claude 3 sonnet, which just missed one question.
I have access to it and I can test it against Pro 1.5
Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
A: Okay, let's organize the information we have:
Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step:
We know that Sally is a girl, so she is a sister to her brothers. Each of Sally's brothers has 2 sisters. One of these sisters is Sally, but who is the other sister? Since Sally has only brothers and no other sisters are mentioned, we can conclude that the other sister for each brother is also Sally.
Therefore, Sally has no sisters other than herself. So, the answer is that Sally has 0 sisters.
Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100. It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.…
Hi, CISO of Anthropic here. Thank you for the feedback! If you can share any details about the image, please share in a private message. No LLM has had an emergent calculator yet.
Another naming disaster! Opus is better than sonnet? And sonnet is better than haiku? Perhaps this makes sense to people familiar with sonnets and haikus and opus....es? Nonsensical to me! I know everyone loves to hate on Google, but at least pro and ultra have a sort of sense of level of sophistication.
Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
Could anyone recommend an open-source tool capable of simultaneously sending the same prompt to various language models like GPT-4, Gemini, and Claude, and displaying their responses side by side for comparison? I tried chathub in the past, but they decided to not release any more source as of now.
Choose Arena (side-by-side), it has Claude 3 Opus, Sonnet and GPT-4
Earlier quoted context omitted.
This. Price is set by value delivered and what the market will pay for whatever capacity they have; it’s not a cost + X% market.
I'm more curious about the input/output token discrepancy Their pricing suggests that either output tokens are more expensive for some technical reason, or they're trying to encourage a specific type of usage pattern, etc.
The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…
(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)