Look at that jump in grade school math. From 55 % with GPT 3.5 to 95 % for both Claude 3 and GPT 4.
Claude 3 model family
21–30 of 723 posts
Re: Claude 3 model family
#22The Opus model that seems to perform better than GPT4 is unfortunately much more expensive than the OpenAI model.
Pricing (input/output per million tokens):
GPT4-turbo: $10/$30
Claude 3 Opus: $15/$75
Re: Claude 3 model family
#23I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output? Claude isn't available in EU yet, else i'd try it myself. :(
> Claude isn't available in EU yet, else i'd try it myself.
I'm currently in EU and I have access to it?
Re: Claude 3 model family
#24The results really aren’t striking enough that it’s clear that this model blows GPT-4 away. It seems roughly equivalent, give or take a bit.
Why can we still not easily surpass a (relatively) ancient model?
Re: Claude 3 model family
#25Wow. 1 million token length.
Yeah this is huge, first Gemini and now Claude!
Re: Claude 3 model family
#26Did some quick tests and Claude 3 Sonnet responses have been mostly wrong compared to Gemini :/ (was asking it to describe certain GitHub projects and Claude was making stuff up)
Re: Claude 3 model family
#27I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output? Claude isn't available in EU yet, else i'd try it myself. :(
You can use Claude 2.1 on openrouter. Hopefully, they will be able to add the Claude 3 family too.
Re: Claude 3 model family
#28Does any of those LLM-as-a-service companies provide a mechanism to "save" a given input? Paying only for the state storage and the extra input when continuing the completion from the snapshot?
Indeed, at 1M token and $15/M tokens, we are talking of $10+ API calls (per call) when maxing out the LLM capacity.
I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact same knowledge base seems very inefficient.
Right now, only ChatGPT (the webapp) seems to be using such those snapshots.
Am I missing something?
Re: Claude 3 model family
#29I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output? Claude isn't available in EU yet, else i'd try it myself. :(
One good sign is they're only a slight improvement on knowledge recall evals but a big improvement on code and reasoning evals. Hope this stands up to scrutiny and we get something better than GPT-4 for code generation. Although the best model is a lot more expensive.
Re: Claude 3 model family
#30I hate that they require a phone number but this might be the only way to prevent abuse so I'll have to bite the bullet. > We’ve made meaningful progress in this area: Opus, Sonnet, and Haiku are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models. Finally someone who takes this into account, Gemini and chatGPT is such an obstacle sometimes…
> I hate that they require a phone number
https://openrouter.ai/ lets you make one account and get API access to a bunch of different models, including Claude (maybe not v3 yet - they tend to lag by a few days). They also provide access to hosted versions of a bunch of open models.
Useful if you want to compare 15 different models without bothering to create 15 different accounts or download 15 x 20GB of models :)