Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

41–50 of 723 posts

Re: Claude 3 model family

#41

Post by ethbr1 »

hackerlight wrote: post #29
RugnirViking wrote: post #16

I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output? Claude isn't available in EU yet, else i'd try it myself. :(

One good sign is they're only a slight improvement on knowledge recall evals but a big improvement on code and reasoning evals. Hope this stands up to scrutiny and we get something better than GPT-4 for code generation. Although the best model is a lot more expensive.

On the other hand, programmers are very expensive.

At some level of accuracy and consistency (human order-of-magnitude?), the pricing of the service should start approaching the pricing of the human alternative.

And first glance at numbers, LLMs are still way underpriced relative to humans.

Re: Claude 3 model family

#42

Post by lumost »

_sword wrote: post #14

At this point I wonder how much of the GPT-4 advantage has been OpenAI's pre-training data advantage vs. fundamental advancements in theory or engineering. Has OpenAI mastered deep nuances others are missing? Or is their data set large enough that most test-cases are already a sub-set of their pre-training data?

This may explain the substantial performance increase in proprietary models over the last 6 months. It also may explain why open-air and others had to drop open models. Distributing copyrighted material via model weights would be problematic.

Re: Claude 3 model family

#43

Post by Alifatisk »

michaelt wrote: post #30
Alifatisk wrote: post #19

I hate that they require a phone number but this might be the only way to prevent abuse so I'll have to bite the bullet. > We’ve made meaningful progress in this area: Opus, Sonnet, and Haiku are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models. Finally someone who takes this into account, Gemini and chatGPT is such an obstacle sometimes…

> I hate that they require a phone number https://openrouter.ai/ lets you make one account and get API access to a bunch of different models, including Claude (maybe not v3 yet - they tend to lag by a few days). They also provide access to hosted versions of a bunch of open models. Useful if you want to compare 15 different models without bothering to create 15 different accounts or download 15 x 20GB of models :)

I could only send one message, after that I had to add more credits to my account. I don't really think it's worth paying if I already get Gemini, chatGPT and Claude for free.

Re: Claude 3 model family

#44

Post by declaredapple »

asah wrote: post #13
nuz wrote: post #5

Earlier quoted context omitted.

GPT-4 was created like 3 years ago internally

the market is evaluating LLMs based on what's actually available. No GPT5 = users go elsewhere. GPT-4 has little "lock-in" and isn't "good enough" the keep users via inertia.

> No GPT5 = users go elsewhere.

You're not wrong, but most of the big players will take a while to switch, at least in my experience you have to put more effort into making sure your prompts result in what you want, and that's annoying especially if GPT4 is already working for you. Claude historically has really bad refusals for safe prompts.

Also, GPT4 is cheaper 10/30 $/m vs 15/75 $/m for claude 3 opus - I'm not sure that price hike is worth the _slight_ benchmark improvement.

Re: Claude 3 model family

#46

Post by ankit219 »

_sword wrote: post #14

At this point I wonder how much of the GPT-4 advantage has been OpenAI's pre-training data advantage vs. fundamental advancements in theory or engineering. Has OpenAI mastered deep nuances others are missing? Or is their data set large enough that most test-cases are already a sub-set of their pre-training data?

More than pretraining data, I think the advantage was ChatGPT and how quickly it grew. Remember it was 3.5, and within a month or two, it generated so many actual q&a pairs with rating, feedback, and production level data of how a model will be used by actual users. Those queries and subsequent RLHF + generating better answers for the questions meant the model would have been improved a lot at the SFT stage. Think this is the reason why Anthropic, Google, and Mistral, all three launched their own chatbots, all providing it to users for free and getting realtime q&a data for them to finetune the models on. Google did it with bard too, but it was so bad that not many used it.

Re: Claude 3 model family

#47

Post by egeozcan »

Alifatisk wrote: post #23
RugnirViking wrote: post #16

I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output? Claude isn't available in EU yet, else i'd try it myself. :(

> Claude isn't available in EU yet, else i'd try it myself. I'm currently in EU and I have access to it?

AFAIK there's no strict EU ban but no EU country is listed here:

https://www.anthropic.com/claude-ai-locations

Perhaps you meant Europe the continent or using a VPN?

edit: They seem to have updated that list after I posted my comment, the outdated list I based my comment on: https://web.archive.org/web/20240225034138/https://www.anthr...

edit2: I was confused. There is another list for API regions, which has all EU countries. The frontend is still not updated.

Re: Claude 3 model family

#48

Post by cjbprime »

ethbr1 wrote: post #31
vermorel wrote: post #28

Does any of those LLM-as-a-service companies provide a mechanism to "save" a given input? Paying only for the state storage and the extra input when continuing the completion from the snapshot? Indeed, at 1M token and $15/M tokens, we are talking of $10+ API calls (per call) when maxing out the LLM capacity. I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact sam…

How would that work technically, from a cost of goods sold perspective? (honestly asking, curious)

I think the answer's in the original question: the provider has to pay for extra storage to cache the model state at the prompt you're asking to snapshot. But it's not necessarily a net increase in costs for the provider, because in exchange for doing so they (as well as you) are getting to avoid many expensive inference rounds.

Re: Claude 3 model family

#49

Post by mrtksn »

up6w6 wrote: post #22

The Opus model that seems to perform better than GPT4 is unfortunately much more expensive than the OpenAI model. Pricing (input/output per million tokens): GPT4-turbo: $10/$30 Claude 3 Opus: $15/$75

That's quite expensive indeed. At full context of 200K, that would be at least $3 per use. I would hate it if I receive a refusal as answer at that rate.

Re: Claude 3 model family

#50

Post by wokwokwok »

nuz wrote: post #5
saliagato wrote: post #2

[flagged]

GPT-4 was created like 3 years ago internally

People don't use GPT-4 because it was created 3 years ago, or because it's pink, or because it has a 4 in the name.

They use it because it's better than any other publicly available model for most people.

If this is better, and people can access it, they'll use it instead of GPT-4.

People would already be using gemini ultra instead if they could access it, but google fucked the rollout out by telling everyone about it and then saying no one could play with it.

> Opus and Sonnet are available to use today in our API, which is now generally available, enabling developers to sign up and start using these models immediately.

Sounds pretty good.

If OpenAI want to stay in the game, they need something more than offering 'GPT-TEAM, all the features you already had!' or 'We made this 3 years ago'.

Sora was really fantastic. No one has access.

> Opus and Sonnet are available to use today in our API, which is now generally available, enabling developers to sign up and start using these models immediately.

Tell me this doesn't sound a littllllle bit more exciting than anything OpenAI has been releasing recently?

I look forward to their response... but I agree with sentiment that they better not sit around twiddling their thumbs; the world is moving fast.

Post reply on HN