Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

91–100 of 723 posts

Re: Claude 3 model family

#91

Post by ankit219 »

simonw wrote: post #60
ankit219 wrote: post #46

Earlier quoted context omitted.

More than pretraining data, I think the advantage was ChatGPT and how quickly it grew. Remember it was 3.5, and within a month or two, it generated so many actual q&a pairs with rating, feedback, and production level data of how a model will be used by actual users. Those queries and subsequent RLHF + generating better answers for the questions meant the model would have been improved a lot at the SFT stage. Think th…

My understanding is that GPT-4 had been almost fully trained before ChatGPT was released - they spent around six months testing GPT-4 before making it available to the public, ChatGPT came out 31st November 2022, GPT-4 came out March 14th 2023. But maybe that was still enough time for them to instruction tune it based on ChatGPT feedback, or at least to focus more of their fine tuning iteration in the areas they lear…

I don't think it was pretrained on knowledge gaps. A version was already available in testing w select customers. The version released to the public would definitely have feedback from those customers, and finetuned/instruction tuned on the data from ChatGPT.

Training data is publicly available internet (and accessible to everyone). It's the SFT step w high quality examples which determines how well a model is able to answer questions. ChatGPT's virality played a part in that in the sense that OAI got the real world examples + feedback others did not have. And yeah, it would have been logical to focus on 3.5's weaknesses too. From Karpathy's videos, it seems they hired a contractual labelling firm to generate q&a pairs.

Re: Claude 3 model family

#92

Post by [deleted] »

declaredapple wrote: post #36
up6w6 wrote: post #22

The Opus model that seems to perform better than GPT4 is unfortunately much more expensive than the OpenAI model. Pricing (input/output per million tokens): GPT4-turbo: $10/$30 Claude 3 Opus: $15/$75

Yeah the output pricing I think is really interesting, 150% more expensive input tokens 250% more expensive output tokens, I wonder what's behind that? That suggests the inference time is more expensive then the memory needed to load it in the first place I guess?

[deleted]

Re: Claude 3 model family

#94

Post by addandsubtract »

egeozcan wrote: post #47
Alifatisk wrote: post #23

Earlier quoted context omitted.

> Claude isn't available in EU yet, else i'd try it myself. I'm currently in EU and I have access to it?

AFAIK there's no strict EU ban but no EU country is listed here: https://www.anthropic.com/claude-ai-locations Perhaps you meant Europe the continent or using a VPN? edit: They seem to have updated that list after I posted my comment, the outdated list I based my comment on: https://web.archive.org/web/20240225034138/https://www.anthr... edit2: I was confused. There is another list for API regions, which has all EU c…

They updated the list of supported countries here: https://www.anthropic.com/supported-countries

I was just able to sign up, while not being able to a few weeks ago.

Re: Claude 3 model family

#95

Post by hobofan »

aaomidi wrote: post #73
HarHarVeryFunny wrote: post #61

Earlier quoted context omitted.

Sure, OpenAI's next model would be expected to regain the lead, just due to their head start, but this level of catch-up from Anthropic is extremely impressive. Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models -…

Anthropic is also not really a traditional startup. It’s just some large companies in a trench coat.

How so? Because they have taken large investments from Amazon and Google? Or would you also characterize OpenAI as "Microsoft in a trench coat"?

Re: Claude 3 model family

#96

Post by drpossum »

One of my standard questions is "Write me fizzbuzz in clojure using condp". Opus got it right on the first try. Most models including ChatGPT have flailed at this as I've done evaluations.

Amazon Bedrock when?

Re: Claude 3 model family

#97

Post by jakderrida »

jorgemf wrote: post #56
mrtksn wrote: post #49

Earlier quoted context omitted.

That's quite expensive indeed. At full context of 200K, that would be at least $3 per use. I would hate it if I receive a refusal as answer at that rate.

cost is relative. how much would it cost for a human to read and give you an answer for 200k tokens? Probably much more than $3.

Yeah, but the human tends not to get morally indignant because my question involves killing a process to save resources.

Re: Claude 3 model family

#98

Post by hackerlight »

bugglebeetle wrote: post #83
Workaccount2 wrote: post #20

Surpassing GPT4 is huge for any model, very impressive to pull off. But then again...GPT4 is a year old and OpenAI has not yet revealed their next-gen model.

MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.

How can they avoid the contents from leaking into the training set somewhere in their internet scrape?

Re: Claude 3 model family

#100

Post by nine_k »

anonymouse008 wrote: post #87
virgildotcodes wrote: post #68

Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100. It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.…

I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE

I personally find approaches like this the correct way forward.

An input analyzer that finds out what kinds of tokens the query contains. A bunch of specialized models which handle each type well: image analysis, OCR, math and formal logic, data lookup,sentiment analysis, etc. Then some synthesis steps that produce a coherent answer in the right format.

Post reply on HN