[deleted]
Claude 3 model family
81–90 of 723 posts
Re: Claude 3 model family
#82At this point I wonder how much of the GPT-4 advantage has been OpenAI's pre-training data advantage vs. fundamental advancements in theory or engineering. Has OpenAI mastered deep nuances others are missing? Or is their data set large enough that most test-cases are already a sub-set of their pre-training data?
I'd guess a bit of both, perhaps more on the data side. One could also flip the question and ask how is this new Anthropic model able to beat GPT-4 in some benchmarks?
As far as data, OpenAI haven't just scraped/bought existing data, they have also on a fairly large scale (hundreds of contractors) had custom datasets created, which is another area they may have a head start unless others can find different ways around this (e.g. synthetic data, or filtering for data quality).
Altman has previously said (on Lex's podcast I think) that OpenAI (paraphrasing) is all about results and have used some ad-hoc approaches to achieve that, without hinting at what those might be. But, given how fast others like Anthropic and Google are catching up I'd assume each has their own bag of tricks too, whether that comes down to data and training or architectural tweaks.
Re: Claude 3 model family
#83Surpassing GPT4 is huge for any model, very impressive to pull off. But then again...GPT4 is a year old and OpenAI has not yet revealed their next-gen model.
MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.
Re: Claude 3 model family
#84Surpassing GPT4 is huge for any model, very impressive to pull off. But then again...GPT4 is a year old and OpenAI has not yet revealed their next-gen model.
From the blog's footnote:
"In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model"
Re: Claude 3 model family
#85Earlier quoted context omitted.
I'd argue it's actually risky to build an AI startup now. Most any feature you bring to the table will be old news when the AI manufacturers add that to their platform.
You just need to focus niche and upmarket, OpenAI is e.g. never going to make that "clone your chats and have your LLM-self go on pre-dates" app that went around Twitter.
Yeah but that kind of stuff doesn't generate income, they're just cute programming toys.
Re: Claude 3 model family
#86It doesn’t matter how advanced these generative AIs get. What matters more is what their companies deem as “reasonable” queries. What’s the point when it responds with a variant of “I’m sorry, but I can’t help you with that Dave”
Claude is just as bad as Gemini at this. Non-binged ChatGPT is still the best at simply agreeing to answer a normal question.
Re: Claude 3 model family
#87Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100. It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.…
I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue.
For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE
Re: Claude 3 model family
#88It's too bad they put Claude in a straight jacket and won't let it answer any question that has a hint of controversy. Worse, it moralizes and implies that you shouldn't be asking those questions. That's my impression from using Claude (my process is to ask the same questions of GPT-4, Pi, Claude and Gemini and take the best anwser). The free Claude I've been using uses something called "constitutional reinforcement…
[deleted]
Re: Claude 3 model family
#89It feels absolutely amazing to build an AI startup right now: - We struggled with limited context windows [solved] - We had issues with consistent JSON output [solved] - We had rate limiting and performance issues with 3rd party models [solved] - Hosting OSS models was a pain [solved] It's like your product becomes automatically cheaper, more reliable, and more scalable with every major LLM advancement. I'm going to…
[deleted]
Re: Claude 3 model family
#90[deleted]
[deleted]