Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

461–470 of 723 posts

Re: Claude 3 model family

#461

Post by woadwarrior01 »

j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

Wow, I really couldn't believe it! Checked it myself, and it seems to get it wrong[1]. 4-bit quantized Mixtral Instruct running locally, gets it right[2].

[1]: https://imgur.com/a/zRI8wKZ

[2]: https://imgur.com/a/0On1I52

Re: Claude 3 model family

#462

Post by emporas »

moffkalast wrote: post #447
elorant wrote: post #439

Earlier quoted context omitted.

Groq's Mixtral 8x7b nails this one though. https://groq.com/ Sally has 1 sister. This may seem counterintuitive at first, but let's reason through it: We know that Sally has 3 brothers, and she is one of the sisters. Then we are told that each brother has 2 sisters. Since Sally's brothers share the same parents as Sally, they share the same sisters. Therefore, Sally's 3 brothers have only 1 additional sister besides…

If you change the names and numbers a bit, e.g. "Jake (a guy) has 6 sisters. Each sister has 3 brothers. How many brothers does Jake have?" it fails completely. Mixtral is not that good, it's just contaminated with this specific prompt. In the same fashion lots of Mistral 7B fine tunes can solve the plate-on-banana prompt but most larger models can't, for the same reason. https://arxiv.org/abs/2309.08632

This is not Mistral 7b, it is Mixtral 7bx8 MoE. I use the Chrome extension Chathub, and i input the same prompts for code to Mixtral and ChatGPT. Most of the time they both get it right, but ChatGpt gets it wrong and Mixtral gets it right more often than you would expect.

That said, when i tried to put many models to explain some lisp code to me, the only model which figured out that the lisp function had a recursion in it, was Claude. Every other LLM failed to realize that.

Re: Claude 3 model family

#463

Post by Spivak »

woadwarrior01 wrote: post #461
j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

Wow, I really couldn't believe it! Checked it myself, and it seems to get it wrong[1]. 4-bit quantized Mixtral Instruct running locally, gets it right[2]. [1]: https://imgur.com/a/zRI8wKZ [2]: https://imgur.com/a/0On1I52

> 4-bit quantized Mixtral Instruct running locally, gets it right

This has been one of my favorite things to play around with when it comes to real life applications. Sometimes a smaller "worse" model will vastly outperform a larger model. This seems to happen when the larger model overthinks the problem. Trying to do something simple like "extract all the names of people in this block of text" Llama 7B will have significantly fewer false positives than LLama 70B or GPT4.

Re: Claude 3 model family

#464

Post by brookman64k »

j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

mixtral:8x7b-instruct-v0.1-q4_K_M got this correct 5 out of 5 times. Running it locally with ollama on a RTX 3090.

Re: Claude 3 model family

#465

Post by throwaway63820 »

scrollop wrote: post #395
throwaway63820 wrote: post #390

Earlier quoted context omitted.

By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature 0 is the correct evaluation.

So your theory would have it that if you repeated the question at temp 1 it would give the wrong answer more often than the correct answer?

There's no theory.

Just in real life usage, it is extremely uncommon to stochastically query the model and use the most common answer. Using it with temperature 0 is the "best" answer as it uses the most likely tokens in each completion.

Re: Claude 3 model family

#466

Post by moffkalast »

emporas wrote: post #462
moffkalast wrote: post #447

Earlier quoted context omitted.

If you change the names and numbers a bit, e.g. "Jake (a guy) has 6 sisters. Each sister has 3 brothers. How many brothers does Jake have?" it fails completely. Mixtral is not that good, it's just contaminated with this specific prompt. In the same fashion lots of Mistral 7B fine tunes can solve the plate-on-banana prompt but most larger models can't, for the same reason. https://arxiv.org/abs/2309.08632

This is not Mistral 7b, it is Mixtral 7bx8 MoE. I use the Chrome extension Chathub, and i input the same prompts for code to Mixtral and ChatGPT. Most of the time they both get it right, but ChatGpt gets it wrong and Mixtral gets it right more often than you would expect. That said, when i tried to put many models to explain some lisp code to me, the only model which figured out that the lisp function had a recursion…

I've tested with the Mixtral on LMSYS direct chat, gen params may vary a bit of course. In my experience running it locally it's been a lot more finicky to get it to work consistently compared to non-MoE models so I don't really keep it around anymore.

3.5-turbo's coding abilities are not that great, specialist 7B models like codeninja and deepseek coder match and sometimes outperform it.

Re: Claude 3 model family

#467

Post by jasondclinton »

j-b wrote: post #451
jasondclinton wrote: post #435

Earlier quoted context omitted.

Seems stochastic? This is what I see from Opus which is correct: https://claude.ai/share/f5dcbf13-237f-4110-bb39-bccb8d396c2b Did you perhaps run this on Sonnet?

Ran with Opus, 0 temp. Screenshot included (original comment) for reference.

Thank you! Might also be seeing performance improved by by our system prompt on claude.ai.

Re: Claude 3 model family

#468

Post by yawnxyz »

behnamoh wrote: post #254
SubiculumCode wrote: post #204

Earlier quoted context omitted.

How many uses do you get per day of Opus with the pro subscription?

100 messages per 8 hours: https://support.anthropic.com/en/articles/8324991-about-clau...

Interesting that Opus and Sonnet have the same limits

Re: Claude 3 model family

#469

Post by ActVen »

technics256 wrote: post #332
ActVen wrote: post #272

Opus just crushed Gemini Pro and GPT4 on a pretty complex question I have asked all of them, including Claude 2. It involved taking a 43 page life insurance investment pdf and identifying various figures in it. No other model has gotten close. Except for Claude 3 sonnet, which just missed one question.

I am curious on this. can you share more?

Here is the list of the questions. https://imgur.com/a/D4xwczU The PDF can't be shared. But, it looks something like the one here: https://content.naic.org/sites/default/files/call_materials/...

Re: Claude 3 model family

#470

Post by lossolo »

j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

It's because they learn small patterns from datasets, it doesn't matter whether the subjects are Sally, George, sisters, or apples. If a particular logic pattern was not in the training dataset, then the model did not learn it and will fail on most variations of this riddle. These transformer models are essentially large collections of local optima over logic patterns in sentences. If a pattern was not present in the dataset, there is no local optimum for it, and the model will likely fail in those cases.
Post reply on HN