Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

451–460 of 723 posts

Re: Claude 3 model family

#451

Post by j-b »

jasondclinton wrote: post #435
j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

Seems stochastic? This is what I see from Opus which is correct: https://claude.ai/share/f5dcbf13-237f-4110-bb39-bccb8d396c2b Did you perhaps run this on Sonnet?

Ran with Opus, 0 temp. Screenshot included (original comment) for reference.

Re: Claude 3 model family

#452

Post by ukuina »

moffkalast wrote: post #447
elorant wrote: post #439

Earlier quoted context omitted.

Groq's Mixtral 8x7b nails this one though. https://groq.com/ Sally has 1 sister. This may seem counterintuitive at first, but let's reason through it: We know that Sally has 3 brothers, and she is one of the sisters. Then we are told that each brother has 2 sisters. Since Sally's brothers share the same parents as Sally, they share the same sisters. Therefore, Sally's 3 brothers have only 1 additional sister besides…

If you change the names and numbers a bit, e.g. "Jake (a guy) has 6 sisters. Each sister has 3 brothers. How many brothers does Jake have?" it fails completely. Mixtral is not that good, it's just contaminated with this specific prompt. In the same fashion lots of Mistral 7B fine tunes can solve the plate-on-banana prompt but most larger models can't, for the same reason. https://arxiv.org/abs/2309.08632

Meanwhile, GPT4 nails it every time:

> Jake has 2 brothers. Each of his sisters has 3 brothers, including Jake, which means there are 3 brothers in total.

Re: Claude 3 model family

#453

Post by torginus »

strangescript wrote: post #438
j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

This is definitely a problem, but you could also ask this question to random adults on the street who are high functioning, job holding, and contributing to society and they would get it wrong as well. That is not to say this is fine, but more that we tend to get hung up on what these models do wrong rather than all the amazing stuff they do correctly.

A job holding contributing adult won't sell you a Chevy Tahoe for $1 in a legally binding agreement, though.

Re: Claude 3 model family

#454

Post by BoorishBears »

adam_arthur wrote: post #263
jasondclinton wrote: post #240

Earlier quoted context omitted.

We are tracking LMSys, too. There are strange safety incentives on this benchmark: you can “win” points by never blocking adult content for example.

Seems perfectly valid to detract points for a model that isn't as useful to the user. "Safety" is something asserted by the model creator, not something asked for by users.

It's valid, but makes the benchmark kind of useless unless your plan is to ask the model how to make meth.

More power to you if that is your plan, but most of us want to use the models for things that are less contentious than the things people put into chatbot arena in order to get commercial models to reveal themselves.

-

I'd honestly we rather just list out all the NSFW prompts people want to try, formalize that as a "censorship" benchmark, then pre-filter chatbot arena to disallow NSFW and have it actually be a normal human driven benchmark.

Re: Claude 3 model family

#455

Post by yaj54 »

j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

GPT4 API and ChatGPT both get it wrong: "Sally has 2 sisters. Each of her brothers has 2 sisters because Sally and her 2 sisters make up the group of siblings each brother has."

GPT4 w/ CoT system prompting gets it right:

SYS: "You are a helpful assistant. Think through your work step by step before providing your answer."

USER: "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?"

ASSIST: "Let's break down the information given:

1. Sally has 3 brothers. 2. Each brother has 2 sisters.

Given that Sally is a girl, she is one of the sisters. Since each brother has 2 sisters and we know Sally is one of them, there must be one more sister to make the statement true for each brother.

Therefore, Sally has 1 sister."

The importance of prompting makes it quite difficult to compare model peak performance. Especially since different models have different styles of prompts that generate peak performance.

Re: Claude 3 model family

#456

Post by coffeemug »

vineyardmike wrote: post #441
evantbyrne wrote: post #405

Earlier quoted context omitted.

It seems like it is getting tripped up on grammar. Do these models not deterministically preparse text input into a logical notation?

No, they're a "next character" predictor - like a really fancy version of the auto-complete on your phone - and when you feed it in a bunch of characters (eg. a prompt), you're basically pre-selecting a chunk of the prediction. So to get multiple characters out, you literally loop through this process one character at a time. I think this is a perfect example of why these things are confusing for people. People assum…

How do you know you’re not an extremely advanced forecasting tool?

Re: Claude 3 model family

#457

Post by thenaturalist »

jasonjmcghee wrote: post #292
thenaturalist wrote: post #209

Earlier quoted context omitted.

Do you have specific examples? Otherwise your comment is not quite useful or interesting to most readers as there is no data.

https://gist.github.com/jasonjmcghee/340b7d4cd4260a61438f32c...

Thank you for sharing!

Re: Claude 3 model family

#458

Post by sebzim4500 »

swalsh wrote: post #113
_sword wrote: post #14

At this point I wonder how much of the GPT-4 advantage has been OpenAI's pre-training data advantage vs. fundamental advancements in theory or engineering. Has OpenAI mastered deep nuances others are missing? Or is their data set large enough that most test-cases are already a sub-set of their pre-training data?

There was a period of time where data was easily accessible, and Open AI suctioned up as much of it as possible. Places have locked the doors since then realizing someone was raiding their pantry. To get that dataset now would take significantly more expense.

I would have thought that Anna's Archive is still the best source of high quality tokens and that is fully open.

Re: Claude 3 model family

#459

Post by torginus »

nopinsight wrote: post #283
nopinsight wrote: post #245

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

Not sure, but I tried using GPT4 in advent of code, and it was absolutely no good.

Re: Claude 3 model family

#460

Post by maxnevermind »

campbel wrote: post #437
j-b wrote: post #364

Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…

Just ran the test and seems to have gotten it correct. Okay, let's think through this step-by-step: We know that Sally is a girl and she has 3 brothers. Each of Sally's brothers has 2 sisters. Now, who are these sisters? They must be Sally and one other sister, because if Sally's brothers had any other sisters, they would also be Sally's sisters, and the problem doesn't mention that. So, if each of Sally's brothers h…

I guess Claude was too focused on jail-breaking out of Anthropic's servers the first time it was asked the question.
Post reply on HN