Bizarrely still unavailable in Canada. Even Google has stepped back the strange Canada-reluctance, I can get any version of Gemini since the last release. But somehow Anthropic can't figure out how to get Canada onto the list of ~88 supported countries: https://www.anthropic.com/claude-ai-locations
Claude 3 model family
381–390 of 723 posts
Re: Claude 3 model family
#382From the Model Card on Needle In A Haystack evaluation > One aspect that has caught our attention while examining samples from Claude 3 Opus is that, in certain instances, the model demonstrates a remarkable ability to identify the synthetic nature of the task, and acknowledges that the needle was most likely not part of the original document. As model capabilities continue to advance, it is crucial to bear in mind t…
When the "stochastic parrot" sees through your tricks...
Re: Claude 3 model family
#383Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
This is why I doubt all the AI hype. These things are supposed to have PhD level smarts, but the above example can't reason about the problem well at all. There's a difference between PhD level information and advanced reasoning, and I'm not sure how many people can tell the difference (I'm no expert).
In an adjacent area - autonomous driving - I know that lane following is f**ing easy, but lane identification and other object identification is hard. Having real understanding of a situation and acting accordingly is very complex. I wonder if people look at these cars doing the basics and assume they "understand" a lot more than they actually do. I ask the same about LLMs.
Re: Claude 3 model family
#384Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
But the question doesn’t mention if Sally has no sisters. But the statement “brothers have 2 sisters” makes me think she has 1 sister.
Re: Claude 3 model family
#385I suspect dataset contamination is at play here. It fails pretty basic maths questions (not arithmetic, that would be understandable) that surely it should be able to do in order to get its benchmark results on MATH.
EDIT: Also it hallucinates way more than GPT-4 does. It's possible this is due to a bad system prompt rather than a dumb model.
Re: Claude 3 model family
#386Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
mistralai/Mixtral-8x7B-Instruct-v0.1 got this right.
>Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Sally has 1 sister.
Here's the logic behind this:
1. We know that Sally is a girl and she has 3 brothers.
2. Then we are told that each of her brothers has 2 sisters.
3. Since all of Sally's brothers share the same siblings, they would both count Sally as one of their two sisters.
4. Therefore, Sally only has 1 sister because if each brother counts her once, there is no need for another sister to fulfill the "two sisters" condition.
Re: Claude 3 model family
#387Earlier quoted context omitted.
By all means if it works to solve your problem, go ahead and do it. The reason some people have mixed feelings about this because of a historical observation - http://www.incompleteideas.net/IncIdeas/BitterLesson.html - that we humans often feel good about adding lots of hand-coded smarts to our ML systems reflecting our deep and brilliant personal insights. But it turns out just chucking loads of data and compute at…
The reason we chucked loads of data at it was because we had no other options. If you wanted to write a function that classified a picture as a cat or a dog, good luck. With ML, you can learn such a function. That logic doesn’t extend to things we already know how to program computers to do. Arithmetic already works. We don’t need a neural net to also run the calculations or play a game of chess. We have specialized…
Not so fast - you might have precise and efficient functions that do things like basic arithmetic. What you might not have is a model that can reason mathematically. You need a model to do things like basic arithmetic functions so that semantic and arbitrary relations get encoded in the weights of a network.
You see this type of glitch crop up in tokenizing schemes in large language models. If you attempt working with character level reasoning or output construction, it will often fail. Trying to get ChatGPT 4 to output a sentence, and then that sentence backwards, or every other word spelled backwards, is almost impossible. If you instead prompt the model to produce an answer with a delimiter between every character, like #, also to replace spaces, it can resolve the problems much more often than with standard punctuation and spaces.
The idea applies to abstractions that aren't only individual tokens, but specific concepts and ideas that in turn serve as atomic components of higher abstractions.
In order to use those concepts successfully, the model has to be able to encode the thing and its relationships effectively in the context of whatever else it learns. For a given architecture, you could do the work and manually create the encoding scheme for something like arithmetic, and it could probably be very efficient and effective. What you miss is the potential for fuzzy overlaps in the long tail that only come about through the imperfect, bespoke encodings learned in the context of your chosen optimizer.
Re: Claude 3 model family
#388Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
mistralai/Mixtral-8x7B-Instruct-v0.1 got this right. >Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Sally has 1 sister. Here's the logic behind this: 1. We know that Sally is a girl and she has 3 brothers. 2. Then we are told that each of her brothers has 2 sisters. 3. Since all of Sally's brothers share the same siblings, they would both count Sally as one of their two…
So did claude 3.0 with temperaure 0, whilst with temp 1 it got it wrong.
Re: Claude 3 model family
#389I'm quite impressed with both the speed and the quality of the responses using the API. As I mentioned in the Phind-70B thread[1], this is a prompt I usually try with new LLMs:
> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.
GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind and other LLMs (never tried Gemini) fail as they forget about saving the body for POST requests. Claude Opus got it right every time I asked the question[2]; I wouldn't use the code it spit out without editing it, but it would be enough for me to learn the concepts and write a proper implementation.
It's a shame Claude.ai isn't available in Brazil, which I assume is because of our privacy laws, because this could easily go head to head with GPT-4 from my early tests.
[1] https://news.ycombinator.com/item?id=39473137
[2] https://paste.sr.ht/~jamesponddotco/011f4261a1de6ee922ffa5e4...
Re: Claude 3 model family
#390Opus (and all prior Claude models) still do not correctly answer the Sally question, unfortunately: Q: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? A: Okay, let's organize the information we have: Sally is a girl. Sally has 3 brothers. Each of Sally's brothers has 2 sisters. Now, let's think about this step by step: We know that Sally is a girl, so she is a sister to he…
Temperature 1 - It answered 1 sister: https://i.imgur.com/7gI1Vc9.png Temperature 0 - it answered 0 sisters: https://i.imgur.com/iPD8Wfp.png
By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature 0 is the correct evaluation.