Claude 3 model family

Post by apsec112 » Mon, Mar 04, 2024, 6:25 PM UTC

wbarber wrote: post #306

nopinsight wrote: post #283

Earlier quoted context omitted.

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

What's to say this isn't just a demonstration of memorization capabilities? For example, rephrasing the logic of the question or even just simple randomizing the order of the multiple choice answers to these questions often dramatically impacts performance. For example, every model in the Claude 3 family repeats the memorized solution to the lion, goat, wolf riddle regardless of how I modify the riddle.

Post by HarHarVeryFunny » Mon, Mar 04, 2024, 6:28 PM UTC

visarga wrote: post #309

HarHarVeryFunny wrote: post #289

Earlier quoted context omitted.

> It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. My take is that prediction, however you do it, is the essence of intelligence. In fact, I'd define intelligence as the degree of ability to correctly predict future outcomes based on prior experience. The ultimate intelligent architecture, for now, is our own cortex, which can…

If you invented the transformer but didn't have trillions of tokens to train it with, no chatGPT. But if you had Mamba/RWKV/SSSM and trillions of tokens you would have almost the same thing with chatGPT. The training set is magical. It took humanity a long time to discover all the nifty ideas we have in it. It's the result of many generations of humans working together, using language to share their experience. Intel…

Post by simonw » Mon, Mar 04, 2024, 6:30 PM UTC

simonw wrote: post #189

I just released a plugin for my LLM command-line tool that adds support for the new Claude 3 models: pipx install llm llm install llm-claude-3 llm keys set claude # paste Anthropic API key here llm -m claude-3-opus '3 fun facts about pelicans' llm -m claude-3-opus '3 surprising facts about walruses' Code here: https://github.com/simonw/llm-claude-3 More on LLM: https://llm.datasette.io/

    #!/bin/bash
    # Validate that the argument is an integer
    if [[ ! $1 =~ ^[0-9]+$ ]]; then
      echo "Please provide a valid integer as the argument."
      exit 1
    fi
    # Make API call, parse and summarize the discussion
    curl -s "https://hn.algolia.com/api/v1/items/$1" | \
      jq -r 'recurse(.children[]) | .author + ": " + .text' | \
      llm -m claude-3-opus -s 'Summarize the themes of the opinions expressed here.
      For each theme, output a markdown header.
      Include direct "quotations" (with author attribution) where appropriate.
      You MUST quote directly from users when crediting them, with double quotes.
      Fix HTML entities. Output markdown. Go long.'

    ./hn-summary.sh 39590666

Post by what_ever » Mon, Mar 04, 2024, 6:32 PM UTC

JacobiX wrote: post #201

VWWHFSfQ wrote: post #184

Earlier quoted context omitted.

There's nothing arbitrary about it and both being located in North Africa means nothing. Tunisia has somewhat strict personal data protection laws and Algeria doesn't. That's the difference.

I know both countries, and in Algeria the Law No. 18-07, effective since August 10, 2023, establishes personal data protection requirements with severe penalties. The text is somewhat more strict than Tunisia.

Post by p1esk » Mon, Mar 04, 2024, 6:34 PM UTC

lukev wrote: post #313

nopinsight wrote: post #283

Earlier quoted context omitted.

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

This doesn't pass the sniff test for me. Not sure if these models are memorizing the answers or something else, but it's simply not the case that they're as capable as a domain expert (yet.) I do not have a PhD, but in areas I do have expertise, you really don't have to push these models that hard to before they start to break down and emit incomplete or wrong analysis.