Earlier quoted context omitted.
(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)
Sounds almost recursive.
[deleted]
Claude 3 model family
301–310 of 723 posts
Re: Claude 3 model family
#302Earlier quoted context omitted.
Hi Simon, Big fan of your work with the LLM tool. I have a cool use for it that I wanted to share with you (on mac). First, I created a quick action in Automator that recieves text. Then I put together this script with the help of ChaptGPT: escaped_args="" for arg in "$@"; do escaped_arg=$(printf '%s\n' "$arg" | sed "s/'/'\\\\''/g") escaped_args="$escaped_args '$escaped_arg'" done result=$(/Users/XXXX/Library/Python/…
That is a brilliant hack! Thanks for sharing. Any chance you could post a screenshot of the Automator workflow somewhere - I'm having trouble figuring out how to reproduce (my effort so far is here: https://gist.github.com/simonw/d3c07969a522226067b8fe099007f... )
[deleted]
Re: Claude 3 model family
#303This is great. I'm also building an LLM evaluation framework with all these benchmarks integrated in one place so anyone can go benchmark these new models on their local setup in under 10 lines of code. Hope someone finds this useful: https://github.com/confident-ai/deepeval
Re: Claude 3 model family
#304Europeans, don't bother signing up - it will not work and it will only tell you once it has your e-mail registered.
https://twitter.com/jackclarkSF/status/1764657500589277296
"The API is generally available in Europe today and we're working on extending http://Claude.ai access over the coming months as well"
Re: Claude 3 model family
#305Earlier quoted context omitted.
Hi Simon, Big fan of your work with the LLM tool. I have a cool use for it that I wanted to share with you (on mac). First, I created a quick action in Automator that recieves text. Then I put together this script with the help of ChaptGPT: escaped_args="" for arg in "$@"; do escaped_arg=$(printf '%s\n' "$arg" | sed "s/'/'\\\\''/g") escaped_args="$escaped_args '$escaped_arg'" done result=$(/Users/XXXX/Library/Python/…
That is a brilliant hack! Thanks for sharing. Any chance you could post a screenshot of the Automator workflow somewhere - I'm having trouble figuring out how to reproduce (my effort so far is here: https://gist.github.com/simonw/d3c07969a522226067b8fe099007f... )
I added some notes to the gist.
Re: Claude 3 model family
#306The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…
“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…
What's to say this isn't just a demonstration of memorization capabilities? For example, rephrasing the logic of the question or even just simple randomizing the order of the multiple choice answers to these questions often dramatically impacts performance. For example, every model in the Claude 3 family repeats the memorized solution to the lion, goat, wolf riddle regardless of how I modify the riddle.
Re: Claude 3 model family
#307Just added Claude 3 to Chat at https://double.bot if anyone wants to try it for coding. Free for now and will push Claude 3 for autocomplete later this afternoon. From my early tests this seems like the first API alternative to GPT4. Huge!
more early impressions on performance: besides the endpoint erroring out at a higher rate than openai, time-to-first-token is also much slower :(
p50: 2.14s p95: 3.02s
And these aren't super long prompts either. vs gpt4 ttft:
p50: 0.63s p95: 1.47s
Re: Claude 3 model family
#308Earlier quoted context omitted.
They are funded mostly by Microsoft, and dependent on them for compute (which is what this funding is mostly buying), but I'd hardly characterize that as meaning they are "Microsoft in a trenchcoat". It's not normal to identify startups as being their "VC in a trenchcoat", even if they are dependent on the money for growth.
Satya Nadella during the OpenAI leadership fiasco: “We have all of the rights to continue the innovation, not just to serve the product, but we can, you know, go and just do what we were doing in partnership ourselves. And so we have the people, we have the compute, we have the data, we have everything.” Doesn’t sound like a startup-investor relationship to me!
Sure, but that's just saying that Microsoft as investor has some rights to the underlying tech. There are limits to this though, which we may fairly soon be nearing. I believe the agreement says that Microsoft's rights to the tech (model + weights? training data? -- not sure how specific it is) end once AGI is achieved, however that is evaluated.
But again, this is not to say that OpenAI is "Microsoft in a trenchcoat". Microsoft don't have developers at OpenAI, weren't behind the tech in any way, etc. Their $10B investment bought them some short-term insurance in limited rights to the tech. It is what is is.
Re: Claude 3 model family
#309Earlier quoted context omitted.
> We now have the compute, the data, and the architecture (transformer) to do it. It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. Conversely, humans getting training on the same topics achieve very similar results, even though brains are very different at low level, not even the same number of neurons, not to mention different…
> It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. My take is that prediction, however you do it, is the essence of intelligence. In fact, I'd define intelligence as the degree of ability to correctly predict future outcomes based on prior experience. The ultimate intelligent architecture, for now, is our own cortex, which can…
If you invented the transformer but didn't have trillions of tokens to train it with, no chatGPT. But if you had Mamba/RWKV/SSSM and trillions of tokens you would have almost the same thing with chatGPT.
The training set is magical. It took humanity a long time to discover all the nifty ideas we have in it. It's the result of many generations of humans working together, using language to share their experience. Intelligence is a social process, even though we like to think about keys and queries, or synapses and neurotransmitters, in fact it is the work of many people that made it possible.
And language is that central medium between all of us, an evolutionary system of ideas, evolving at a much faster rate than biology. Now AI have become language replicators like humans, a new era in the history of language has begun. The same language trains humans and LLMs to achieve similar sets of abilities.
Re: Claude 3 model family
#310Earlier quoted context omitted.
Anthropic is also not really a traditional startup. It’s just some large companies in a trench coat.
How so? Because they have taken large investments from Amazon and Google? Or would you also characterize OpenAI as "Microsoft in a trench coat"?
Absolutely to OpenAI being Microsoft in a trench coat.
This is not an uncommon tactic for companies to use.