Claude 3 model family • Hacker News Board

Re: Claude 3 model family

#181

Post by bvm » Mon, Mar 04, 2024, 3:53 PM UTC

hubraumhugo wrote: post #66

It feels absolutely amazing to build an AI startup right now: - We struggled with limited context windows [solved] - We had issues with consistent JSON output [solved] - We had rate limiting and performance issues with 3rd party models [solved] - Hosting OSS models was a pain [solved] It's like your product becomes automatically cheaper, more reliable, and more scalable with every major LLM advancement. I'm going to…

>- Hosting OSS models was a pain [solved]

what's the solution here? vllm?

Re: Claude 3 model family

#182

Post by lr1970 » Mon, Mar 04, 2024, 3:53 PM UTC

HarHarVeryFunny wrote: post #61

Workaccount2 wrote: post #20

Surpassing GPT4 is huge for any model, very impressive to pull off. But then again...GPT4 is a year old and OpenAI has not yet revealed their next-gen model.

Sure, OpenAI's next model would be expected to regain the lead, just due to their head start, but this level of catch-up from Anthropic is extremely impressive. Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models -…

> Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021.

Keep in mind that Antropic was founded by former OpenAI people (Dario Amadei and others). Both companies share a lot of R&D "DNA".

Re: Claude 3 model family

#183

Post by CuriouslyC » Mon, Mar 04, 2024, 3:53 PM UTC

nine_k wrote: post #100

anonymouse008 wrote: post #87

Earlier quoted context omitted.

I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE

I personally find approaches like this the correct way forward. An input analyzer that finds out what kinds of tokens the query contains. A bunch of specialized models which handle each type well: image analysis, OCR, math and formal logic, data lookup,sentiment analysis, etc. Then some synthesis steps that produce a coherent answer in the right format.

Yeah. Have a multimodal parser model that can decompose prompts into pieces, generate embeddings for each of them and route those embeddings to the correct model based on the location of the embedding in latent space. Then have a "combiner/resolver" model that is trained to take answer embeddings from multiple models and render it in one of a variety of human readable formats.

Eventually there is going to be a model catalog that describes model inputs/outputs in a machine parseable format, all models will use a unified interface (embedding in -> embedding out, with adapters for different latent spaces), and we will have "agent" models designed to be rapidly fine tuned in an online manner that act as glue between all these different models.

Re: Claude 3 model family

#184

Post by VWWHFSfQ » Mon, Mar 04, 2024, 3:54 PM UTC

JacobiX wrote: post #166

spyder wrote: post #99

What's up with the weird list of the supported countries? It isn't available in most European countries (except for Ukraine and UK) but on the other hand lot of African counties are listed... https://www.anthropic.com/claude-ai-locations

Arbitrary region locking : for example supported in Algeria and not in the neighboring Tunisia ... both are in North Africa

There's nothing arbitrary about it and both being located in North Africa means nothing. Tunisia has somewhat strict personal data protection laws and Algeria doesn't. That's the difference.

Re: Claude 3 model family

#185

Post by samstave » Mon, Mar 04, 2024, 3:55 PM UTC

virgildotcodes wrote: post #174

jasondclinton wrote: post #135

Earlier quoted context omitted.

Hi, CISO of Anthropic here. Thank you for the feedback! If you can share any details about the image, please share in a private message. No LLM has had an emergent calculator yet.

Hey Jason, checked your HN bio and I don't see a contact. Found you on twitter but it seems I'm unable to DM you. Went ahead and uploaded the image here: https://imgur.com/pJlzk6z

An "LLM crawler app" is needed -- in that you should be able to shift Tokenized Workloads between executioners in a BGP routing sort of sense...

Least cost routing of prompt response. especially if time-to-respond is not as important as precision...

Also, is there a time-series ability in any LLM model (meaning "show me this [thing] based on this [input] but continually updated as I firehose the crap out of it"?

--

What if you could get execution estimates for a prompt?

Re: Claude 3 model family

#186

Post by Cheezemansam » Mon, Mar 04, 2024, 3:55 PM UTC

Claude.ai web version is beyond useless, it is an actual scam. Like straight up it is not ethical for them to treat their web client as a product they are allowed to charge money for, the filters will actually refuse to do anything. You pay for increased messages and whatever but all you get is "I apologize..." and treats you as if you were about to commit mass genocide with calling 21+ year old individuals minors and any references to any disability as "reinforcing harmful stereotypes". You often cannot get it to summarize a generally innocuous statement.

Claude will only function through the API properly.

Re: Claude 3 model family

#187

Post by wesleyyue » Mon, Mar 04, 2024, 3:55 PM UTC

brainless wrote: post #130

wesleyyue wrote: post #62

Just added Claude 3 to Chat at https://double.bot if anyone wants to try it for coding. Free for now and will push Claude 3 for autocomplete later this afternoon. From my early tests this seems like the first API alternative to GPT4. Huge!

Hey Wesley, I just checked Double. Do you plan to support open source models hosted locally or on a cloud instance? Asking out of curiosity as I am building a product in the same space and have had a few people ask this. I guess since Double is an extension in IDEs, it can connect to AI models running anywhere.

it's an interesting idea. We asked our users this as well but at least for those we talked to, running their own model wasn't a big priority. What actually mattered to them is being able to try different (but high performance) models, privacy (their code not being trained on), and latency. We have some optimizations around time-to-first-token latency that would be difficult to do if we didn't have information about the model and their servers.

Re: Claude 3 model family

#188

Post by behnamoh » Mon, Mar 04, 2024, 3:56 PM UTC

wesleyyue wrote: post #177

behnamoh wrote: post #175

Earlier quoted context omitted.

To be clear: Is this Claude 3 Opus or the Sonnet model?

opus. only the best!

Awesome! I like the inline completions.

But could you let the users choose their keyboard shortcuts before setting the default ones?

Re: Claude 3 model family

#189

Post by simonw » Mon, Mar 04, 2024, 3:56 PM UTC

I just released a plugin for my LLM command-line tool that adds support for the new Claude 3 models:

    pipx install llm
    llm install llm-claude-3
    llm keys set claude
    # paste Anthropic API key here
    llm -m claude-3-opus '3 fun facts about pelicans'
    llm -m claude-3-opus '3 surprising facts about walruses'

Code here: https://github.com/simonw/llm-claude-3

Re: Claude 3 model family

#190

Post by bugglebeetle » Mon, Mar 04, 2024, 3:56 PM UTC

rafaelero wrote: post #172

bugglebeetle wrote: post #83

Earlier quoted context omitted.

MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.

MMLU is garbage. A lot of incorrect answers there.

And yet it’s still a good indicator of general performance. Any model that scores under GPT-4 on that benchmark, but above it in other, tends to be worse overall.