Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

291–300 of 723 posts

Re: Claude 3 model family

#291

Post by [deleted] »

AaronFriel wrote: post #246
paradite wrote: post #160

I just tried one prompt for a simple coding task involving DB and frontend, and Claude 3 Sonnet (the free and less powerful model) gave a better response than ChatGPT Classic (GPT-4). It used the correct method of a lesser-known SQL ORM library, where GPT-4 made a mistake and used the wrong method. Then I tried another prompt to generate SQL and it gave a worse response than ChatGPT Classic, still looks correct but m…

Are you aware you're using GPT-3 or weaker in those chats? The green icon indicates that you're using the first generation of ChatGPT models, and it is likely to be GPT-3.5 Turbo. I'm unsure but it's possible that it's an even further distilled or quantized optimization than is available via API. Using GPT-4, I get the result I think you'd expect: https://chat.openai.com/share/da15f295-9c65-4aaf-9523-601bf4... This i…

[deleted]

Re: Claude 3 model family

#292

Post by jasonjmcghee »

thenaturalist wrote: post #209
jasonjmcghee wrote: post #164

I've tried all the top models. GPT4 beats everything I've tried, including Gemini 1.5- until today. I use GPT4 daily on a variety of things. Claude 3 Opus (been using temperature 0.7) is cleaning up. I'm very impressed.

Do you have specific examples? Otherwise your comment is not quite useful or interesting to most readers as there is no data.

https://gist.github.com/jasonjmcghee/340b7d4cd4260a61438f32c...

Re: Claude 3 model family

#293

Post by hackerlight »

nine_k wrote: post #100
anonymouse008 wrote: post #87

Earlier quoted context omitted.

I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE

I personally find approaches like this the correct way forward. An input analyzer that finds out what kinds of tokens the query contains. A bunch of specialized models which handle each type well: image analysis, OCR, math and formal logic, data lookup,sentiment analysis, etc. Then some synthesis steps that produce a coherent answer in the right format.

Doesn't the human brain work like this? Yeah it's all connected together and plastic and so on, but functions tend to be localized, e.g vision is in occipital area. These base areas are responsible for the basic latent representations (edge detectors) which get fed forward to the AGI module (prefrontal cortex) that coordinates the whole thing based on the high quality representations it sees from these base modules.

This strikes me as the most compute efficient approach.

Re: Claude 3 model family

#294

Post by Havoc »

Havoc wrote: post #71
hubraumhugo wrote: post #66

It feels absolutely amazing to build an AI startup right now: - We struggled with limited context windows [solved] - We had issues with consistent JSON output [solved] - We had rate limiting and performance issues with 3rd party models [solved] - Hosting OSS models was a pain [solved] It's like your product becomes automatically cheaper, more reliable, and more scalable with every major LLM advancement. I'm going to…

What was the solution on Jain? Gbnf grammars?

JSON not Jain sigh autocorrect

Re: Claude 3 model family

#295

Post by vok »

nopinsight wrote: post #245

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…

APPS has 3 subsets by difficulty level: introductory, interview, and competition. It isn't clear which subset Claude 3 was benchmarked on. Even if it is just "introductory" it is still pretty good, but it would be good to know.

Re: Claude 3 model family

#296

Post by declaredapple »

brookst wrote: post #163
flawn wrote: post #52

Earlier quoted context omitted.

Either something like that or just because the model's output is basically the best you can get and they utilize their market position. Probably that and what you mentioned.

This. Price is set by value delivered and what the market will pay for whatever capacity they have; it’s not a cost + X% market.

I'm more curious about the input/output token discrepancy

Their pricing suggests that either output tokens are more expensive for some technical reason, or they're trying to encourage a specific type of usage pattern, etc.

Re: Claude 3 model family

#298

Post by coffeebeqn »

bufferoverflow wrote: post #280
Workaccount2 wrote: post #125

Earlier quoted context omitted.

I'd almost say anyone not doing that is being foolish. The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all. I think some engineers feel that if they are leaning on an old school classically programed tool to assist the LLM, it's somehow cheating or impure.

> The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all. No, that's the actual end goal. We want a NN that does everything, trained end-to-end.

Like a AGI? I think we’ll put up with hacks for some more time still. Unless the model gets really really good at generalizing and then it’s probably close to human level already

Re: Claude 3 model family

#299

Post by moffkalast »

monkeydust wrote: post #206
SirensOfTitan wrote: post #173

What is the probability that newer models are just overfitting various benchmarks? A lot of these newer models seem to underperform GPT-4 in most of my daily queries, but I'm obviously swimming in the world of anecdata.

High. The only benchmark I look at is LMSys Chatbot Arena. Lets see how it perform on that https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Opus and Sonnet seem to be already available for direct chat on the arena interface.

Re: Claude 3 model family

#300

Post by 098799 »

bkrausz wrote: post #270
098799 wrote: post #192

Earlier quoted context omitted.

Actually, I also noticed 400 to consumer_pricing with response "Invalid country" even though I'm in Switzerland, which should be supported?

Claude.ai is not currently available in the EU...we should have prevented you from signing up in the first place though (unless you're using a VPN...) Sorry about that, we really want to expand availability and are working to do so.

Switzerland is not in the EU. Didn't use VPN.
Post reply on HN