Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

261–270 of 723 posts

Re: Claude 3 model family

#261

Post by helsinki »

samstave wrote: post #178
virgildotcodes wrote: post #68

Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100. It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.…

I cant wait until this is the true disruptor in the economy: " Take this $1,000 and maximise my returns and invest it where appropriate. Goal is to make this $1,000 100X " And just let your r/wallStreetBets BOT run rampant with it...

That will only work for the first few people who try it.

Re: Claude 3 model family

#262

Post by lanstin »

brokencode wrote: post #251
ignoramous wrote: post #232

Earlier quoted context omitted.

> I'd almost say anyone not doing that is being foolish The problem is, such tricks are sold as if there's superior built-in multi-modal reasoning and intelligence instead of taped up heuristics, exacerbating the already amped up hype cycle in the vacuum left behind by web3.

Why is this a trick or somehow inferior to getting the AI model to be able to do it natively? Most humans also can’t reliably do complex arithmetic without the use of something like a calculator. And that’s no trick. We’ve built the modern world with such tools. Why should we fault AI for doing what we do? To me, training the AI use a calculator is not just a trick for hype, it’s exciting progress.

It would be exciting if the LLM knew it needed a calculator for certain things and went out and got it. If the human supervisors are pre-screening the input and massaging what the LLM is doing that is a sign we don't understand LLMs enough to engineer them precisely and can't count on them to be aware of their own limitations, which would seem to be a useful part of general intelligence.

Re: Claude 3 model family

#263

Post by adam_arthur »

jasondclinton wrote: post #240
monkeydust wrote: post #206

Earlier quoted context omitted.

High. The only benchmark I look at is LMSys Chatbot Arena. Lets see how it perform on that https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

We are tracking LMSys, too. There are strange safety incentives on this benchmark: you can “win” points by never blocking adult content for example.

Seems perfectly valid to detract points for a model that isn't as useful to the user.

"Safety" is something asserted by the model creator, not something asked for by users.

Re: Claude 3 model family

#264

Post by paradite »

AaronFriel wrote: post #246
paradite wrote: post #160

I just tried one prompt for a simple coding task involving DB and frontend, and Claude 3 Sonnet (the free and less powerful model) gave a better response than ChatGPT Classic (GPT-4). It used the correct method of a lesser-known SQL ORM library, where GPT-4 made a mistake and used the wrong method. Then I tried another prompt to generate SQL and it gave a worse response than ChatGPT Classic, still looks correct but m…

Are you aware you're using GPT-3 or weaker in those chats? The green icon indicates that you're using the first generation of ChatGPT models, and it is likely to be GPT-3.5 Turbo. I'm unsure but it's possible that it's an even further distilled or quantized optimization than is available via API. Using GPT-4, I get the result I think you'd expect: https://chat.openai.com/share/da15f295-9c65-4aaf-9523-601bf4... This i…

I see the purple icon with "ChatGPT Classic" on my share link, but if I open it in incognito without login, it shows as green "ChatGPT". You can try opening in incognito your own chat share link.

I use ChatGPT Classic, which is an official GPT from OpenAI without the extra system prompt from normal ChatGPT.

https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic

It is explicitly mentioned in the GPT that it uses GPT-4. Also, it does have purple icon in the chat UI.

I have observed an improved quality of using it compared for GPT-4 (ChatGPT Plus). You can read about it more in my blog post:

https://16x.engineer/2024/02/03/chatgpt-coding-best-practice...

Re: Claude 3 model family

#265

Post by msp26 »

hobofan wrote: post #157
msp26 wrote: post #132

Earlier quoted context omitted.

> I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact same knowledge base seems very inefficient. If you don't care about latency or can wait to set up a batch of inputs in one go there's an alternative method. I call it batch prompting and pretty much everything we do at work with gpt-4 uses this now. If people are interested I'll do a proper writeup on how to i…

I attempted similar mechanics multiple times in the past, but always ditched them, as there was always a non-negligable amount of cross-contamination happening between the individual instances you are batching. That caused so much of a headache that it wasn't really worth it.

Yeah that's definitely a risk with language models but it doesn't seem to be too bad for my use cases. Can I ask what tasks you used it for?

I don't really intend for this method to be final. I'll switch everything over to finetunes at some point. But this works way better than I would have expected so I kept using it.

Re: Claude 3 model family

#266

Post by nprateem »

SirensOfTitan wrote: post #173

What is the probability that newer models are just overfitting various benchmarks? A lot of these newer models seem to underperform GPT-4 in most of my daily queries, but I'm obviously swimming in the world of anecdata.

The fact it beats other benchmarks consistently by 0.1% tells me everything I need to know.

Re: Claude 3 model family

#267

Post by Der_Einzige »

anonymouse008 wrote: post #87
virgildotcodes wrote: post #68

Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100. It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.…

I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE

I wrote a whole paper about ways to "fix" tokenization in a plug-and-play fashion for poetry generation: Filter the vocabulary before decoding.

https://paperswithcode.com/paper/most-language-models-can-be...

Re: Claude 3 model family

#268

Post by gkfasdfasdf »

wesleyyue wrote: post #62

Just added Claude 3 to Chat at https://double.bot if anyone wants to try it for coding. Free for now and will push Claude 3 for autocomplete later this afternoon. From my early tests this seems like the first API alternative to GPT4. Huge!

How do you guys compare to codium [0]? Also, any plans to support vim/neovim integration (codium has pretty good support in place [1]). Thanks.

[0] - https://www.codium.ai

[1] - https://github.com/Exafunction/codeium.vim

Re: Claude 3 model family

#269

Post by AaronFriel »

paradite wrote: post #264
AaronFriel wrote: post #246

Earlier quoted context omitted.

Are you aware you're using GPT-3 or weaker in those chats? The green icon indicates that you're using the first generation of ChatGPT models, and it is likely to be GPT-3.5 Turbo. I'm unsure but it's possible that it's an even further distilled or quantized optimization than is available via API. Using GPT-4, I get the result I think you'd expect: https://chat.openai.com/share/da15f295-9c65-4aaf-9523-601bf4... This i…

I see the purple icon with "ChatGPT Classic" on my share link, but if I open it in incognito without login, it shows as green "ChatGPT". You can try opening in incognito your own chat share link. I use ChatGPT Classic, which is an official GPT from OpenAI without the extra system prompt from normal ChatGPT. https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic It is explicitly mentioned in the GPT that it uses GPT-4.…

Oh, I see. That must be frustrating to folks at OpenAI. Their product rests on the quality of their models, and making users unable to see which results came from their best doesn't help.

FWIW, GPT-4 and GPT-4 Turbo via developer API call both seem to produce the result you expect.

Re: Claude 3 model family

#270

Post by bkrausz »

098799 wrote: post #192
098799 wrote: post #165

Trying to subscribe to pro but website keeps loading (404 to stripe's /invoices is the only non 2xx I see)

Actually, I also noticed 400 to consumer_pricing with response "Invalid country" even though I'm in Switzerland, which should be supported?

Claude.ai is not currently available in the EU...we should have prevented you from signing up in the first place though (unless you're using a VPN...)

Sorry about that, we really want to expand availability and are working to do so.

Post reply on HN