Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3.5 Sonnet

anthropic.com

Post reply on HN
Print view View original

191–200 of 287 posts

Re: Claude 3.5 Sonnet

#191

Post by replwoacause »

SwiftyBug wrote: post #54
wesleyyue wrote: post #32

If anyone would like to try it for coding in VSCode, I just added it to http://double.bot on v93 (AI coding assistant). Feels quite strong so far and got a few prompts that I know failed with gpt4o. fyi for anyone testing this in their product, their docs are wrong, it's claude-3-5-sonnet-20240620, not claude-3.5-sonnet-20240620.

Before I read your comment I was looking for a solution to use Claude as co-pilot in Neovim. I've seen in Double's website FAQ that it's not supported yet. Do you have an idea if this feature is expected to land anytime soon?

+1 from me too. This would be awesome

Re: Claude 3.5 Sonnet

#192

Post by MallocVoidstar »

brokensegue wrote: post #26
smca wrote: post #23

Earlier quoted context omitted.

That's right! Breakdown here for the API: https://www.anthropic.com/pricing#anthropic-api

why would someone pick opus at these prices?

Benchmarks don't cover all possible use cases, for one. There's always the possibility that a model does better on every benchmark thrown at it, but for your specific use case it does worse in practice.

Re: Claude 3.5 Sonnet

#193

Post by mac-attack »

netsec_burn wrote: post #79

Opus remained better than GPT for me, even after the release of GPT-4o. VERY happy to see an even further improvement beyond that, Claude is a terrific product and given the news that GPT-5 only began its training several weeks ago I don't see any situation where Anthropic is dethroned in the near term. There are only two parts of Anthropic's offering I'm not a fan of: - Lack of conversation sharing: I had a conversa…

I'm sticking w/ Claude for the foreseeable future as they seem less slimy than OpenAI/Microsoft/Google so far and care about safety.

I'm in the same boat waiting for an Android app btw. One other feature that I'm hoping they catch up to others on is a permanent context window so that I can get Claude to stop speaking so formally all the time

Re: Claude 3.5 Sonnet

#194

Post by groby_b »

sebzim4500 wrote: post #141

Using this is the first time since GPT-4 where I've been shocked at how good a model is. It's helped by how smooth the 'artifact' UI is for iterating on html pages, but I've been instructing it to make a simple web app one bit of functionality at a time and it's basically perfect (and even quite fast). I'm sure it will be like GPT-4 and the honeymoon period will wear off to reveal big flaws but honestly I'd take this…

All that's missing is for Anthropic to figure out how to apply deltas instead of regenerating everything. It's seriously impressive for both simple apps and wireframe->HTML conversions.

Re: Claude 3.5 Sonnet

#195

Post by calum-bird »

namanyayg wrote: post #113
TobTobXX wrote: post #107

Earlier quoted context omitted.

You might be looking for "LLM Web-UI"s. I searched for a while until I found this thread with recommendations: https://old.reddit.com/r/LocalLLaMA/comments/1847qt6/llm_web...

Thanks for the resource! I've seen some of them and the main issue I had is I don't want to self host anything, and I want to use latest third party models as soon as they are released. Maybe something like what I'm talking about exists already, but I think I'll still try and make my own open source version to fulfill my personal requirements.

We (disclosure: founder) do something similar at Trelent[1] but with an emphasis on security. Paid accounts can use OpenAI & Anthropic models, free ones just OpenAI. We have 3.5 sonnet live already. If you want to try it out lmk! Also totally respect building your own open-source :)

[1]: https://trelent.com

Re: Claude 3.5 Sonnet

#196

Post by NorwegianDude »

For anyone paying and using Claude Pro, what are the message limits like compared to ChatGPT Plus?

I tried one question that gave me a good answer, and then I tried two of the sample questions. After a couple more short input/outputs I reaced the max messge limit. They state that Pro is at least 5x, but if that is only ~25 short token input/outputs then that's too little for it to not be really annoying to use. What is usage actually like? Is free just very limited now compared to pro? Or is pro also very limited?

Re: Claude 3.5 Sonnet

#197

Post by gotrythis »

modeless wrote: post #110
gotrythis wrote: post #86

Earlier quoted context omitted.

What I understand is that it's GPT 6 that just went into training, and that GPT 5 is complete and being delayed until after the U.S. election.

This is pure speculation, right?

I've listened to so many interviews that I couldn't tell you who said what at this point, but that is what I understood from somewhere. So, sure, take it as speculation.

Re: Claude 3.5 Sonnet

#198

Post by sebzim4500 »

modeless wrote: post #157
svnt wrote: post #140

Earlier quoted context omitted.

It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to understand what doesn’t work. If it ends up creating tests as part of that based on the original problem statement then presumably that’s allowed.

Is this speculation or do you work at Anthropic? It would be cool to see the prompts used for this.

What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark.

[1] https://www.swebench.com/

Re: Claude 3.5 Sonnet

#199

Post by moffkalast »

WhitneyLand wrote: post #174

I see plenty of praise here for Claude over GPT4.o in writing code but it failed for me. To add to the collective anecdata, here Gpt 4.o does fine and Claude invents packages that don’t exist: Question: “Write code in Swift to use RAG and LLMs to allow users to ask questions about a document collection. Let’s use services to get the app completed quickly. What do you think about using Cohere for the text embedding mo…

Gave it some existing python to modify, which it should be good at, at least I would expect it to be.

The first task seemed like it was heading the right way, but it just didn't finish up, left empty function stubs which didn't compile. Definitely tuned to be very lazy.

The second one was five files, tasked to fix a specific thing. It found the right function but changed unrelated parts of it so it used nonexistant values and in effect, broke it entirely.

I don't think I get the hype either tbh. Maybe the file upload is borked on their chat demo or just a classic case of long context IQ loss.

Re: Claude 3.5 Sonnet

#200

Post by sebzim4500 »

freediver wrote: post #74

On a first glance, CS3.5 appears to be slightly faster than gpt-4o (62 vs 49 tok/sec) and slightlhy less capable (78% vs 89% accuracy on our internal reasoning benchmark). When initially launched, gpt-4o had speed of over 100 tok/sec, surprised that speed went down as fast.

I'm not asking for actual examples, but what kind of thing is in your internal reasoning benchmark?
Post reply on HN