Ask HN: What is your experience with Claude 3.5? • Hacker News Board

Re: Ask HN: What is your experience with Claude 3.5?

#21

Post by CSMastermind » Tue, Jul 02, 2024, 3:58 PM UTC

I run all my queries through all of the SOTA models so that I can do a direct comparison. Currently, I've run over 600 queries through all the models. Claude has a clearly better answer than GPT-4 around 6% of the time and GPT-4 has a clearly better answer than Claude around 4% of the time.

Gemini is laughably behind the others though slightly better than Meta's current offering.

I find GPT-4 and Claude roughly on par with a slight edge to Claude. But when it fails, it does so catastrophically, whereas GPT-4 normally gives you at least something approaching a reasonable answer.

Another clear difference between them is that GPT-4 has a built-in code interpreter, and Claude doesn't. Some of the questions that GPT-4 gets correct that Claude doesn't are because GPT-4 can actually write code, run it, analyze the results, and self correct. Sometimes to a very impressive degree.

Re: Ask HN: What is your experience with Claude 3.5?

#22

Post by wenc » Tue, Jul 02, 2024, 4:28 PM UTC

causal wrote: post #9

If you haven't yet, pay for the upgrade to use Projects. I've uploaded entire codebases, then opened different conversations about that codebase, generating very useful artifacts. Also, Claude artifacts can display MermaidJS. You can give it a whole codebase and ask something like "generate a sequence diagram of this process" and it does a remarkable job of doing so, displaying results in realtime.

Claude doesn’t render LaTeX like ChatGPT though which is a huge miss for those of us who use LLMs for math stuff.

Re: Ask HN: What is your experience with Claude 3.5?

#23

Post by glimshe » Tue, Jul 02, 2024, 5:03 PM UTC

It depends. I use the free models and most tasks are better on 4o, but I fairly often get better results with Claude. I often try the same task on both when I'm not satisfied with the first answer I get.

Re: Ask HN: What is your experience with Claude 3.5?

#24

Post by runjake » Tue, Jul 02, 2024, 8:16 PM UTC

makk wrote: post #15

I haven’t been blown away by its coding as compared to 4o. It feels similar. I use it via the chat interface and copy/paste. I’d prefer to use it directly in IntelliJ but haven’t found a plugin that doesn’t get in my way to the point of being too distracting. The part of the Claude UI that writes code in a sidebar next to the chat is much better than writing code inline like ChatGPT.

  > The part of the Claude UI that writes code in a sidebar next to the chat is much better than writing code inline like ChatGPT.

How do I enable this? I am new to Claude, got a Pro account, but I only get inline code and can't find an option to change this.

Re: Ask HN: What is your experience with Claude 3.5?

#25

Post by beaugunderson » Tue, Jul 02, 2024, 8:56 PM UTC

i'll keep trying new models as they're updated but 3.5 sonnet is hot garbage. not worth the power to train and run it.

Re: Ask HN: What is your experience with Claude 3.5?

#26

Post by cvjcvjcvj » Tue, Jul 02, 2024, 9:30 PM UTC

marvstazar wrote: post #16

How does it compare to Github Copilot?

Claude is better in every aspect. Better yet if you use with Cursor editor.

Re: Ask HN: What is your experience with Claude 3.5?

#27

Post by makk » Wed, Jul 03, 2024, 2:57 AM UTC

runjake wrote: post #24

makk wrote: post #15

I haven’t been blown away by its coding as compared to 4o. It feels similar. I use it via the chat interface and copy/paste. I’d prefer to use it directly in IntelliJ but haven’t found a plugin that doesn’t get in my way to the point of being too distracting. The part of the Claude UI that writes code in a sidebar next to the chat is much better than writing code inline like ChatGPT.

> The part of the Claude UI that writes code in a sidebar next to the chat is much better than writing code inline like ChatGPT. How do I enable this? I am new to Claude, got a Pro account, but I only get inline code and can't find an option to change this.

Under the accounts menu in the bottom left, open Feature Preview and enable Artifacts.

Re: Ask HN: What is your experience with Claude 3.5?

#28

Post by makk » Wed, Jul 03, 2024, 3:07 AM UTC

beaugunderson wrote: post #25

i'll keep trying new models as they're updated but 3.5 sonnet is hot garbage. not worth the power to train and run it.

That's interesting. Really curious about where it's failing for you, where other models are succeeding.

Re: Ask HN: What is your experience with Claude 3.5?

#29

Post by beaugunderson » Wed, Jul 03, 2024, 7:21 PM UTC

makk wrote: post #28

beaugunderson wrote: post #25

i'll keep trying new models as they're updated but 3.5 sonnet is hot garbage. not worth the power to train and run it.

That's interesting. Really curious about where it's failing for you, where other models are succeeding.

the most recent fail involved YAML anchors (the ampersand syntax). i had a vague idea that you might be able to extend an anchor elsewhere in the file (like "abc " + &anchor). claude dreamed up three ways to do it, all three were hallucinated nonsense. when i said they didn't work, i get the classic "YAML doesn't support string concatenation in the way I suggested. My previous examples were incorrect."

any time i've saved via claude or chatgpt is dwarfed by the time they have wasted.