Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 2.1

anthropic.com

Post reply on HN
Print view View original

91–100 of 339 posts

Re: Claude 2.1

#91

Post by michaelt »

inciampati wrote: post #45
sebgr wrote: post #19

For coding it is still 10x worse than gpt4. I asked it to write a simple database sync function and it gives me tons of pseudocode like `//sync object with best practices`. When I ask it to give me real code it forgets tons of key aspects.

Except: you can feed it an entire programming language manual, all the docs for all the modules you want to use, and _then_ it's stunningly good, whipping chatgpt4 that same 10x.

I gather the pricing is $8 for a million input tokens [1] so if your language's manual is the size of a typical paperback novel, that'd be about $0.8 per question. And presumably you get to pay that if you ask any follow-up questions too.

Sounds like a kinda expensive way of doing things, to me.

[1] https://www-files.anthropic.com/production/images/model_pric...

Re: Claude 2.1

#92

Post by whimsicalism »

swatcoder wrote: post #67
sebgr wrote: post #19

For coding it is still 10x worse than gpt4. I asked it to write a simple database sync function and it gives me tons of pseudocode like `//sync object with best practices`. When I ask it to give me real code it forgets tons of key aspects.

Because they're ultimately training data simulators and not actually brilliant aritifical programmers, we can expect Microsoft-affiliated models like ChatGPT4 and beyond to have much stronger value for coding because they have unmediated access to GitHub content. So it's most useful to look at other capabilities and opportunities when evaluating LLM's with a different heritage. Not to say we shouldn't evaluate this o…

idk we're just "have more kids" simulators and we do pretty good at programming as a side-task

Re: Claude 2.1

#93

Post by phillipcarter »

whimsicalism wrote: post #86
phillipcarter wrote: post #78

Although we're not using Claude in production (yet), it's a regular part of our testing when we build new features with LLMs. Part of the reason why we haven't used it (yet) is because OpenAI had more certifications faster, so we went to market with them. And their API has just gotten better and more reliable ever since, and it's cheap. But now that Claude is in AWS Bedrock that opens up some things for us that were…

Makes sense as claude instant is likely better than 3.5

I dunno about that. GPT 3.5 is extremely good. I would wager that most apps that use RAG to pass context in and get JSON (or some other thing) out that you can pass to some other part of your product don't need GPT 4 or anything else equally as powerful.

Re: Claude 2.1

#94

Post by robga »

legendofbrando wrote: post #31

I would love to use their API but I can never get anyone to respond to me. It's like they have no real interest in being a developer platform. Has anyone gotten their vague application approved?

I applied a few months ago. Last week I received an email:

“We’re pleased to let you know that we’re expanding access to the Claude API.

As the next step in considering your application, we’ll need some further information from you. Please fill out our onboarding form.”

The form seems to be the same form I filled in months before. I’ve not heard back in the 7 days since.

Re: Claude 2.1

#95

Post by swatcoder »

whimsicalism wrote: post #92
swatcoder wrote: post #67

Earlier quoted context omitted.

Because they're ultimately training data simulators and not actually brilliant aritifical programmers, we can expect Microsoft-affiliated models like ChatGPT4 and beyond to have much stronger value for coding because they have unmediated access to GitHub content. So it's most useful to look at other capabilities and opportunities when evaluating LLM's with a different heritage. Not to say we shouldn't evaluate this o…

idk we're just "have more kids" simulators and we do pretty good at programming as a side-task

Sure, and those of us who have more robust preparation and expoure generally do a better job of it.

Re: Claude 2.1

#96

Post by minimaxir »

simonw wrote: post #84

The documentation for system prompts is setting off all of my prompt injection red flags: https://docs.anthropic.com/claude/docs/how-to-use-system-pro... I guess that design is at least honest: OpenAI field the system prompt in a separate fragment of JSON, but it all gets concatenated back together (with some magic delimiter tokens) when it's fed to the underlying model.

Relatedly, I checked and OpenAI deleted all references to their ChatML spec from their GitHub repo.

This is what it said in an earlier commit: https://github.com/openai/openai-python/blob/2942bf4bb635b1e...

Re: Claude 2.1

#97

Post by YetAnotherNick »

swatcoder wrote: post #67
sebgr wrote: post #19

For coding it is still 10x worse than gpt4. I asked it to write a simple database sync function and it gives me tons of pseudocode like `//sync object with best practices`. When I ask it to give me real code it forgets tons of key aspects.

Because they're ultimately training data simulators and not actually brilliant aritifical programmers, we can expect Microsoft-affiliated models like ChatGPT4 and beyond to have much stronger value for coding because they have unmediated access to GitHub content. So it's most useful to look at other capabilities and opportunities when evaluating LLM's with a different heritage. Not to say we shouldn't evaluate this o…

Github full (public) scrape is available to anyone. GPT-4 was trained before Microsoft deal so I don't think it is because of Github access. And GPT-4 is significantly better in everything compared to second best model for that field, not just coding.

Re: Claude 2.1

#98

Post by UrineSqueegee »

s1gnp0st wrote: post #70
wouldbecouldbe wrote: post #54

Earlier quoted context omitted.

Yeah but to be honest been a pain last days to get gpt 4 to write full pieces of code for more the 10-15 lines. Have to re-ask many times and at some point it forgets my initial specifications.

Earlier in the year I had ChatGPT 4 write a large, complicated C program. It did so remarkably well, and most of the code worked without further tweaking. Today I have the same experience. The thing fills in placeholder comments to skip over more difficult regions of the code, and routinely forgets what we were doing. Aside all the recent OpenAI drama, I've been displeased as a paying customer that their products rou…

[dead]

Re: Claude 2.1

#99

Post by gmfawcett »

Y_Y wrote: post #61
mark_l_watson wrote: post #52

So cool! I usually use Racket Scheme when playing with Anthropic's Claude. I just changed the model name to "claude-2.1" in my client library [1] and all is excellent. [1] https://leanpub.com/racket-ai/read#leanpub-auto-using-the-an...

For the sake of pedantry, I believe that Racket thinks it's separate to Scheme now, history and similarity notwithstanding.

It sure does. the renaming of PLT Scheme to Racket was very intentional: https://racket-lang.org/new-name.html
Post reply on HN