Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

271–280 of 723 posts

Re: Claude 3 model family

#271

Post by Spivak »

lanstin wrote: post #262
brokencode wrote: post #251

Earlier quoted context omitted.

Why is this a trick or somehow inferior to getting the AI model to be able to do it natively? Most humans also can’t reliably do complex arithmetic without the use of something like a calculator. And that’s no trick. We’ve built the modern world with such tools. Why should we fault AI for doing what we do? To me, training the AI use a calculator is not just a trick for hype, it’s exciting progress.

It would be exciting if the LLM knew it needed a calculator for certain things and went out and got it. If the human supervisors are pre-screening the input and massaging what the LLM is doing that is a sign we don't understand LLMs enough to engineer them precisely and can't count on them to be aware of their own limitations, which would seem to be a useful part of general intelligence.

It can if you let it, that's the whole premise of LangChain style reasoning and it works well enough. My dumb little personal chatbot knows it can access a Python REPL to carry out calculations and it does.

Re: Claude 3 model family

#272

Post by ActVen »

Opus just crushed Gemini Pro and GPT4 on a pretty complex question I have asked all of them, including Claude 2. It involved taking a 43 page life insurance investment pdf and identifying various figures in it. No other model has gotten close. Except for Claude 3 sonnet, which just missed one question.

Re: Claude 3 model family

#273

Post by ActVen »

jasonjmcghee wrote: post #164

I've tried all the top models. GPT4 beats everything I've tried, including Gemini 1.5- until today. I use GPT4 daily on a variety of things. Claude 3 Opus (been using temperature 0.7) is cleaning up. I'm very impressed.

Same here. Opus just crushed Gemini Pro and GPT4 on a pretty complex question I have asked all of them, including Claude 2. It involved taking a 43 page life insurance investment pdf and identifying various figures in it. No other model has gotten close. Except for Claude 3 sonnet, which just missed one question.

Re: Claude 3 model family

#274

Post by behnamoh »

eliya_confiant wrote: post #224
simonw wrote: post #189

I just released a plugin for my LLM command-line tool that adds support for the new Claude 3 models: pipx install llm llm install llm-claude-3 llm keys set claude # paste Anthropic API key here llm -m claude-3-opus '3 fun facts about pelicans' llm -m claude-3-opus '3 surprising facts about walruses' Code here: https://github.com/simonw/llm-claude-3 More on LLM: https://llm.datasette.io/

Hi Simon, Big fan of your work with the LLM tool. I have a cool use for it that I wanted to share with you (on mac). First, I created a quick action in Automator that recieves text. Then I put together this script with the help of ChaptGPT: escaped_args="" for arg in "$@"; do escaped_arg=$(printf '%s\n' "$arg" | sed "s/'/'\\\\''/g") escaped_args="$escaped_args '$escaped_arg'" done result=$(/Users/XXXX/Library/Python/…

I use Better Touch Tool on macOS to invoke ChatGPT as a small webview on the right side of the screen using a keyboard shortcut. Here it is: https://dropover.cloud/0db372

Re: Claude 3 model family

#275

Post by Der_Einzige »

SubiculumCode wrote: post #75
ethbr1 wrote: post #41

Earlier quoted context omitted.

On the other hand, programmers are very expensive. At some level of accuracy and consistency (human order-of-magnitude?), the pricing of the service should start approaching the pricing of the human alternative. And first glance at numbers, LLMs are still way underpriced relative to humans.

NVidia's execs think so. It would be an ironic thing that it was open source that killed the programmer; as how would they train it otherwise? As a scientist, should I continue to support open access journals, just so I can be trained away? Slightly tongue in check, but not really.

This is also why I have about negative sympathy for artists who are crying about AI taking their jobs.

Programmers (specifically AI researchers) looked at their 300K+ a year salaries and embraced the idea of automating away the work despite how lucrative it would be to continue to spin one's wheels on it. The culture of open source is strong among SWEs, even one's who would lose millions of unrealized gains/earnings as a result of embracing it.

Artists looked at their 30K+ a year salaries from drawing furry hentai on furaffinity and panic at the prospect of losing their work, to the point of making whole political protest movements against AI art. Artists have also never defended open source en mass, and are often some of the first to defend crappy IP laws.

Why be a luddite over something so crappy to defend?

(edit to respond)

I grew up poor as shit and got myself out of that with code. I don't need a lecture about appearing as an elitist.

I'm more than "poking fun" at them - I'm calling them out for lying about their supposed left-wing sensibilities. Artists have postured as being the "vanguard" of the left wing revolution for awhile (i.e. situationalist international and may 68), but the moment that they had a chance to implement their tactics in the art world (open source AI art), they shunned it and cried and embraced ludditism.

Compare this to the world of AI right now. AI has somehow "legally circumvented" copyright laws and we are living in a de-facto post-copyright world. Huggingface and Richard Stallman as an entity/community and individual have done more to democratize access to and give the poors real access to social and economy mobility than any artists have done in the last 10 years, anywhere in the entire world.

You should embrace shit jobs going away, especially in a world where the speed to "re-skill" is often on the orders of hours when AI is involved. I am pointing out that the well-paid AI professional had much to lose and embraced losing it anyway, while the furry artist acted greedily over their pretty awful situation.

Re: Claude 3 model family

#276

Post by eschluntz »

nopinsight wrote: post #245

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…

(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)

Re: Claude 3 model family

#277

Post by lanstin »

HarHarVeryFunny wrote: post #241
zurfer wrote: post #227

Earlier quoted context omitted.

Every big tech in the beginning looked fragile/no moats. I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.

Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances. Never say never though - look at Tesla coming out of no…

Also worth keeping in mind the lock in for the big tech firms is due to business decisions not the technology per se. If we had say micropaynents in http1 headers in 1998 we might have a much more decentralized system supported by distributed subscriptions rather than ads. To this day I cannot put up $50 to mastodon and have it split amongst the posts I like or boost or whatever. Instead we have all the top content authors trying to get me to subscribe to their email subscriptions which Isa vastly inferior interface and too expensive to get money to all the good writers out there.

Re: Claude 3 model family

#278

Post by paradite »

AaronFriel wrote: post #269
paradite wrote: post #264

Earlier quoted context omitted.

I see the purple icon with "ChatGPT Classic" on my share link, but if I open it in incognito without login, it shows as green "ChatGPT". You can try opening in incognito your own chat share link. I use ChatGPT Classic, which is an official GPT from OpenAI without the extra system prompt from normal ChatGPT. https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic It is explicitly mentioned in the GPT that it uses GPT-4.…

Oh, I see. That must be frustrating to folks at OpenAI. Their product rests on the quality of their models, and making users unable to see which results came from their best doesn't help. FWIW, GPT-4 and GPT-4 Turbo via developer API call both seem to produce the result you expect.

FYI, the correct method is

  created_at: timestamp('created_at').defaultNow(), // Add created_at column definition
Which Claude 3 Sonnet correctly produces.

ChatGPT Classic (GPT-4) gives:

  created_at: timestamp('created_at').default(sql`NOW()`), // Add this line
Which is okay, but not ideal. And it also misses the need to import `sql` template tag.

Your share link gives:

  created_at: timestamp('created_at').default('NOW()'),
Which would throw a TypeScript error for the wrong type used in arguments for `default`.

Re: Claude 3 model family

#279

Post by wesleyyue »

gkfasdfasdf wrote: post #268
wesleyyue wrote: post #62

Just added Claude 3 to Chat at https://double.bot if anyone wants to try it for coding. Free for now and will push Claude 3 for autocomplete later this afternoon. From my early tests this seems like the first API alternative to GPT4. Huge!

How do you guys compare to codium [0]? Also, any plans to support vim/neovim integration (codium has pretty good support in place [1]). Thanks. [0] - https://www.codium.ai [1] - https://github.com/Exafunction/codeium.vim

I think the tldr would be that they have more products (for example, their agent to write git commit messages). In the products we do have (autocomplete, chat), we spend a lot of time to get the details right. For example for autocomplete:

* we always close any brackets opened by autocomplete (and never extra brackets, which is the most annoying thing about github copilot)

* we automatically add import statements for libraries that autocomplete used

* mid-line completions

* we turn off autocomplete when you're writing a comment to avoid disrupting your train of thought

You can read more about these small details here: https://docs.double.bot/copilot

As you noted we don't have a vim integration yet, but it is on our roadmap!

Re: Claude 3 model family

#280

Post by bufferoverflow »

Workaccount2 wrote: post #125
anonymouse008 wrote: post #87

Earlier quoted context omitted.

I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE

I'd almost say anyone not doing that is being foolish. The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all. I think some engineers feel that if they are leaning on an old school classically programed tool to assist the LLM, it's somehow cheating or impure.

> The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all.

No, that's the actual end goal. We want a NN that does everything, trained end-to-end.

Post reply on HN