Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

311–320 of 723 posts

Re: Claude 3 model family

#311

Post by janalsncm »

michaelt wrote: post #286
brokencode wrote: post #251

Earlier quoted context omitted.

Why is this a trick or somehow inferior to getting the AI model to be able to do it natively? Most humans also can’t reliably do complex arithmetic without the use of something like a calculator. And that’s no trick. We’ve built the modern world with such tools. Why should we fault AI for doing what we do? To me, training the AI use a calculator is not just a trick for hype, it’s exciting progress.

By all means if it works to solve your problem, go ahead and do it. The reason some people have mixed feelings about this because of a historical observation - http://www.incompleteideas.net/IncIdeas/BitterLesson.html - that we humans often feel good about adding lots of hand-coded smarts to our ML systems reflecting our deep and brilliant personal insights. But it turns out just chucking loads of data and compute at…

The reason we chucked loads of data at it was because we had no other options. If you wanted to write a function that classified a picture as a cat or a dog, good luck. With ML, you can learn such a function.

That logic doesn’t extend to things we already know how to program computers to do. Arithmetic already works. We don’t need a neural net to also run the calculations or play a game of chess. We have specialized programs that are probably as good as we’re going to get in those specialized domains.

Re: Claude 3 model family

#312

Post by gzer0 »

Did anthropic just kill every small model?

If I'm reading this right, Haiku benchmarks almost as good as GPT4, but its priced at $0.25/m tokens

It absolutely blows 3.5 + OSS out of the water

For reference gpt4 turbo is 10m/1m tokens, so haiku is 40X cheaper.

Re: Claude 3 model family

#313

Post by lukev »

nopinsight wrote: post #283
nopinsight wrote: post #245

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation resul…

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

This doesn't pass the sniff test for me. Not sure if these models are memorizing the answers or something else, but it's simply not the case that they're as capable as a domain expert (yet.)

I do not have a PhD, but in areas I do have expertise, you really don't have to push these models that hard to before they start to break down and emit incomplete or wrong analysis.

Re: Claude 3 model family

#314

Post by janalsncm »

bufferoverflow wrote: post #282
brokencode wrote: post #251

Earlier quoted context omitted.

Why is this a trick or somehow inferior to getting the AI model to be able to do it natively? Most humans also can’t reliably do complex arithmetic without the use of something like a calculator. And that’s no trick. We’ve built the modern world with such tools. Why should we fault AI for doing what we do? To me, training the AI use a calculator is not just a trick for hype, it’s exciting progress.

Because if NN is smart enough, it should be able to do arithmetic flawlessly. Basic arithmetic doesn't even require that much intelligence, it's mostly attention to detail.

Well it’s obviously not smart enough so the question is what do you do about it? Train another net that’s 1000x as big for 99% accuracy or hand it off to the lowly calculator which will get it right 100% of the time?

And 1000x is just a guess. We have no scaling laws about this kind of thing. It could be a million. It could be 10.

Re: Claude 3 model family

#315

Post by michaelt »

numeri wrote: post #144
michaelt wrote: post #112

Earlier quoted context omitted.

Then you might enjoy looking up the "Mixture of Experts" model design.

That has nothing to do with the idea of ensembling multiple specialized/single-purpose models. Mixture of Experts is an method of splitting the feed-forwards in a model such that only a (hopefully) relevant subset of parameters is run for each token. The model learns how to split them on its own, and usually splits based not on topic or domain, but on grammatical function or category of symbol (e.g., punctuation, cou…

An ensemble of specialists is different to a mixture of experts?

I thought half the point of MoE was to make the training tractable by allowing the different experts to be trained independently?

Re: Claude 3 model family

#316

Post by mattlondon »

Another naming disaster! Opus is better than sonnet? And sonnet is better than haiku? Perhaps this makes sense to people familiar with sonnets and haikus and opus....es?

Nonsensical to me! I know everyone loves to hate on Google, but at least pro and ultra have a sort of sense of level of sophistication.

Re: Claude 3 model family

#317

Post by netghost »

bufferoverflow wrote: post #280
Workaccount2 wrote: post #125

Earlier quoted context omitted.

I'd almost say anyone not doing that is being foolish. The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all. I think some engineers feel that if they are leaning on an old school classically programed tool to assist the LLM, it's somehow cheating or impure.

> The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all. No, that's the actual end goal. We want a NN that does everything, trained end-to-end.

"We" contains more than just one perspective though.

As someone applying LLMs to a set of problems in a production application, I just want a tool that solves the problem. Today, that tool is an LLM, tomorrow it could be anything. If there are ~hacks~ elegant techniques that can get me the results I need faster, cheaper, or more accurately, I absolutely will use those until there's a better alternative.

Re: Claude 3 model family

#318

Post by michaelt »

janalsncm wrote: post #311
michaelt wrote: post #286

Earlier quoted context omitted.

By all means if it works to solve your problem, go ahead and do it. The reason some people have mixed feelings about this because of a historical observation - http://www.incompleteideas.net/IncIdeas/BitterLesson.html - that we humans often feel good about adding lots of hand-coded smarts to our ML systems reflecting our deep and brilliant personal insights. But it turns out just chucking loads of data and compute at…

The reason we chucked loads of data at it was because we had no other options. If you wanted to write a function that classified a picture as a cat or a dog, good luck. With ML, you can learn such a function. That logic doesn’t extend to things we already know how to program computers to do. Arithmetic already works. We don’t need a neural net to also run the calculations or play a game of chess. We have specialized…

> We don’t need a neural net to also run the calculations or play a game of chess.

That's actually one of the specific examples from the link I mentioned:-

> In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not.

While it's true that they didn't use an LLM specifically, it's still an example of chucking loads of compute at the problem instead of something more elegant and human-like.

Of course, I agree that if you're looking for a good game of chess, Stockfish is a better choice than ChatGPT.

Re: Claude 3 model family

#319

Post by Ninjinka »

One-off anecdote: I pasted a question I asked GPT-4 last night regarding a bug in some game engine code (including the 2000 lines of relevant code). Whereas GPT-4 correctly guessed the issue, Claude Opus gave some generic debugging tips that ultimately would not lead to finding the answer, such as "add logging", "verify the setup", and "seek community support."

Re: Claude 3 model family

#320

Post by pknerd »

It's kind of funny that I can't access the main Claude.AI web interface as my country(Pakistan) is not in the list but they are giving away API Access to me
Post reply on HN