Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

351–360 of 723 posts

Re: Claude 3 model family

#351

Post by matchagaucho »

lukev wrote: post #313
nopinsight wrote: post #283

Earlier quoted context omitted.

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

This doesn't pass the sniff test for me. Not sure if these models are memorizing the answers or something else, but it's simply not the case that they're as capable as a domain expert (yet.) I do not have a PhD, but in areas I do have expertise, you really don't have to push these models that hard to before they start to break down and emit incomplete or wrong analysis.

They claim the model was grounded with a 25-shot Chain-of-Thought (CoT) prompt.

Re: Claude 3 model family

#352

Post by brokencode »

michaelt wrote: post #286
brokencode wrote: post #251

Earlier quoted context omitted.

Why is this a trick or somehow inferior to getting the AI model to be able to do it natively? Most humans also can’t reliably do complex arithmetic without the use of something like a calculator. And that’s no trick. We’ve built the modern world with such tools. Why should we fault AI for doing what we do? To me, training the AI use a calculator is not just a trick for hype, it’s exciting progress.

By all means if it works to solve your problem, go ahead and do it. The reason some people have mixed feelings about this because of a historical observation - http://www.incompleteideas.net/IncIdeas/BitterLesson.html - that we humans often feel good about adding lots of hand-coded smarts to our ML systems reflecting our deep and brilliant personal insights. But it turns out just chucking loads of data and compute at…

I agree we should teach our AI models how to do math, but that doesn’t mean they shouldn’t use tools as well.

Certain problems are always going to be very algorithmic and computationally expensive to solve. Asking an LLM to multiply each row in a spreadsheet by pi for example would be a total waste.

To handle these kinds of problems, the AI should be able to write and execute its own code for example. Then save the results in a database or other long term storage.

Another thing it would need is access to realtime data sources and reliable databases to draw on data not in the training set. No matter how much you train a model, these will still be useful.

Re: Claude 3 model family

#353

Post by sirsinsalot »

Der_Einzige wrote: post #275
SubiculumCode wrote: post #75

Earlier quoted context omitted.

NVidia's execs think so. It would be an ironic thing that it was open source that killed the programmer; as how would they train it otherwise? As a scientist, should I continue to support open access journals, just so I can be trained away? Slightly tongue in check, but not really.

This is also why I have about negative sympathy for artists who are crying about AI taking their jobs. Programmers (specifically AI researchers) looked at their 300K+ a year salaries and embraced the idea of automating away the work despite how lucrative it would be to continue to spin one's wheels on it. The culture of open source is strong among SWEs, even one's who would lose millions of unrealized gains/earnings…

Group A making 300K embraces risk more readily than group B making 30k

Wow who would've thought a large income allowed you to take risks and embrace change?

Imagine being a copywriter for 25 years, on 30k, paying a mortgage, running a car, feeding a family, trying to save on what's left... And all your clients dry up. You've got no other skills, you invested your career in copywriting. You don't have the savings to pivot and your kids need new school uniforms now, not when you reskill to a new career.

You lost your clients. Now your home. Maybe your wife and kids too.

Money is a buffer from risk most don't have.

I hope you never feel this and get to keep the luxury of poking fun at other people for being risk averse without the buffer. Maybe bring some compassion to the table tho? Furry art or copywriting, it isn't anyone's place to judge the merit of the income.

Re: Claude 3 model family

#354

Post by josh-sematic »

rthnbgrredf wrote: post #350

Could anyone recommend an open-source tool capable of simultaneously sending the same prompt to various language models like GPT-4, Gemini, and Claude, and displaying their responses side by side for comparison? I tried chathub in the past, but they decided to not release any more source as of now.

Not open-source, but https://airtrain.ai lets you do this. Disclaimer: I’m an engineer there.

Edit: aiming to have Claude 3 support by tomorrow.

Re: Claude 3 model family

#355

Post by danielcampos93 »

Ninjinka wrote: post #319

One-off anecdote: I pasted a question I asked GPT-4 last night regarding a bug in some game engine code (including the 2000 lines of relevant code). Whereas GPT-4 correctly guessed the issue, Claude Opus gave some generic debugging tips that ultimately would not lead to finding the answer, such as "add logging", "verify the setup", and "seek community support."

Claude's answers sometimes fill the niche of 'management consultant'

Re: Claude 3 model family

#356

Post by jlas »

SirensOfTitan wrote: post #173

What is the probability that newer models are just overfitting various benchmarks? A lot of these newer models seem to underperform GPT-4 in most of my daily queries, but I'm obviously swimming in the world of anecdata.

Non-zero probability I think, one interesting measure of overfitting I've seen is contamination (where the model has seen the exact questions it is being evaluated on) see stats at https://hitz-zentroa.github.io/lm-contamination/

Re: Claude 3 model family

#357

Post by [deleted] »

_sword wrote: post #14

At this point I wonder how much of the GPT-4 advantage has been OpenAI's pre-training data advantage vs. fundamental advancements in theory or engineering. Has OpenAI mastered deep nuances others are missing? Or is their data set large enough that most test-cases are already a sub-set of their pre-training data?

[deleted]

Re: Claude 3 model family

#358

Post by chessgecko »

vermorel wrote: post #28

Does any of those LLM-as-a-service companies provide a mechanism to "save" a given input? Paying only for the state storage and the extra input when continuing the completion from the snapshot? Indeed, at 1M token and $15/M tokens, we are talking of $10+ API calls (per call) when maxing out the LLM capacity. I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact sam…

The problem is that it’s probably often not a lot cheaper. Most of the high end gpus have comparatively little bandwidth over pcie (that you’d need to use to store the context on a nvme for example). The cost there would scale with length too so you wouldn’t necessarily save more in that situation either. I think if you used a small enough gqa ratio and you knew for sure you would reuse the weights it could work, but my suspicion is that in general it would just be cheaper to recalculate.

Re: Claude 3 model family

#359

Post by dwaltrip »

causal wrote: post #134
7moritz7 wrote: post #21

Look at that jump in grade school math. From 55 % with GPT 3.5 to 95 % for both Claude 3 and GPT 4.

Yeah I've been throwing arithmetic at Claude 3 Opus and so far it has been solid in responses.

Claude has a specialized calculation feature that doesn't use model inference. Just FYI.

Re: Claude 3 model family

#360

Post by uptownfunk »

What is the logic behind giving sonnet free, which is not very good, and saying hey try this for free and then pay us to use our actual good model. Like trust us, it’s really good. Uh no thanks. We need better benchmarks, this is a joke, it started w google Gemini and extend to Anthropocene. How Much money and compute wasted on this. It is a shame
Post reply on HN