Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4.6

anthropic.com

Post reply on HN
Print view View original

641–650 of 1001 posts

Re: Claude Opus 4.6

#641

Post by tedsanders »

thinkingtoilet wrote: post #464
wasmainiac wrote: post #265

Earlier quoted context omitted.

Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?

We know Open AI got caught getting benchmark data and tuning their models to it already. So the answer is a hard no. I imagine over time it gives a general view of the landscape and improvements, but take it with a large grain of salt.

Are you referring to FrontierMath?

We had access to the eval data (since we funded it), but we didn't train on the data or otherwise cheat. We didn't even look at the eval results until after the model had been trained and selected.

Re: Claude Opus 4.6

#642

Post by esafak »

jonatron wrote: post #571
jaggederest wrote: post #568

Earlier quoted context omitted.

> what is the current carrying capacity of 25mm multicore armoured thermoplastic insulated cables with aluminium conductors, on perforated cable tray? This is an electrical engineering question rather than a software engineering one, but let me look it up for you. WEB SEARCH ("current carrying capacity 25mm multicore armoured thermoplastic insulated cable aluminium conductors perforated cable tray BS 7671") ↳ Found 1…

That's useless, unexpectedly just Google gives the correct answer.

Great. So it did not waste its memory on trivia. Don't memorize what you can look up.

Re: Claude Opus 4.6

#643

Post by esafak »

breuleux wrote: post #437
bopbopbop7 wrote: post #389

Earlier quoted context omitted.

> Besides, what is the human brain if not a machine that generates "tokens" that the body propagates through nerves to produce physical actions? Ah yes, the brain is as simple as predicting the next token, you just cracked what neuroscientists couldn't for years.

The point is that "predicting the next token" is such a general mechanism as to be meaningless. We say that LLMs are "just" predicting the next token, as if this somehow explained all there was to them. It doesn't, not any more than "the brain is made out of atoms" explains the brain, or "it's a list of lists" explains a Lisp program. It's a platitude.

It's not meaningless, it's a prediction task, and prediction is commonly held to be closely related if not synonymous with intelligence.

Re: Claude Opus 4.6

#644

Post by charcircuit »

thunfischtoast wrote: post #504
charcircuit wrote: post #41

From the press release at least it sounds more expensive than Opus 4.5 (more tokens per request and fees for going over 200k context). It also seems misleading to have charts that compare to Sonnet 4.5 and not Opus 4.5 (Edit: It's because Opus 4.5 doesn't have a 1M context window). It's also interesting they list compaction as a capability of the model. I wonder if this means they have RL trained this compaction as o…

On Openrouter it has the same cost per token as 4.5

You missed my point. If the average request uses more tokens than 4.5, then you will pay more sending those requests to 4.6 than 4.5.

Imagine 2 models where when asking a yes or no question the first model just outputs a single yes or no then but the second model outputs a 10 page essay and then either yes or no. They could have the same price per token but ultimately one will be cheaper to ask questions to.

Re: Claude Opus 4.6

#645

Post by obirunda »

ck_one wrote: post #562
golfer wrote: post #560

Earlier quoted context omitted.

There's lots of websites that list the spells. It's well documented. Could Claude simply be regurgitating knowledge from the web? Example: https://harrypotter.fandom.com/wiki/List_of_spells

It didn't use web search. But for sure it has some internal knowledge already. It's not a perfect needle in the hay stack problem but gemini flash was much worse when I tested it last time.

This underestimates how much of the Internet is actually compressed into and is an integral part of the model's weights. Gemini 2.5 can recite the first Harry Potter book verbatim for over 75% of the book.

Re: Claude Opus 4.6

#646

Post by majewsky »

muzani wrote: post #548
meroes wrote: post #521

Earlier quoted context omitted.

What is this supposed to show exactly? Those books have been feed into LLMs for years and there's even likely specific RLHF's on extracting spells from HP.

There was a time when I put the EA-Nasir text into base64 and asked AI to convert it. Remarkably it identified the correct text but pulled the most popular translation of the text than the one I gave it.

Sucks that you got a really shitty response to your prompt. If I were you, the model provider would be receiving my complaint via clay tablet right away.

Re: Claude Opus 4.6

#647

Post by IhateAI »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

[flagged]

Re: Claude Opus 4.6

#648

Post by siwatanejo »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

> All 7 books come to ~1.75M tokens

How do you know? Each word is one token?

Re: Claude Opus 4.6

#649

Post by zozbot234 »

slopusila wrote: post #563
zozbot234 wrote: post #56

Earlier quoted context omitted.

> i.e. plans/API calls that make this practical at scale are expensive Local AI's make agent workflows a whole lot more practical. Making the initial investment for a good homelab/on-prem facility will effectively become a no-brainer given the advantages on privacy and reliability, and you don't have to fear rugpulls or VC's playing the "lose money on every request" game since you know exactly how much you're paying…

on prem economics dont work because you can't batch requests. unless you are able to run 100 agents at the same time all the time

> unless you are able to run 100 agents at the same time all the time

Except that newer "agent swarm" workflows do exactly that. Besides, batching requests generally comes with a sizeable increase in memory footprint, and memory is often the main bottleneck especially with the larger contexts that are typical of agent workflows. If you have plenty of agentic tasks that are not especially latency-critical and don't need the absolutely best model, it makes plenty of sense to schedule these for running locally.

Re: Claude Opus 4.6

#650

Post by GorbachevyChase »

tedsanders wrote: post #369
wasmainiac wrote: post #265

Earlier quoted context omitted.

Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?

We don't vary our model quality with time of day or load (beyond negligible non-determinism). It's the same weights all day long with no quantization or other gimmicks. They can get slower under heavy load, though. (I'm from OpenAI.)

Hi Ted. I think that language models are great, and they’ve enabled me to do passion projects I never would have attempted before. I just want to say thanks.
Post reply on HN