Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4.6

anthropic.com

Post reply on HN
Print view View original

591–600 of 1001 posts

Re: Claude Opus 4.6

#591

Post by fragmede »

tedsanders wrote: post #369
wasmainiac wrote: post #265

Earlier quoted context omitted.

Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?

We don't vary our model quality with time of day or load (beyond negligible non-determinism). It's the same weights all day long with no quantization or other gimmicks. They can get slower under heavy load, though. (I'm from OpenAI.)

I believe you when you say you're not changing the model file loaded onto the H100s or whatever, but there's something going on, beyond just being slower, when the GPUs are heavily loaded.

Re: Claude Opus 4.6

#592

Post by MadnessASAP »

georgeven wrote: post #356
ifwinterco wrote: post #340

Earlier quoted context omitted.

On benchmarks GPT 5.2 was roughly equivalent to Opus 4.5 but most people who've used both for SWE stuff would say that Opus 4.5 is/was noticeably better

Interesting. Everyone in my circle said the opposite.

My experience is that Codex follows directions better but Claude writes better code.

ChatGPT-5.2-Codex follows directions to ensure a task [bead](https://github.com/steveyegge/beads) is opened before starting a task and to keep it updated almost to a fault. Claude-Opus-4.5 with the exact same directions, forgets about it within a round or two. Similarly, I had a project that required very specific behaviour from a couple functions, it was documented in a few places including comments at the top and bottom of the function. Codex was very careful in ensuring the function worked as was documented. Claude decided it was easier to do the exact opposite, rewrote the function, the comments, and the documentation to saynit now did the opposite of what was previously there.

If I believed a LLM could be spiteful, I would've believed it on that second one. I certainly felt some after I realised what it had done. The comment literally said:

  // Invariant regardless of the value of X, this function cannot return Y
And it turned it into:

  // Returns Y if X is true

Re: Claude Opus 4.6

#593

Post by fdeage »

stkai wrote: post #232
simonw wrote: post #40

The bicycle frame is a bit wonky but the pelican itself is great: https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe...

Would love to find out they're overfitting for pelican drawings.

OpenAI claims not to: https://x.com/aidan_mclau/status/1986255202132042164

Re: Claude Opus 4.6

#594

Post by dwa3592 »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

have another LLM (gemini, chatgpt) make up 50 new spells. insert those and test and maybe report here :)

Re: Claude Opus 4.6

#595

Post by Trufa »

tedsanders wrote: post #450
Trufa wrote: post #379

Earlier quoted context omitted.

Can you be more specific than this? does it vary in time from launch of a model to the next few months, beyond tinkering and optimization?

Yeah, happy to be more specific. No intention of making any technically true but misleading statements. The following are true: - In our API, we don't change model weights or model behavior over time (e.g., by time of day, or weeks/months after release) - Tiny caveats include: there is a bit of non-determinism in batched non-associative math that can vary by batch / hardware, bugs or API downtime can obviously change…

I ask then unironically then, am I imagining that models are great when they start and degrade over time?

I've had this perceived experience so many times, and while of course it's almost impossible to be objective about this, it just seem so in your face.

I don't discard being novelty plus getting used to it, plus psychological factors, do you have any takes on this?

Re: Claude Opus 4.6

#596

Post by suddenlybananas »

nodja wrote: post #442
Someone1234 wrote: post #383

Earlier quoted context omitted.

> This is obviously not true, you can use real data and common sense. It isn't "common sense" at all. You're comparing several companies losing money, to one another, and suggesting that they're obviously making money because one is under-cutting another more aggressively. LLM/AI ventures are all currently under-water with massive VC or similar money flowing in, they also all need training data from users, so it is v…

Doing some math in my head, buying the GPUs at retail price, it would take probably around half a year to make the money back, probably more depending how expensive electricity is in the area you're serving from. So I don't know where this "losing money" rhetoric is coming from. It's probably harder to source the actual GPUs than making money off them.

electricity

Re: Claude Opus 4.6

#597

Post by irishcoffee »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

The top comment is about finding basterized latin words from childrens books. The future is here.

Re: Claude Opus 4.6

#598

Post by clhodapp »

eek2121 wrote: post #574
ck_one wrote: post #562

Earlier quoted context omitted.

It didn't use web search. But for sure it has some internal knowledge already. It's not a perfect needle in the hay stack problem but gemini flash was much worse when I tested it last time.

Honestly? My advice would be to cook something custom up! You don't need to do all the text yourself. Maybe have AI spew out a bunch of text, or take obscure existing text and insert hidden phrases here or there. Shoot, I'd even go so far as to write a script that takes in a bunch of text, reorganizes sentences, and outputs them in a random order with the secrets. Kind of like a "Where's Waldo?", but for text Just a…

Having AI spew it might suffer from the fact that the spew itself is influenced by AI's weights. I think your best bet would be to use a new human-authored work that was released after the model's context cutoff.

Re: Claude Opus 4.6

#599

Post by kybernetikos »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

I recently got junie to code me up an MCP for accessing my calibre library. https://www.npmjs.com/package/access-calibre

My standard test for that was "Who ends up with Bilbo's buttons?"

Re: Claude Opus 4.6

#600

Post by Geste »

irishcoffee wrote: post #597
ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

The top comment is about finding basterized latin words from childrens books. The future is here.

I'll have some of that coffee too, this is quite a sad time we're living where this is a proper use of our limited resources.
Post reply on HN