Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4.6

anthropic.com

Post reply on HN
Print view View original

571–580 of 1001 posts

Re: Claude Opus 4.6

#571

Post by jonatron »

jaggederest wrote: post #568
jonatron wrote: post #565

Can someone ask: "what is the current carrying capacity of 25mm multicore armoured thermoplastic insulated cables with aluminium conductors, on perforated cable tray?" just to see how well it can look up information in BS 7671?

> what is the current carrying capacity of 25mm multicore armoured thermoplastic insulated cables with aluminium conductors, on perforated cable tray? This is an electrical engineering question rather than a software engineering one, but let me look it up for you. WEB SEARCH ("current carrying capacity 25mm multicore armoured thermoplastic insulated cable aluminium conductors perforated cable tray BS 7671") ↳ Found 1…

That's useless, unexpectedly just Google gives the correct answer.

Re: Claude Opus 4.6

#572

Post by joshmlewis »

ck_one wrote: post #562
golfer wrote: post #560

Earlier quoted context omitted.

There's lots of websites that list the spells. It's well documented. Could Claude simply be regurgitating knowledge from the web? Example: https://harrypotter.fandom.com/wiki/List_of_spells

It didn't use web search. But for sure it has some internal knowledge already. It's not a perfect needle in the hay stack problem but gemini flash was much worse when I tested it last time.

I think the OP was implying that it's probably already baked into its training data. No need to search the web for that.

Re: Claude Opus 4.6

#573

Post by viraptor »

ck_one wrote: post #562
golfer wrote: post #560

Earlier quoted context omitted.

There's lots of websites that list the spells. It's well documented. Could Claude simply be regurgitating knowledge from the web? Example: https://harrypotter.fandom.com/wiki/List_of_spells

It didn't use web search. But for sure it has some internal knowledge already. It's not a perfect needle in the hay stack problem but gemini flash was much worse when I tested it last time.

If you want to really test this, search/replace the names with your own random ones and see if it lists those.

Otherwise, LLMs have most of the books memorised anyway: https://arstechnica.com/features/2025/06/study-metas-llama-3...

Re: Claude Opus 4.6

#574

Post by eek2121 »

ck_one wrote: post #562
golfer wrote: post #560

Earlier quoted context omitted.

There's lots of websites that list the spells. It's well documented. Could Claude simply be regurgitating knowledge from the web? Example: https://harrypotter.fandom.com/wiki/List_of_spells

It didn't use web search. But for sure it has some internal knowledge already. It's not a perfect needle in the hay stack problem but gemini flash was much worse when I tested it last time.

Honestly? My advice would be to cook something custom up! You don't need to do all the text yourself. Maybe have AI spew out a bunch of text, or take obscure existing text and insert hidden phrases here or there.

Shoot, I'd even go so far as to write a script that takes in a bunch of text, reorganizes sentences, and outputs them in a random order with the secrets. Kind of like a "Where's Waldo?", but for text

Just a few casual thoughts.

I'm actually thinking about coming up with some interesting coding exercises that I can run across all models. I know we already have benchmarks, however some of the recent work I've done has really shown huge weak points in every model I've run them on.

Re: Claude Opus 4.6

#575

Post by simonw »

m101 wrote: post #361
chis wrote: post #231

Earlier quoted context omitted.

It's quite clear that these companies do make money on each marginal token. They've said this directly and analysts agree [1]. It's less clear that the margins are high enough to pay off the up-front cost of training each model. [1] https://epochai.substack.com/p/can-ai-companies-become-profi...

It’s not clear at all because model training upfront costs and how you depreciate them are big unknowns, even for deprecated models. See my last comment for a bit more detail.

They are obviously losing money on training. I think they are selling inference for less than what it costs to serve these tokens.

That really matters. If they are making a margin on inference they could conceivably break even no matter how expensive training is, provided they sign up enough paying customers.

If they lose money on every paying customer then building great products that customers want to pay for them will just make their financial situation worse.

Re: Claude Opus 4.6

#576

Post by simonw »

magicalist wrote: post #316
chis wrote: post #231

Earlier quoted context omitted.

It's quite clear that these companies do make money on each marginal token. They've said this directly and analysts agree [1]. It's less clear that the margins are high enough to pay off the up-front cost of training each model. [1] https://epochai.substack.com/p/can-ai-companies-become-profi...

> They've said this directly and analysts agree [1] chasing down a few sources in that article leads to articles like this at the root of claims[1], which is entirely based on information "according to a person with knowledge of the company’s financials", which doesn't exactly fill me with confidence. [1] https://www.theinformation.com/articles/openai-getting-effic...

"according to a person with knowledge of the company’s financials" is how professional journalists tell you that someone who they judge to be credible has leaked information to them.

I wrote a guide to deciphering that kind of language a couple of years ago: https://simonwillison.net/2023/Nov/22/deciphering-clues/

Re: Claude Opus 4.6

#577

Post by emp17344 »

chis wrote: post #231
cootsnuck wrote: post #100

Earlier quoted context omitted.

I have not see any reporting or evidence at all that Anthropic or OpenAI is able to make money on inference yet. > Turns out there was a lot of low-hanging fruit in terms of inference optimization that hadn't been plucked yet. That does not mean the frontier labs are pricing their APIs to cover their costs yet. It can both be true that it has gotten cheaper for them to provide inference and that they still are subsid…

It's quite clear that these companies do make money on each marginal token. They've said this directly and analysts agree [1]. It's less clear that the margins are high enough to pay off the up-front cost of training each model. [1] https://epochai.substack.com/p/can-ai-companies-become-profi...

Sue, but if they stop training new models, the current models will be useless in a few years as our knowledge base evolves. They need to continually train new models to have a useful product.

Re: Claude Opus 4.6

#578

Post by nl »

wasmainiac wrote: post #566
tedsanders wrote: post #369

Earlier quoted context omitted.

We don't vary our model quality with time of day or load (beyond negligible non-determinism). It's the same weights all day long with no quantization or other gimmicks. They can get slower under heavy load, though. (I'm from OpenAI.)

Thanks for the response, I appreciate it. I do notice variation in quality throughout the day. I use it primarily for searching documentation since it’s faster than google in most case, often it is on point, but also it seems off at times, inaccurate or shallow maybe. In some cases I just end the session.

Usually I find this kind of variation is due to context management.

Accuracy can decreases at large context sizes. OpenAI's compaction handles this better than anyone else, but it's still an issue.

If you are seeing this kind of thing start a new chat and re-run the same query. You'll usually see an improvement.

Re: Claude Opus 4.6

#579

Post by pron »

ozim wrote: post #540
xiomrze wrote: post #520

Earlier quoted context omitted.

Honest question, how do you know if it's pulling from context vs from memory? If I use Opus 4.6 with Extended Thinking (Web Search disabled, no books attached), it answers with 130 spells.

Exactly there was this study where they were trying to make LLM reproduce HP book word for word like giving first sentences and letting it cook. Basically they managed with some tricks make 99% word for word - tricks were needed to bypass security measures that are there in place for exactly reason to stop people to retrieve training material.

This reminds me of https://en.wikipedia.org/wiki/Pierre_Menard,_Author_of_the_Q... :

> Borges's "review" describes Menard's efforts to go beyond a mere "translation" of Don Quixote by immersing himself so thoroughly in the work as to be able to actually "re-create" it, line for line, in the original 17th-century Spanish. Thus, Pierre Menard is often used to raise questions and discussion about the nature of authorship, appropriation, and interpretation.

Re: Claude Opus 4.6

#580

Post by muzani »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

There's a benchmark which works similarly but they ask harder questions, also based on books https://fiction.live/stories/Fiction-liveBench-Feb-21-2025/o...

I guess they have to add more questions as these context windows get bigger.

Post reply on HN