It's hard to tell with these releases if Anthropic's astroturfing campaign has come to HN or not but I feel like it probably has
Anthropic's models are really good!
Claude Opus 4.6
871–880 of 1001 posts
Re: Claude Opus 4.6
#872Earlier quoted context omitted.
It could still remember where each spell is mentioned. I think the only way to properly test this would be to run it against an unpublished manuscript.
Any obscure work of fiction or fanfiction would likely be fine as a casual test. If you ask a model to discuss an obscure work it'll have no clue what it's about. This is very different than asking about Harry Potter.
Yeah, that's what I've been doing as well, and at least Gemini 3 Pro did not fare very well.
Re: Claude Opus 4.6
#873The AI wars continue
Re: Claude Opus 4.6
#874Earlier quoted context omitted.
Surely the corpus Opus 4.6 ingested would include whatever reference you used to check the spells were there. I mean, there are probably dozens of pages on the internet like this: https://www.wizardemporium.com/blog/complete-list-of-harry-p... Why is this impressive? Do you think it's actually ingesting the books and only using those as a reference? Is that how LLMs work at all? It seems more likely it's predicting t…
Most people still don't realize that general public world knowledge is not really a test for a model that was trained on general public world knowledge. I wouldn't be surprised if even proprietary content like the books themselves found their way into the training data, despite what publishers and authors may think of that. As a matter of fact, with all the special deals these companies make with publishers, it is ge…
> even proprietary content like the books themselves
This definitely raises an interesting question. It seems like a good chunk of popular literature (especially from the 2000s) exists online in big HTML files. Immediately to mind was House of Leaves, Infinite Jest, Harry Potter, basically any Stephen King book - they've all been posted at some point.
Do LLMS have a good way of inferring where knowledge from the context begins and knowledge from the training data ends?
Re: Claude Opus 4.6
#875Does anyone with more insight into the AI/LLM industry happen to know if the cost to run them in normal user-workflows is falling? The reason I'm asking is because "agent teams" while a cool concept, it largely constrained by the economics of running multiple LLM agents (i.e. plans/API calls that make this practical at scale are expensive). A year or more ago, I read that both Anthropic and OpenAI were losing money o…
The cost per token served has been falling steadily over the past few years across basically all of the providers. OpenAI dropped the price they charged for o3 to 1/5th of what it was in June last year thanks to "engineers optimizing inferencing", and plenty of other providers have found cost savings too. Turns out there was a lot of low-hanging fruit in terms of inference optimization that hadn't been plucked yet. >…
But a) that's the cost to the user -- we don't know how much loss they're taking on those and b) the number of tokens to serve a similar prompt has been going up, so that the total cost to serve a prompt has been going up in general. Any cost analysis that doesn't mention these is hugely misleading.
Re: Claude Opus 4.6
#876Earlier quoted context omitted.
Surely the corpus Opus 4.6 ingested would include whatever reference you used to check the spells were there. I mean, there are probably dozens of pages on the internet like this: https://www.wizardemporium.com/blog/complete-list-of-harry-p... Why is this impressive? Do you think it's actually ingesting the books and only using those as a reference? Is that how LLMs work at all? It seems more likely it's predicting t…
Most people still don't realize that general public world knowledge is not really a test for a model that was trained on general public world knowledge. I wouldn't be surprised if even proprietary content like the books themselves found their way into the training data, despite what publishers and authors may think of that. As a matter of fact, with all the special deals these companies make with publishers, it is ge…
[deleted]
Re: Claude Opus 4.6
#877wow
Re: Claude Opus 4.6
#878Earlier quoted context omitted.
Surely the corpus Opus 4.6 ingested would include whatever reference you used to check the spells were there. I mean, there are probably dozens of pages on the internet like this: https://www.wizardemporium.com/blog/complete-list-of-harry-p... Why is this impressive? Do you think it's actually ingesting the books and only using those as a reference? Is that how LLMs work at all? It seems more likely it's predicting t…
Most people still don't realize that general public world knowledge is not really a test for a model that was trained on general public world knowledge. I wouldn't be surprised if even proprietary content like the books themselves found their way into the training data, despite what publishers and authors may think of that. As a matter of fact, with all the special deals these companies make with publishers, it is ge…
> I wouldn't be surprised if even proprietary content like the books themselves found their way into the training data
No need for surprises! It is publicly known that the corpus of 'shadow libraries' such as Library Genesis and Anna's Archive were specifically and manually requested by at least NVIDIA for their training data [1], used by Google in their training [2], downloaded by Meta employees [3] etc.
[1] https://news.ycombinator.com/item?id=46572846
[2] https://www.theguardian.com/technology/2023/apr/20/fresh-con...
[3] https://www.theverge.com/2023/7/9/23788741/sarah-silverman-o...
Re: Claude Opus 4.6
#879[dead]
Re: Claude Opus 4.6
#880Earlier quoted context omitted.
Most people still don't realize that general public world knowledge is not really a test for a model that was trained on general public world knowledge. I wouldn't be surprised if even proprietary content like the books themselves found their way into the training data, despite what publishers and authors may think of that. As a matter of fact, with all the special deals these companies make with publishers, it is ge…
> even proprietary content like the books themselves This definitely raises an interesting question. It seems like a good chunk of popular literature (especially from the 2000s) exists online in big HTML files. Immediately to mind was House of Leaves, Infinite Jest, Harry Potter, basically any Stephen King book - they've all been posted at some point. Do LLMS have a good way of inferring where knowledge from the cont…
> It seems like a good chunk of popular literature (especially from the 2000s) exists online in big HTML files
Anna's Archive alone claims to currently publicly host 61,654,285 books, more than 1PB in total.