551–560 of 1001 posts
Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…
Earlier quoted context omitted.
Honest question, how do you know if it's pulling from context vs from memory? If I use Opus 4.6 with Extended Thinking (Web Search disabled, no books attached), it answers with 130 spells.
Exactly there was this study where they were trying to make LLM reproduce HP book word for word like giving first sentences and letting it cook. Basically they managed with some tricks make 99% word for word - tricks were needed to bypass security measures that are there in place for exactly reason to stop people to retrieve training material.
Earlier quoted context omitted.
Also explains why Claude Code is a React app outputting to a Terminal. (Seriously.)
I did some debugging on this today. The results are... sobering. Memory comparison of AI coding CLIs (single session, idle): | Tool | Footprint | Peak | Language | |-------------|-----------|--------|---------------| | Codex | 15 MB | 15 MB | Rust | | OpenCode | 130 MB | 130 MB | Go | | Claude Code | 360 MB | 746 MB | Node.js/React | That's a 24x to 50x difference for tools that do the same thing: send text to an API…
Earlier quoted context omitted.
Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?
We know Open AI got caught getting benchmark data and tuning their models to it already. So the answer is a hard no. I imagine over time it gives a general view of the landscape and improvements, but take it with a large grain of salt.
You always have to question these benchmarks, especially when the in-house researchers can potentially game them if they wanted to.
Which is why it must be independent.
[0] https://gizmodo.com/meta-cheated-on-ai-benchmarks-and-its-a-...
I just tested both codex 5.3 and opus 4.6 and both returned pretty good output, but opus 4.6's limits are way too strict. I am probably going to cancel my Claude subscription for that reason: What do you want to do? 1. Stop and wait for limit to reset 2. Switch to extra usage 3. Upgrade your plan Enter to confirm · Esc to cancel How come they don't have "Cancel your subscription and uninstall Claude Code"? Codex last…
I'm still not sure I understand Anthropic's general strategy right now. They are doing these broad marketing programs trying to take on ChatGPT for "normies". And yet their bread and butter is still clearly coding. Meanwhile, Claude's general use cases are... fine. For generic research topics, I find that ChatGPT and Gemini run circles around it: in the depth of research, the type of tasks it can handle, and the qual…
I kinda agree. Their model just doesn't feel "daily" enough. I would use it for any "agentic" tasks and for using tools, but definitely not for day to day questions.
> We build Claude with Claude. Our engineers write code with Claude Code every day well that explains quite a bit
Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…
Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…