Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4.6

anthropic.com

Post reply on HN
Print view View original

951–960 of 1001 posts

Re: Claude Opus 4.6

#951

Post by rlt »

grey-area wrote: post #818
ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

Surely the corpus Opus 4.6 ingested would include whatever reference you used to check the spells were there. I mean, there are probably dozens of pages on the internet like this: https://www.wizardemporium.com/blog/complete-list-of-harry-p... Why is this impressive? Do you think it's actually ingesting the books and only using those as a reference? Is that how LLMs work at all? It seems more likely it's predicting t…

They should try the same thing but replace the original spell names with something else.

Re: Claude Opus 4.6

#952

Post by nwienert »

jpcompartir wrote: post #950

4.6 is a beast. Everything in plan mode first + AskUserQuestionTool, review all plans, get it to write its own CLAUDE.md for coding standards and edit where necessary and away you go. Seems noticeably better than 4.5 at keeping the codebase slim. Obviously it still needs to be kept an eye on, but it's a step up from 4.5.

Not clearly a step up for me, it's way more hesitant it seems and I don't notice context being larger at all it seems to compact just as often.

Re: Claude Opus 4.6

#953

Post by kylehotchkiss »

ck_one wrote: post #482

Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…

I love the fun metric.

My hope is that locally run models can pass this test in the next year or two!

Re: Claude Opus 4.6

#954

Post by tatjam »

atonse wrote: post #513
krystofbe wrote: post #395

Earlier quoted context omitted.

I did some debugging on this today. The results are... sobering. Memory comparison of AI coding CLIs (single session, idle): | Tool | Footprint | Peak | Language | |-------------|-----------|--------|---------------| | Codex | 15 MB | 15 MB | Rust | | OpenCode | 130 MB | 130 MB | Go | | Claude Code | 360 MB | 746 MB | Node.js/React | That's a 24x to 50x difference for tools that do the same thing: send text to an API…

Jarred Sumner (bun creator, bun was recently acquired by Anthropic) has been working exclusively on bringing down memory leaks and improving performance in CC the last couple weeks. He's been tweeting his progress. This is just regular tech debt that happens from building something to $1bn in revenue as fast as you possibly can, optimize later. They're optimizing now. I'm sure they'll have it under control in no time…

Claude, fix the memory leaks, or you'll go to jail!

Re: Claude Opus 4.6

#955

Post by Willish42 »

endymion-light wrote: post #833
bobbylarrybobby wrote: post #729

Earlier quoted context omitted.

I really like that Claude feels transactional. It answers my question quickly and concisely and then shuts up. I don't need the LLM I use to act like my best friend.

I love doing a personal side project code review with claude code, because it doesn't beat around the bush for criticism. I recently compared a class that I wrote for a side project that had quite horrible temporal coupling for a data processor class. Gemini - ends up rating it a 7/10, some small bits of feedback etc Claude - Brutal dismemberment of how awful the naming convention, structure, coupling etc, provides e…

I feel like this anecdote represents the differing incentives / philosophies of each group rather well.

I've noticed ChatGPT is rather high in its praise regardless of how valuable the input is, Gemini is less placating but still largely influenced by the perspective of the prompter, and Claude feels the most "honest" but humans are rather easy poor at judging this sort of thing.

Does anyone know if "sycophancy" has documented benchmarks the models are compared against? Maybe it's subjective and hard to measure, but given the issues with GPT 4o, this seems like a good thing to measure model to model to compare individual companies' changes as well as compare across companies.

Re: Claude Opus 4.6

#956

Post by jorl17 »

euph0ria wrote: post #756
jorl17 wrote: post #185

This is the first model to which I send my collection of nearly 900 poems and an extremely simple prompt (in Portuguese), and it manages to produce an impeccable analysis of the poems, as a (barely) cohesive whole, which span 15 years. It does not make a single mistake, it identifies neologisms, hidden meaning, 7 distinct poetic phases, recurring themes, fragments/heteronyms, related authors. It has left me completel…

Could you please post the key poems? Would love to read them.

I am way too self-conscious to do that :) Plus they are almost all in Portuguese!

Re: Claude Opus 4.6

#957

Post by rendx »

sigmoid10 wrote: post #884
rendx wrote: post #878

Earlier quoted context omitted.

> I wouldn't be surprised if even proprietary content like the books themselves found their way into the training data No need for surprises! It is publicly known that the corpus of 'shadow libraries' such as Library Genesis and Anna's Archive were specifically and manually requested by at least NVIDIA for their training data [1], used by Google in their training [2], downloaded by Meta employees [3] etc. [1] https:/…

The big AI houses are all in involved in varying degrees of litigation (all the way to class action lawsuits) with the big publishing houses. I think they at least have some level of filtering for their training data to keep them legally somewhat compliant. But considering how much copyrighted stuff is spread blisfully online, it is probably not enough to filter out the actual ebooks of certain publishers.

> I think they at least have some level of filtering for their training data to keep them legally somewhat compliant.

So far, courts are siding with the "fair use" argument. No need to exclude any data.

https://natlawreview.com/article/anthropic-and-meta-fair-use...

"Even if LLM training is fair use, AI companies face potential liability for unauthorized copying and distribution. The extent of that liability and any damages remain unresolved."

https://www.whitecase.com/insight-alert/two-california-distr...

Re: Claude Opus 4.6

#958

Post by zmmmmm »

I'm finding it quite a lot more assertive. It's doing things without asking every now and then. It cleaned up a whole lot of commented out of code that was unrelated to the change it was asked to make. Yes it's not great to have sections of commented out code, but destructive changes really should never be happening outside the scope of what it is asked to do.

And it refuses to do things it doesn't think are on task - I asked it to write a poem about cookies related to the code and it said:

> I appreciate the fun request, but writing poems about cookies isn't a code change — it's outside the scope of what I should be doing here. I'm here to help with code modifications.

I don't think previous models outright refused to help me. While I can see how Anthropic might feel it is helpful to focus it on task, especially for safety reasons, I'm a little concerned at the amount of autonomy it's exhibiting due to that.

Re: Claude Opus 4.6

#959

Post by tedsanders »

thinkingtoilet wrote: post #920
tedsanders wrote: post #641

Earlier quoted context omitted.

Are you referring to FrontierMath? We had access to the eval data (since we funded it), but we didn't train on the data or otherwise cheat. We didn't even look at the eval results until after the model had been trained and selected.

No one believes you.

If you don't believe me, that's fair enough. Some pieces of evidence that might update you or others:

- a member of the team who worked with this eval has left OpenAI and now works at a competitor; if we cheated, he would have every incentive to whistleblow

- cheating on evals is fairly easy to catch and risks destroying employee morale, customer trust, and investor appetite; even if you're evil, the cost-benefit doesn't really pencil out to cheat on a niche math eval

- Epoch made a private held-out set (albeit with a different difficulty); OpenAI performance on that set doesn't suggest any cheating/overfitting

- Gemini and Claude have since achieved similar scores, suggesting that scoring ~40% is not evidence of cheating with the private set

- The vast majority of evals are open-source (e.g., SWE-bench Pro Public), and OpenAI along with everyone else has access to their problems and the opportunity to cheat, so FrontierMath isn't even unique in that respect

Re: Claude Opus 4.6

#960

Post by bossyTeacher »

codexon wrote: post #946
netdevphoenix wrote: post #839

Earlier quoted context omitted.

Waiting until the moment they get good enough is not a smart thing to do either. If you are a farmer and know it is going to snow, at some point in the next 5 months, you make plans NOW, you don't wait until the temperatures drop and you see the snow falling. Right now, people are waiting for the snowfall before moving their proverbial chickens indoors

Top AI researchers like Yann LeCunn have said that LLMs are a dead end. It seems to me that LLM performance is plateuing and not improving exponentially anymore. This recent hubbub about rewriting a worse GCC for $20,000 is another example of overhype and regurgitating training data. You don't know for sure if it is going to "snow" (AI reaches general intelligence) Snow happens frequently, AI reaching general intelli…

There is no reason why an AI model capable of pushing a significant chunk of devs into lower paid and highly competitive dev jobs as a result of automation needs to be a general artificial intelligence. There is a lack of nuance that comes with thinking that either AI is dumb or it has human level general intelligence. As much as devs hate to admit it, you don't need that much of what we understand as general intelligence to write software. Only a portion of your intelligence is needed and arguably not all of it at the same time.

While general purpose models might be plateauing soon (arguably they have for a while). Highly specialised models (especially for programming) haven't necessarily plateaud yet. And anyway, existing functionality seem like a good foundation to build upon systems that remove the need of hiring as many devs. It's not the "being out of a job" that should worry you. Open up your binary thinking and consider that facing a 08 job market for the rest of your career is not the same permanent unemployment but it is not a market you would like to have.

That is the real concern.

Post reply on HN