Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Sonnet 4 now supports 1M tokens of context

anthropic.com

Post reply on HN
Print view View original

661–670 of 706 posts

Re: Claude Sonnet 4 now supports 1M tokens of context

#661

Post by jimbokun »

airbreather wrote: post #542
ehnto wrote: post #532

Earlier quoted context omitted.

I disagree, I may not have the whole codebase in my head in one moment but I have had all of it in my head at some point, and it is still there, that is not true of an LLM. I use LLMs and am impressed by them, but they just do not approximate a human in this particular area. My ability to break a problem down does not start from listing the files out and reading a few. I have a high level understanding of the whole p…

you will have abstractions - black boxing, interface overviews etc, humans can only hold so much detail in current context memory, some say 7 items on average.

And LLMs can’t leverage these abstractions nearly as well as humans…so far.

Re: Claude Sonnet 4 now supports 1M tokens of context

#662

Post by jimbokun »

xwolfi wrote: post #541
ehnto wrote: post #532

Earlier quoted context omitted.

I disagree, I may not have the whole codebase in my head in one moment but I have had all of it in my head at some point, and it is still there, that is not true of an LLM. I use LLMs and am impressed by them, but they just do not approximate a human in this particular area. My ability to break a problem down does not start from listing the files out and reading a few. I have a high level understanding of the whole p…

You only worked on very small codebase then. When you work on giant ones, you Ctrl+F a lot, build a limited model of the problem space, and pray the unit tests will catch anything you might have missed...

Why are LLMs so bad at doing the same thing?

Re: Claude Sonnet 4 now supports 1M tokens of context

#663

Post by johnisgood »

krainboltgreene wrote: post #648
adastra22 wrote: post #556

Earlier quoted context omitted.

Not with pixel perfect accuracy. You vaguely remember, although it may not feel like that because your brain fills in the details (hallucinates) as you recall. The comparisons are closer than you might think.

You have no idea if I remember with pixel perfect accuracy (whatever that even means). There are plenty of people with photographic memory. Also, you're a programmer you have no foundation of knowledge on which to make that assessment. You might as well opine on quarks or martian cellular life. My god the arrogance of people in my industry.

> There are plenty of people with photographic memory.

I thought it was rare.

Re: Claude Sonnet 4 now supports 1M tokens of context

#664

Post by HarHarVeryFunny »

HarHarVeryFunny wrote: post #642
aliljet wrote: post #16

This is definitely one of my CORE problem as I use these tools for "professional software engineering." I really desperately need LLMs to maintain extremely effective context and it's not actually that interesting to see a new model that's marginally better than the next one (for my day-to-day). However. Price is king. Allowing me to flood the context window with my code base is great, but given that the price has su…

Even 1 MB context is only roughly 20K LOC so pretty limiting, especially if you're also trying to fit API documents or any other lengthy material into the context. Anthropic also recently said that they think that longer/compressed context can serve as an alternative (not sure what was the exact wording/characterization they used) to continual/incremental learning, so context space is also going to be competing with…

Just as a self follow-up, another motivation to lean on tool use rather than massive context (cf. short-term memory) is to keep LLM/AI written/modified code understandable to humans ...

At least part of the reason that humans use hierarchical decomposition and divide-and-conquor is presumably because of our own limited short term memory, since hierarchical organization (modules, classes, methods, etc) allows us to work on a problem at different levels of abstraction while only needing to hold that level of the hierarchy in memory.

Imagine what code might look like if written by something with no context limit - just a flat hierarchy of functions, perhaps, at least until it perhaps eventually learned, or was told, the other reasons for hierarchical and modular design/decomposition to assist in debugging and future enhancement, etc!

Re: Claude Sonnet 4 now supports 1M tokens of context

#665

Post by jimbokun »

vidarh wrote: post #600
adastra22 wrote: post #556

Earlier quoted context omitted.

Not with pixel perfect accuracy. You vaguely remember, although it may not feel like that because your brain fills in the details (hallucinates) as you recall. The comparisons are closer than you might think.

The comparison would be apt if the LLM was trained on your codebase .

Isn’t that the problem?

I don’t see any progress on incrementally training LLMs on specific projects. I believe it’s called fine tuning, right?

Why isn’t that the default approach anywhere instead of the hack of bigger “context windows”?

Re: Claude Sonnet 4 now supports 1M tokens of context

#666

Post by jimbokun »

brulard wrote: post #473
ants_everywhere wrote: post #355

Earlier quoted context omitted.

I exhaust the 1 million context windows on multiple models multiple times per day. I haven't used the Llama 4 10 million context window so I don't know how it performs in practice compared to the major non-open-source offerings that have smaller context windows. But there is an induced demand effect where as the context window increases it opens up more possibilities, and those possibilities can get bottlenecked on r…

I think you misunderstand how context in current LLMs works. To get the best results you have to be very careful to provide what is needed for immediate task progression, and postpone context thats needed later in the process. If you give all the context at once, you will likely get quite degraded output quality. Thats like if you want to give a junior developer his first task, you likely won't teach him every corner…

It seems like LLM need to become experts at managing their OWN context.

Selectively gripping and searching the code to pull into context only those parts relevant to the task at hand.

Re: Claude Sonnet 4 now supports 1M tokens of context

#667

Post by sdesol »

mrits wrote: post #652
sdesol wrote: post #638

Earlier quoted context omitted.

I am not sure I follow what you are saying. What would the line be and how would it become deterministically searchable?

frontend path: /src/frontend/* backend path: /src/* I suppose the problem you have might be unique to nextJS ?

The issue is frontend can be a loaded question, especially if you are dealing with legacy stuff, different frameworks, etc. You also can't tell what the frontend code does by looking at that single line.

Now imagine as part of your analyzer, you have the following instructions for the llm:

--- For all files in `src/frontend/` treat them as frontend code. For all files in 'src/' excluding `src/frontend` treat as backend. Create a metadata called `scope` which can be 'frontend', 'backend' or 'mix' where mix means the code can be used for both front and backend like utilities.

Now for each file, create a `keywords` metadata that includes up to 10 unique keywords that describes the core functionality for the file. ---

So with this you can say

- `!ask find all frontend files`

- `!ask find all mix use files`

- `!ask find all frontend files that does [this]`

and so forth.

The whole point of analyzers is to make it easy for the LLM to map your natural language query to a deterministic search.

If the code base is straightforward and follows a well known framework, asking for frontend or backend wouldn't even need an entry as you can just include in the instructions that I use framework X and the LLM would know what to consider.

Re: Claude Sonnet 4 now supports 1M tokens of context

#668

Post by didibus »

photon_lines wrote: post #206
didibus wrote: post #129

Earlier quoted context omitted.

You seem possibly more knowledgeable then me on the matter. My impression is that LLMs predict the next token based on the prior context. They do that by having learned a probability distribution from tokens -> next-token. Then as I understand, the models are never reasoning about the problem, but always about what the next token should be given the context. The chain of thought is just rewarding them so that the nex…

So - in the pre-training step you are right -- they are simple 'statistical' predictors but there are more steps involved in their training which turn them from simple predictors to being able to capture patterns and reason -- I tried to come up with an intuitive overview of how they do this in the write-up and I'm not sure I can give you a simple explanation here, but I would recommend you play around with Deep-Seek…

If you observe the failure modes of current models, you see that they fail in ways that align with probabilistic token prediction.

I don't mean that the textual prediction is simple, it's very advanced and it learns all kinds of relationships, patterns and so on.

But it doesn't have a real model and thinking process relating to the the actual problem. It thinks about what text could describe a solution that is linguistically and language semantically probable.

Since human language embedds so many of the logics and ground truths that's good enough to result in a textual description that approximate or nails the actual underlying problem.

And this is why we see them being able to solve quite advanced problems.

I admit that people are wondering now, what's different about human thinking? Maybe we do the same, you invent a probable sounding answer and then check if it was correct, rinse and repeat until you find one that works.

But this in itself is a big conjecture. We don't really know how human thinking works. We've found a method that works well for computers and now we wonder if maybe we're just the same but scaled even higher or with slight modifications.

I've heard from ML experts though that they don't think so. Most seem to believe different architecture will be needed, world models, model ensembles with various specialized models with different architecture working together, etc. That LLMs fundamentaly are kind of limited by their nature as next token predictors.

Re: Claude Sonnet 4 now supports 1M tokens of context

#669

Post by ericmcer »

qingcharles wrote: post #378
ericmcer wrote: post #275

Earlier quoted context omitted.

Agreed, daily Cursor user. Just got out of a 15m huddle with someone trying to understand what they were doing in a PR before they admitted Claude generated everything and it worked but they weren't sure why... Ended up ripping about 200 LoC out because what Claude "fixed" wasn't even broken. So never let it generate code, but the autocomplete is absolutely killer. If you understand how to code in 2+ languages you ca…

The other day I caught it changing the grammar and spelling in a bunch of static strings in a totally different part of a project, for no sane reason.

That sounds similar to what it was doing here. It basically took a function like `thing = getThing(); id = thing.id` and created `id = getThingId()` and replaced hundreds of lines and made a new API endpoint.

Not a huge deal because it works, but it seems like you would have 100,000 extra lines if you let Claude do whatever it wanted for a few months.

Re: Claude Sonnet 4 now supports 1M tokens of context

#670

Post by didibus »

coderenegade wrote: post #521
didibus wrote: post #129

Earlier quoted context omitted.

You seem possibly more knowledgeable then me on the matter. My impression is that LLMs predict the next token based on the prior context. They do that by having learned a probability distribution from tokens -> next-token. Then as I understand, the models are never reasoning about the problem, but always about what the next token should be given the context. The chain of thought is just rewarding them so that the nex…

I think the intuitive leap (or at least, what I believe) is that meaning is encoded in the media. A given context and input encodes a particular meaning that the model is able to map to an output, and because the output is also in the same medium (tokens, text), it also has meaning. Even reasoning can fit in with this, because the model generates additional meaningful context that allows it to better map to an output…

Yes exactly.

The text of human natural language that it is trained on encodes the solutions to many problems as well as a lot of ground truths.

The way I think of it is. First you have a random text generator. This generative "model" in theory can find the solution to all problems that text can describe.

If you had a way to assert if it found the correct solution, you could run it and eventually it would generate the text that describes the working solution.

Obviously inefficient and not practical.

What if you made it so it skipped generating all text that aren't valid sensical English?

Well now it would find the correct solution in way less iterations, but still too slow.

What if it generated only text that made sense to follow the context of the question?

Now you might start to see it 100-shot, 10-shot, maybe even 1-shot some problems.

What if you tuned that to the max? Well you get our current crop of LLMs.

What else can you do to make it better?

Tune the dataset, remove text that describe wrong answers to prior context so it learns not to generate those. Add more quality answers to prior context, add more problems/solutions, etc.

Instead of generating the answer to a mathematical equation the above way, generate the Python code to run to get the answer.

Instead of generating the answer to questions about current real world events/facts (like the weather). Have it generate the web search query to find it.

If you're asking a more complex question, instead of generating the answer directly, have it generate smaller logical steps towards the answer.

Etc.

Post reply on HN