Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Sonnet 4 now supports 1M tokens of context

anthropic.com

Post reply on HN
Print view View original

651–660 of 706 posts

Re: Claude Sonnet 4 now supports 1M tokens of context

#651

Post by giancarlostoro »

mamp wrote: post #211
giancarlostoro wrote: post #136

Earlier quoted context omitted.

Here's a paper from MIT that covers how this could be resolved in an interesting fashion: https://hanlab.mit.edu/blog/streamingllm The AI field is reusing existing CS concepts for AI that we never had hardware for, and now these people are learning how applied Software Engineering can make their theoretical models more efficient. It's kind of funny, I've seen this in tech over and over. People discover new thing, the…

Unfortunately, I think the context rot paper [1] found that the performance degradation when context increased still occurred in models using attention sinks. 1. https://research.trychroma.com/context-rot

Saw that paper have not had a chance to read it yet, are there other techniques that help then? I assume theres a few different ones used.

Re: Claude Sonnet 4 now supports 1M tokens of context

#652

Post by mrits »

sdesol wrote: post #638
mrits wrote: post #602

Earlier quoted context omitted.

How is that better than just writing a line in the md?

I am not sure I follow what you are saying. What would the line be and how would it become deterministically searchable?

frontend path: /src/frontend/* backend path: /src/*

I suppose the problem you have might be unique to nextJS ?

Re: Claude Sonnet 4 now supports 1M tokens of context

#653

Post by brookst »

HarHarVeryFunny wrote: post #642
aliljet wrote: post #16

This is definitely one of my CORE problem as I use these tools for "professional software engineering." I really desperately need LLMs to maintain extremely effective context and it's not actually that interesting to see a new model that's marginally better than the next one (for my day-to-day). However. Price is king. Allowing me to flood the context window with my code base is great, but given that the price has su…

Even 1 MB context is only roughly 20K LOC so pretty limiting, especially if you're also trying to fit API documents or any other lengthy material into the context. Anthropic also recently said that they think that longer/compressed context can serve as an alternative (not sure what was the exact wording/characterization they used) to continual/incremental learning, so context space is also going to be competing with…

1M tokens ~= 3.5M characters ~= 58k LOC at an average of 60 chars/line. 88k LOC at 40 chars/line

Re: Claude Sonnet 4 now supports 1M tokens of context

#654

Post by HarHarVeryFunny »

brookst wrote: post #653
HarHarVeryFunny wrote: post #642

Earlier quoted context omitted.

Even 1 MB context is only roughly 20K LOC so pretty limiting, especially if you're also trying to fit API documents or any other lengthy material into the context. Anthropic also recently said that they think that longer/compressed context can serve as an alternative (not sure what was the exact wording/characterization they used) to continual/incremental learning, so context space is also going to be competing with…

1M tokens ~= 3.5M characters ~= 58k LOC at an average of 60 chars/line. 88k LOC at 40 chars/line

OK - I was thinking 1M chars (@ 50 chars/line) vs tokens, but I'm not sure it makes much difference to the argument. There are plenty of commercial code bases WAY bigger, and as noted other things may also be competing for space in the context.

Re: Claude Sonnet 4 now supports 1M tokens of context

#655

Post by brookst »

lll-o-lll wrote: post #395
Workaccount2 wrote: post #207

Earlier quoted context omitted.

Humans have a neuro-chemical system that performs operations with electrical signals. That's the level to look at, unless you have a dualist view of the brain (we are channeling a super-natural forces).

Yep, just like like looking at a birds feather through a microscope explains the principles of flight… Complexity theory doesn’t have a mathematics (yet), but that doesn’t mean we can’t see that it exists. Studying the brain at the lowest levels haven’t lead to any major insights in how cognition functions.

I personally believe that quantum effects play a role and we’ll learn more once we understand the brain at that level, but I recognize that is an intuition and may well be wrong.

Re: Claude Sonnet 4 now supports 1M tokens of context

#656

Post by GuB-42 »

csomar wrote: post #562
GuB-42 wrote: post #511

Earlier quoted context omitted.

I added this to my toolbox in addition to traditional linters. My experience is that it is about 10% harmful, 80% useless and 10% helpful. Which is actually great, the 10% is worth it, but it is far from a hands off experience. By harmful I mean something like suggesting a wrong fix to code that works, it usually happens when I am doing something unusual or counter intuitive, for example having a function "decrease_x…

> it usually happens when I am doing something unusual or counter intuitive, That’s usually your signal that your code needs refactoring.

I wouldn't say it needs refactoring. Maybe more documentation, or some work on naming. But I believe that code you write has to be at least a bit unusual.

Every project worth making is unique. Otherwise, why not use something off the shelf?

For example, let's say you want to shuffle songs for a music player, you write your shuffling algorithm and it is "wrong", but there is a reason it is "wrong": it better matches the expectations of the user than a truly random shuffle. A LLM trained on thousands of truly random shuffles may try to "fix" your code, but it is actually the worst thing you can do. That "wrong" shuffle is the reason why you wrote that code in the first place, the "wrongness" is what adds value. But now, imagine that you realize that a true random shuffle is actually the way to go, then "fixing" your code is not what you should do either, instead, you should delete it and use the shuffle function your standard library offers.

The unusual/unique/surprising parts of your code is where the true value is, and if there is none of that in your codebase, maybe you are just reinventing the wheel. Now, if a LLM trips off these parts, maybe you need some documentation, as a way to tell both the LLM and a human reading that part that it is something you should pay attention to. I am not a fan of comments in general, but that's where they are useful: explaining why you wrote that weird code, something along the lines of "I know it is not the correct algorithm, but users prefer it that way".

Re: Claude Sonnet 4 now supports 1M tokens of context

#657

Post by Vegenoid »

csomar wrote: post #562
GuB-42 wrote: post #511

Earlier quoted context omitted.

I added this to my toolbox in addition to traditional linters. My experience is that it is about 10% harmful, 80% useless and 10% helpful. Which is actually great, the 10% is worth it, but it is far from a hands off experience. By harmful I mean something like suggesting a wrong fix to code that works, it usually happens when I am doing something unusual or counter intuitive, for example having a function "decrease_x…

> it usually happens when I am doing something unusual or counter intuitive, That’s usually your signal that your code needs refactoring.

While this can be true, I think a lot of people are far too eager to say that because someone is doing something in an unusual way, it's probably wrong. Not everything fits the cookie cutter model, there is tons of software written for all kinds of purposes. Suggesting that they're writing code wrong when someone says "an LLM struggles with my unusual code", when we aren't actually looking at the code and the context, is not helpful.

Re: Claude Sonnet 4 now supports 1M tokens of context

#658

Post by bamboozled »

mattmanser wrote: post #595
bamboozled wrote: post #530

Earlier quoted context omitted.

I always ask it to read the last 5 commits and anaylize and modified or staged files, works well...

Why do you find this better than just starting again at that point? I'm trying to understand the benefit of using this 'trick', without being able to try it as I'm away from my computer. Couldn't you start a new context and achieve the same thing, without any of the risks of this approach?

LLMs have no "memory" so it generally gives it something to go off, I forgot to add that I only do this if the change i'm making is related to whatever I did yesterday.

I do this because sometimes I just manually edit code and the LLM doesn't know everything that's happened.

I also find the. best way to work with "AI" is to make very small changes and commit frequently, I truly think it's a slot machine and if it does go wild, you can lose hours of work.

Re: Claude Sonnet 4 now supports 1M tokens of context

#659

Post by t43562 »

I think this highlights some problems with software development in general. i.e. the code isn't enough - you need to have domain knowledge too and a lot of knowledge about how and why the company needs things done in some way or another. You might imagine that dumping the contents of your wiki and all your chat channels into some sort of context might do it but that would miss the 100s of verbal conversations between people in the company. It would also fall foul of the way everything tends to work in any way you can imagine except what the wiki says.

Even if you transcribed all the voice chats and meetings and added it in, it challenges a human to work out what is going on. No-context human developers are pretty useless too.

Re: Claude Sonnet 4 now supports 1M tokens of context

#660

Post by jimbokun »

scott_s wrote: post #644
ehnto wrote: post #627

Earlier quoted context omitted.

> If you're completely new to the problem then ... yes, it does. Of course, because I am not new to the problem, whereas an LLM is new to it every new prompt. I am not really trying to find a fair comparison because I believe humans have an unfair advantage in this instance, and am trying to make that point, rather than compare like for like abilities. I think we'll find even with all the context clues from MCPs and…

> Of course, because I am not new to the problem, whereas an LLM is new to it every new prompt. That is true for the LLMs you have access to now. Now imagine if the LLM had been trained on your entire code base. And not just the code, but the entire commit history, commit messages and also all of your external design docs. And code and docs from all relevant projects. That LLM would not be new to the problem every pr…

Why haven’t the bug AI companies been pursuing that approach, vs just ramping up context window size?
Post reply on HN