The context window has been shattered: Subquadratic debuts a 12M token window
1–10 of 15 posts
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#2I believe it, when I see it.
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#3Have they published?
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#4Previous discussion here (with links to actual primary source):
https://news.ycombinator.com/item?id=48023079
No technical report published yet, unlikely code or weights will be either given VC funding.
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#5It’s probably something like deepseek’s native sparse attention with content based granularity. They might not be publishing anything because it’s not such a strong value proposition and doing so would lead to commentary that would tank their investment opportunities.
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#6[dead]
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#7It’s probably something like deepseek’s native sparse attention with content based granularity. They might not be publishing anything because it’s not such a strong value proposition and doing so would lead to commentary that would tank their investment opportunities.
Or maybe because giving it away would tank their investment opportunities.
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#8It’s probably something like deepseek’s native sparse attention with content based granularity. They might not be publishing anything because it’s not such a strong value proposition and doing so would lead to commentary that would tank their investment opportunities.
Or maybe because giving it away would tank their investment opportunities.
There's ways and means. Pushing something out in the sub-30B range would gain them mindshare and they could keep bigger models to themselves. I can't see any indication of what size their model is though.
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#9For Claude Code, I feel 1M is enough. I've had a compaction once, but that was because I was forcing Claude to do something it clearly had a hard time understanding.
For general chat bots where the user doesn't understand what a context window is, what do you do about context? Latest few messages and then a memory tool? Compaction?
Re: The context window has been shattered: Subquadratic debuts a 12M token window
#10For Claude Code, I feel 1M is enough. I've had a compaction once, but that was because I was forcing Claude to do something it clearly had a hard time understanding. For general chat bots where the user doesn't understand what a context window is, what do you do about context? Latest few messages and then a memory tool? Compaction?
Claude does compaction in the regular web chat interface now, too