Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SubQ 1.1 Small

subq.ai

Post reply on HN
Print view View original

11–20 of 54 posts

Re: SubQ 1.1 Small

#11

Post by wxw »

> SSA replaces the O(n²) dense attention pass with a learned sparse formulation that scales linearly with context length.

> At 1M tokens, SubQ 1.1 Small requires 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2.

Awesome stuff. Solving context at the model architecture layer rather than trying to bolt on extra memory is the right direction IMO.

Re: SubQ 1.1 Small

#13

Post by supern0va »

cmogni1 wrote: post #6

I don’t understand why this lab is allergic to providing details on what they actually made, especially when Chinese labs are more than willing to share architectural specs/code/kernels (eg NSA/FSA, RAMBa, HISA, DSA LightningIndexer, etc). I don’t doubt that they’ve done something here, but the lack of details makes me default not trust this, particularly when this is the second time that they’ve released a “technica…

You don't understand why the thing their entire company is valued upon is...not being given away freely? They literally are taking an open source model and then adapting it with this technique. If they disclose it, the frontier labs will immediately copy it and outperform them.

My guess is that they're angling for an acquisition.

Re: SubQ 1.1 Small

#15

Post by robmccoll »

giancarlostoro wrote: post #9
robmccoll wrote: post #4

Earlier quoted context omitted.

Yes - I want that and dramatically faster. Newer models don't seem to need any more or less guidance and iteration, so let's make the time-to-wrong-answer as short as possible.

I'm not as crazy about speed as long as it's reasonably as "quick" as Opus. Which is faster than most developers can spit out code. I do get annoyed with Claude Code because it looks like it chooses to be as slow as possible, but maybe that's by design so its not pounding their backend every milisecond? Would probably be bad. Local inference is insanely fast on my M4 Pro MBP though, so I can understand where you're c…

I just don't want to have to have a pipeline going in order to fully occupy my time. I don't want to wait on the model to review the prompt, read the parts of the codebase indicated, do its own research in the codebase and documentation, plan, run agents ... actually write the code and NOW I can start reading it and reviewing it. That means I either need to run a lot of operations in parallel so that I always have something to do and the agent(s) are highly utilized or I'm writing something on my own that I keep getting that keeps getting interrupted. It's the constant context switching that kills me. I want to work on one problem at a time and really focus on it - even if I'm not writing every line myself.

Re: SubQ 1.1 Small

#16

Post by maz1b »

They've done multiple "evaluations" by third parties, but still, it seems that they aren't being fully transparent. I think the approach is quite interesting and novel, but this feels like deja vu.

I get why they aren't disclosing all the details, but it seems more hype-train-esque to me for this moment. I don't disagree that this could be big.

Re: SubQ 1.1 Small

#17

Post by Depurator »

What kind of hardware would be needed to serve an instance with the full 12m context? And what kind of speeds can one expwct at those extremes at 10m+?

Re: SubQ 1.1 Small

#18

Post by monster_truck »

embedding-shape wrote: post #7

> SubQ 1.1 Small scores near-perfect at 1M, 2M, 6M, and 12M tokens. The model was trained predominantly at 1M tokens yet the retrieval held near perfectly at 12x that length, despite compressing attention to just 0.13% of relationships. This generalization is a direct consequence of SSA routing attention based on content relevance rather than fixed positional patterns. If the results persists from 1M to 12M, why not…

It likely falls off very steeply after that. 8 to 1 (which I am assuming based on the 0.13% figure) is a pretty common ratio for sparse matrix stuff.

Re: SubQ 1.1 Small

#19

Post by samber »

According to Subquadratic, Needle in a Haystack is strong up to 12m tokens, but RULER has not been tested above 128k tokens ??

Re: SubQ 1.1 Small

#20

Post by samber »

Comparing compute cost versus FlashAttention-2 is not very honest to me.

FlashAttention-2 is not used anymore for at least 2y.

This architecture would have been a massive improvement 3 years ago, but it is a ~solved~ problem IMO.

Post reply on HN