SubQ 1.1 Small • Hacker News Board

Re: SubQ 1.1 Small

#31

Post by cmogni1 » Tue, Jun 16, 2026, 5:46 PM UTC

supern0va wrote: post #13

cmogni1 wrote: post #6

I don’t understand why this lab is allergic to providing details on what they actually made, especially when Chinese labs are more than willing to share architectural specs/code/kernels (eg NSA/FSA, RAMBa, HISA, DSA LightningIndexer, etc). I don’t doubt that they’ve done something here, but the lack of details makes me default not trust this, particularly when this is the second time that they’ve released a “technica…

You don't understand why the thing their entire company is valued upon is...not being given away freely? They literally are taking an open source model and then adapting it with this technique. If they disclose it, the frontier labs will immediately copy it and outperform them. My guess is that they're angling for an acquisition.

Ahh cf my comment above. The cost of failure at scale is too high for a major to just take a new architecture/mechanism and implement it, especially because a) most claims papers make aren't rigorously tested and b) plenty of things that work at one scale do not work at the scale on which the labs operate. If they want to get acquired, then they should show that they know what they're doing. Otherwise, it looks sketchy.

Re: SubQ 1.1 Small

#32

Post by mstkllah » Tue, Jun 16, 2026, 5:49 PM UTC

mritchie712 wrote: post #28

giancarlostoro wrote: post #3

This one's interesting, and I think the next frontier for LLMs should really just be, how can we get something like Opus 4.6 to cost drastically less, for the same output? I say 4.6 because from 4.6 onwards it's been pretty darn good, at least for me, always feels like every model upgrade someone hates it, heck even 4.5 was fine.

I agree on opus 4.5-4.8, but Fable 5 was a noticeable upgrade.

Did not feel as an upgrade to me at all, felt way slower at the same quality level as 4.8 to me.

Re: SubQ 1.1 Small

#33

Post by supern0va » Tue, Jun 16, 2026, 6:10 PM UTC

cmogni1 wrote: post #31

supern0va wrote: post #13

Earlier quoted context omitted.

You don't understand why the thing their entire company is valued upon is...not being given away freely? They literally are taking an open source model and then adapting it with this technique. If they disclose it, the frontier labs will immediately copy it and outperform them. My guess is that they're angling for an acquisition.

Ahh cf my comment above. The cost of failure at scale is too high for a major to just take a new architecture/mechanism and implement it, especially because a) most claims papers make aren't rigorously tested and b) plenty of things that work at one scale do not work at the scale on which the labs operate. If they want to get acquired, then they should show that they know what they're doing. Otherwise, it looks sketc…

>The cost of failure at scale is too high for a major to just take a new architecture/mechanism and implement it,

Is it, though? This scrappy startup was able to take a large(-ish) open weights model and adapt it. Why can't the frontier labs do the same cost effectively?

>If they want to get acquired, then they should show that they know what they're doing.

I'm sure they would do so under an appropriate NDA as part of negotiations. I'm not sure why you think a full public disclosure is necessary.

Re: SubQ 1.1 Small

#34

Post by famouswaffles » Tue, Jun 16, 2026, 6:37 PM UTC

cmogni1 wrote: post #30

famouswaffles wrote: post #10

Earlier quoted context omitted.

Business wise, it would make sense to hold off on details till they're at least ready to serve. Look at what happened with Open AI and reasoning models. Everyone struggled with getting RL to work with LLMs for a good while. Open AI figured it out, and a few months later everyone had their prototypes out in short order. Don't forget who these labs employ. They're some of the brightest people around. Sub-q aren't reall…

I don't think it makes sense from a business perspective to hold off on details as a new lab. OpenAI will not implement new architectural changes unless they've tested the changes themselves internally. Even if someone claims some great innovation, they'd need to do scaling experiments to somewhere between the size of GPT-4 to GPT-5 before they'd decide it is worth it to implement themselves. Plenty of mechanisms tha…

>OpenAI will not implement new architectural changes unless they've tested the changes themselves internally.

OpenAI validating it can still happen faster than they can get the compute to serve the models themselves[1]. It doesn't make a lot of sense to give out details if they want to be a serious contender or even as some have said, be acquired.

Yeah there's noise but if they have the real deal then it doesn't matter. They only thing they need to do is let people pay to use the models.

[1] I'm assuming this is the primary cause of the delay. That may not be the case of course.

Re: SubQ 1.1 Small

#35

Post by jmward01 » Tue, Jun 16, 2026, 6:48 PM UTC

jmward01 wrote: post #22

cmogni1 wrote: post #6

I don’t understand why this lab is allergic to providing details on what they actually made, especially when Chinese labs are more than willing to share architectural specs/code/kernels (eg NSA/FSA, RAMBa, HISA, DSA LightningIndexer, etc). I don’t doubt that they’ve done something here, but the lack of details makes me default not trust this, particularly when this is the second time that they’ve released a “technica…

Well, I know this is possible because I have built things that work just like it is promising to do. The two key technologies needed are: - guided window attn. Predict where to attend to but in a fixed window. If you do this to just the token/vocab you can keep effectively unlimited context and perfect recall. (yes, I can do that. There is a trick to teaching it how to predict position. This also immediately opens ot…

As a follow-up, I can see there is not a lot of belief which is why it is also hard to find a company to partner with on this. So, how -do- you make money on something like this as an independent researcher. Maybe I release trick one, show how guided window attn (and nn memory and probably a lot of robotics) can be trained? Thoughts? I can do that pretty quickly. By itself that is a pretty great tech (combined with fixed windows of full attn it is pretty amazing). The second trick, I think, is a bit more powerful although both are general purpose. If I do this, think people will believe trick two (and all the real time multi-modal streaming stuff)?

Re: SubQ 1.1 Small

#36

Post by jmward01 » Tue, Jun 16, 2026, 6:55 PM UTC

bratao wrote: post #26

jmward01 wrote: post #22

Earlier quoted context omitted.

Well, I know this is possible because I have built things that work just like it is promising to do. The two key technologies needed are: - guided window attn. Predict where to attend to but in a fixed window. If you do this to just the token/vocab you can keep effectively unlimited context and perfect recall. (yes, I can do that. There is a trick to teaching it how to predict position. This also immediately opens ot…

I´m super curious about those "Two Weird Tricks". I would like that you would release more. It remember me the MiniMax Sparse Attention https://arxiv.org/html/2606.13392v1

Yeah, looks like fun stuff. You still need to preserve the entire kv cache though right? So even if compute is drastically less, memory keeps growing. The system I described keeps memory constant (well, if you keep the entire token history you technically are gaining one long of data per token generated but I think we can agree that is negligible and could be capped at something high like 1B or so with no meaningful impact). I think I will probably release trick one and see if people then believe trick two even without seeing it.

Re: SubQ 1.1 Small

#37

Post by giancarlostoro » Tue, Jun 16, 2026, 7:33 PM UTC

yorwba wrote: post #29

cmogni1 wrote: post #6

I don’t understand why this lab is allergic to providing details on what they actually made, especially when Chinese labs are more than willing to share architectural specs/code/kernels (eg NSA/FSA, RAMBa, HISA, DSA LightningIndexer, etc). I don’t doubt that they’ve done something here, but the lack of details makes me default not trust this, particularly when this is the second time that they’ve released a “technica…

They don't need to provide any details at all. They just need to give people access to their model and charge them for it. That they don't do that and instead pay for external evaluations indicates that they believe people would be unimpressed if they could access the model directly. The only purpose of this press release seems to be making investors give them more money.

Could also be they don't have enough funding to sustain that many users, or even the infrastructure lined up.

Re: SubQ 1.1 Small

#38

Post by giancarlostoro » Tue, Jun 16, 2026, 7:37 PM UTC

eikenberry wrote: post #27

jmward01 wrote: post #22

Earlier quoted context omitted.

Well, I know this is possible because I have built things that work just like it is promising to do. The two key technologies needed are: - guided window attn. Predict where to attend to but in a fixed window. If you do this to just the token/vocab you can keep effectively unlimited context and perfect recall. (yes, I can do that. There is a trick to teaching it how to predict position. This also immediately opens ot…

Isn't the classic way of making money off an invention is to patent it... so why not patent those "Two Weird Tricks"?

Expensive and if someone figures out a slight different way to do it you arent really “covered” its not a unique umbrella plus you would sort of give away the secrets.

Re: SubQ 1.1 Small

#39

Post by cmogni1 » Tue, Jun 16, 2026, 7:53 PM UTC

supern0va wrote: post #33

cmogni1 wrote: post #31

Earlier quoted context omitted.

Ahh cf my comment above. The cost of failure at scale is too high for a major to just take a new architecture/mechanism and implement it, especially because a) most claims papers make aren't rigorously tested and b) plenty of things that work at one scale do not work at the scale on which the labs operate. If they want to get acquired, then they should show that they know what they're doing. Otherwise, it looks sketc…

>The cost of failure at scale is too high for a major to just take a new architecture/mechanism and implement it, Is it, though? This scrappy startup was able to take a large(-ish) open weights model and adapt it. Why can't the frontier labs do the same cost effectively? >If they want to get acquired, then they should show that they know what they're doing. I'm sure they would do so under an appropriate NDA as part o…

I don't mean to be shady, but there are plenty of details that they did release that show that they don't know what they're doing.

They make comparisons to FlashAttention-2 when FlashAttention-4 has been out (even if they wanted to stick to Hopper class GPUs for whatever reason there's still FlashAttention-3). The two orders of magnitude claim look like they're for prefill not next-token decoding, which is a bit duplicitous. Long context extrapolation experiments typically go well beyond 2x context length. Etc etc etc.

I never said they should have a full public disclosure, but I do think sharing something of substance helps build trust and also get people excited.

Lastly, frontier labs have other incentives than to eek out every dollar and cent. Having the most capable models, not the most cost effective, is of significantly higher priority as OpenAI and Anthropic march towards IPOs. The same is not necessarily true for Google/DeepMind, and one can see from their public releases alone for some of their open weight models that this may be more of a priority for them today.

Re: SubQ 1.1 Small

#40

Post by ballon_monkey » Tue, Jun 16, 2026, 7:59 PM UTC

Its funny that some people on HN think this whole thing is legit. The company is started by a bunch of no-bodies with 0 experience in AI in general let alone ML/Data.

Edit: Typical HN "I can downvote but I cannot dispute facts"