Viewing profile — thunderbird120

Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic. I already harped on C…

It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size…

This is a near perfect encapsulation of the exact type of person you should not take financial advice from.

Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallel…

Medical expenses would be a combination of cost of insurance / treatment (PPP relevant) and government transfers in a country with partially or fully public healthcare, which is wh…

I see. That paper is looking a median income from a PPP perspective but without accounting for taxes and transfers, which is a tiny bit unorthodox in this exact context which tries…

>median is higher in Germnay I cannot think of any standard by which this is true, certainly not by nominal or PPP income for either personal or household income. [1] https://en.wi…

Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV…

The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flex…

You can merge it into OpenWhip https://old.reddit.com/r/ClaudeAI/comments/1sek7gk/someone_m...

>Its turn around time is ridiculous, it has to be maintained with specialized equipment/hangers, along with external contractor assistance. >Compared to the Gripen, as an example, …

Cerebras has effectively 100% yield on these chips. They have an internal structure made by just repeating the same small modular units over and over again. This means they can jus…

That's what it's running on. It's optimized for very high throughput using Cerebras' hardware which is uniquely capable of running LLMs at very, very high speeds.

You prevent enemy air defenses from shooting down your aircraft by blowing them up as part of SEAD/DEAD missions, which is exactly what the US did.

Takes much longer to build, requires a much larger up-front investment, and requires a lot more land. The footprint needed when trying to generate this much power from solar or win…

Precision bombing during WW2 was not possible at the required scale. To put a bomb precisely on target back then you needed something like a dive bomber, a tactic which is incompat…

Nice to see that they added that, but that section wasn't in the article when I wrote that comment.

This article doesn't mention TPUs anywhere. I don't think it's obvious for people outside of google's ecosystem just how extraordinarily good the JAX + TPU ecosystem is. Google sev…

People would correctly identify that their standard of living is being reduced for ideological reasons without tangible individual benefits and would likely not respond well to tha…

Yeah, that's the normal outcome for papers like this. Papers which claim to be groundbreaking improvements on Transformers universally aren't. Same story roughly once a month for t…

If that were totally true you would expect to see more or less uniform ratios of HP/HD cells mixes across different product types, but that's very much not the case. Dennard scalin…

If it were that simple fabs wouldn't offer a standard cell libraries in both high performance and high density varieties. TSMC continues to provide both for their 2nm process. A tr…

>Intel on 18A is literally TSMC's 3nm process + backside power delivery, which means more power efficiency, performance also less heat. That's a pretty serious abuse of the word "l…

The cost for both training and inference is vaguely quadratic while, for the vast majority of users, the marginal utility of additional context is sharply diminishing. For 99% of C…

No, additional context does not cause exponential slowdowns and you absolutely can use FlashAttention tricks during training, I'm doing it right now. Transformers are not RNNs, the…