Viewing profile — chillee

I'd also note that you can more or less write the same code in PyTorch with torch.vmap

this is almost certainly not being done on cerebras

I think unlike Gluon/CuTe/ThunderKittens (which distinguish themselves from Triton by being lower level giving you more control, thus being less performance portable and harder to …

What's the point of Triton compared to Gluon? What's the point of PyTorch compared to Triton? One of the main values of Triton is that it significantly expanded the scope of folks …

If you think of Triton as a "baseline", most other DSLs are lower-level than Triton, whereas this is higher-level.

Their gross profits are very high even though they're not making operating profit.

Clearly not true anymore given OpenAI and Anthropic's revenue growth.

The 32 parallel sequences is also arbitrary and significantly changes your conclusions. For example, if they run with 256 parallel sequences then that would result in a 8x cheaper …

No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it…

I mean, vllm and sglang are both "pure python" essentially as well. But yeah, in ML you rarely require C++ to get good performance for most of the systems people are writing.

A couple things: 1. The academy has had a significant increase of young voters in the past 10 years or so. Generally speaking, young voters are more likely to take animation as a "…

https://www.theinformation.com/articles/to-reduce-ai-costs-g... Which takes it from > Broadcom generates a 70% profit margin from its work on TPUs, said a person with direct knowle…

One of the big things this article misses is that Google pays Broadcom a significant amount for the actual chip design, also around a 70% margin. Google certainly has infra/cost ad…

For latency-bound inference (i.e. one request) you don't need tensor-cores since all your operations are just matrix vector multiplications.

The big issue with Strassen isn't performance - it's numerical stability.

Well, that'll help with a lot :) But dealing with wave quantization requires dimensions that aren't neceessarily a multiple of 2, and often are a multiple of the number of SMs on a…

It's very much in the "worse is better" camp.

It's always fun when people use autodiff in packages like PyTorch for completely unrelated usecases :)

> it wouldn’t use functions like these and the generated comparable code would be on-pare performance wise Perhaps if XLA generated all functions from scratch, this would be more c…

To be clear, this performance is quite bad (presumably because you didn't manage to get compilation working). You're getting 35 tokens/s for a 405B model, which comes out to about …

> And was trying to make a broader point about the lack of transparency (in performance, lower-level impl) in PyTorch when running on NVIDIA vs. non-NVIDIA hardware. I don't quite …

I believe it should run on V100 as well (although definitely not tested as well), and an user reported that they got it running on T4 too.

Ah sorry, should have put that in the blog post. This leverages Triton heavily, so it'll only work on machines that have Triton backends (at least, we've tested on Nvidia and AMD G…

In my (very amateurish) testing, I think the performance seemed pretty comparable (for non-dilated natten). I need to do some proper benchmarking though!