11–20 of 63 posts
> FlashAttention-3 is optimized for Hopper GPUs (e.g. H100). How does FA3 fare for consumer GPUs such as 3090 and 4090?
hoping an expert can answer a few Qs I have :) Is FlashAttention simply a drop-in replacement for the attention operation in an LLM? Can it be used anywhere that an "attention" operation is used? Or does a LLM need to be trained specially to use FA? How does FA relate to attention strategies like GQA (grouped query attention) or sliding-window attention? Are they orthogonal concepts? Or you need a specific FA impleme…
Yes
> How does FA relate to attention strategies like GQA (grouped query attention) or sliding-window attention? Are they orthogonal concepts? Or you need a specific FA implementation for each strategy?
Flash Attention is a way of calculating the Softmax(QK^T)V part of attention, whereas GQA is a way of calculating the Q, K, and V matricies. Sliding window attention (less sure about this, there are a bunch of windowed attention techniques) change the attention mask (the thing that controls which queries can attend to which keys).
> Recently llama.cpp added flash attention support - does this just mean they started consuming a flash attention-provided CUDA kernel or something?
I don't use llama.cpp but that sounds about right.
> lastly, in this post, they compare FlashAttention to Triton. I thought Triton was like an abstraction layer? Couldn't FA be implemented in Triton? I just don't really get what it means to say "FlashAttention vs. Triton".
They're talking about a previous Flash Attention implementation written in Triton.
hoping an expert can answer a few Qs I have :) Is FlashAttention simply a drop-in replacement for the attention operation in an LLM? Can it be used anywhere that an "attention" operation is used? Or does a LLM need to be trained specially to use FA? How does FA relate to attention strategies like GQA (grouped query attention) or sliding-window attention? Are they orthogonal concepts? Or you need a specific FA impleme…
2) Flash Attention 2 added support for GQA in past version updates:
https://github.com/Dao-AILab/flash-attention
3) They're comparing this implementation of Flash Attention (which is written in raw CUDA C++) to the Triton implementation of a similar algorithm (which is written in Triton): https://triton-lang.org/main/getting-started/tutorials/06-fu...
spoiler: $xxx,xxx hardware required to run
This is one of the most important improvements in all of AI, because it benefits most AI users by giving them access to more, faster, for the same hardware with little to no tradeoffs.
...for all those users with H100s.
[0] Small footprint time: before B100 ships; for actually large language models; for prefill only; may cause cancer in California.
Compiler folks: Is there any chance compilers will be able to find optimizations like FlashAttention on their own? Seems like TVM and tinygrad are working in that direction but I find it hard to believe that that would be feasible
> Seems like TVM
Fair enough, though technically they are still about different things but it's indeed very close, but
> and tinygrad
?????? what gives you this impression?
> FlashAttention-3 is optimized for Hopper GPUs (e.g. H100). How does FA3 fare for consumer GPUs such as 3090 and 4090?
It's Hopper-specific, the improvements are closely tied to Hopper features like warp groups and TMA. For 4090s, you might get a speedup by using the Triton implementation of FP8 attention: https://triton-lang.org/main/getting-started/tutorials/06-fu...
Compiler folks: Is there any chance compilers will be able to find optimizations like FlashAttention on their own? Seems like TVM and tinygrad are working in that direction but I find it hard to believe that that would be feasible
No. Think of it like a different algorithm. You just take the shape of the hardware into consideration when designing the algorithm instead of considering math only. > Seems like TVM Fair enough, though technically they are still about different things but it's indeed very close, but > and tinygrad ?????? what gives you this impression?
Earlier quoted context omitted.
It's Hopper-specific, the improvements are closely tied to Hopper features like warp groups and TMA. For 4090s, you might get a speedup by using the Triton implementation of FP8 attention: https://triton-lang.org/main/getting-started/tutorials/06-fu...
The original flash attention (v1?) took like a year to get added to llama.cpp and only provides single digit percent VRAM savings for typical context lengths and practically no speed boost. Still nice to have, but man was this thing overhyped. I doubt v3 will do more than marginally better on the RTX 5000 series.