Show HN: Flash Attention in ~100 lines of CUDA
1–10 of 44 posts
Re: Show HN: Flash Attention in ~100 lines of CUDA
#2Pretty neat implementation. In general, for these sort of exercises (and even if the intention is to go to prod with custom kernels) I lean towards Triton to write the kernels themselves. It is much more easier to integrate to the tool chain, and allows a level of abstraction that doesn't affect performance even a little bit while providing useful constructs.
Re: Show HN: Flash Attention in ~100 lines of CUDA
#3For those who have no idea what's being discussed, quick background.
Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training.
Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. [3]
As a side comment, this entire industry is sorely in need of at least intros. The entire space has moved so fast in the last year I need an entire new dictionary and thesaurus for all the terms they've created. Notably, because of this, found out Google has a glossary of machine learning terms. Actually somewhat handy.
[1] Google Machine Learning Glossary (Transformer): https://developers.google.com/machine-learning/glossary/#tra...
[2] Same (Attention): https://developers.google.com/machine-learning/glossary/#att...
[2] arXiv: https://arxiv.org/abs/2205.14135
Re: Show HN: Flash Attention in ~100 lines of CUDA
#4Pretty neat implementation. In general, for these sort of exercises (and even if the intention is to go to prod with custom kernels) I lean towards Triton to write the kernels themselves. It is much more easier to integrate to the tool chain, and allows a level of abstraction that doesn't affect performance even a little bit while providing useful constructs.
You mean triton the inference server or triton the DSL for cuda?
Re: Show HN: Flash Attention in ~100 lines of CUDA
#5Pretty neat implementation. In general, for these sort of exercises (and even if the intention is to go to prod with custom kernels) I lean towards Triton to write the kernels themselves. It is much more easier to integrate to the tool chain, and allows a level of abstraction that doesn't affect performance even a little bit while providing useful constructs.
You mean triton the inference server or triton the DSL for cuda?
they mean the dsl (not just necessarily for cuda)
Re: Show HN: Flash Attention in ~100 lines of CUDA
#6Pretty neat implementation. In general, for these sort of exercises (and even if the intention is to go to prod with custom kernels) I lean towards Triton to write the kernels themselves. It is much more easier to integrate to the tool chain, and allows a level of abstraction that doesn't affect performance even a little bit while providing useful constructs.
You mean triton the inference server or triton the DSL for cuda?
The DSL: https://openai.com/research/triton
Re: Show HN: Flash Attention in ~100 lines of CUDA
#7Pretty neat implementation. In general, for these sort of exercises (and even if the intention is to go to prod with custom kernels) I lean towards Triton to write the kernels themselves. It is much more easier to integrate to the tool chain, and allows a level of abstraction that doesn't affect performance even a little bit while providing useful constructs.
yeah even the official flashattention is moving many implementations from cutlass to triton except for the main mha backward/forward pass
Re: Show HN: Flash Attention in ~100 lines of CUDA
#8What does __syncthreads() do here exactly? I'm new to CUDA, could get the overall idea of the FlashAttention paper but not the details.
Re: Show HN: Flash Attention in ~100 lines of CUDA
#9For those who have no idea what's being discussed, quick background. Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training. Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. [3] As a side comment, this entire industry is sorely in need of at least intros. The entire space has moved so fast in the last year I need an entire new dic…
[deleted]
Re: Show HN: Flash Attention in ~100 lines of CUDA
#10For those who have no idea what's being discussed, quick background. Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training. Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. [3] As a side comment, this entire industry is sorely in need of at least intros. The entire space has moved so fast in the last year I need an entire new dic…
Regarding your comment about how fast the research and industry is moving, would HN readers be interested in relevant one or two paragraph summaries that are basically "explain it like I am a machine learning engineer from 2020" but also knows the power of these models from a perspective of using ChatGPT or MS Copilot? That is, assume a fair amount of technical knowledge about the fundamentals, but don't assume that the reader is paying any attention to have whitebox knowledge of the current state of the art.