Forcing Flash Attention onto a TPU and Learning the Hard Way
1–10 of 17 posts
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#2One of my lessons in using different accelerators, whether they be different NVIDIA versions, or GPU->TPU, etc is that someone needs to do this work of indexing, partitioning, mapping, scheduling, and benchmarking. That work is labor intensive.
In this case, google has already done it, and that will be true for high resourced accelerator companies like Google working with the most popular operations like attention.
As long as you use those operations, you are okay. But if you do something different, you need to be prepared to do all of this yourself.
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#3It broke my heart to have a visceral "I'm being slop'd" reaction reading this: it's such good work, and AI's barely used AFAICT, but there's enough odd transitions and copy-pasta'd markdown that you get the subconcious "this is AI" reaction regardless.
Many sentences are 3x as long as it normally would be in subtle ways (to wit: "My flash attention is 35x slower than the fused standard at n=4096. Not a little worse. Catastrophically worse."), it really wears on attention. (pun intended) It brings literary voice to a technical blog post, and a very difficult process-oriented technical blog post. I have to reallocate my unfortunately-limited brain cells from "maintaining state of where we are in the process" to "is this cutesy fluff or important" and I've never had to do that in 37 years with technical blog posts.
The Markdown gets bad. Bolding is used for important phrases (like a human would), then, all of a sudden, after the "Inside a TPU chip" header its being used every other sentence, on anything that is a proper noun/would have a Wikipedia article. It got so weird that at some point I was like "a human definitely didn't let this through...they must be links?" and tried clicking them.
It's doubly bad at that point, because markdown tables start coming in hot and heavy too. So you're left with "It's pretty apparent the LLM did it from here, and I can't keep trying to keep the state of the process in my head while trying to figure out if the bolding is important, reflexive close tab
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#4Why ruin good work by letting Claude write it all? Full of em dashes, riddled with Claudisms.
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#5Why ruin good work by letting Claude write it all? Full of em dashes, riddled with Claudisms.
I personally don't mind letting Claude write about work.
You could spend 80% doing the work and 20% writing about it, or 99% doing the work and 1% copy-pasting Claude's writeup about it into a blog.
There is nothing wrong with writing if you are into it, and yes you can probably do better than Claude, but I can related to engineers who just want to build.
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#6Why ruin good work by letting Claude write it all? Full of em dashes, riddled with Claudisms.
I personally don't mind letting Claude write about work. You could spend 80% doing the work and 20% writing about it, or 99% doing the work and 1% copy-pasting Claude's writeup about it into a blog. There is nothing wrong with writing if you are into it, and yes you can probably do better than Claude, but I can related to engineers who just want to build.
If you can’t be bothered to write it, why should I bother to read it?
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#7Why ruin good work by letting Claude write it all? Full of em dashes, riddled with Claudisms.
I personally don't mind letting Claude write about work. You could spend 80% doing the work and 20% writing about it, or 99% doing the work and 1% copy-pasting Claude's writeup about it into a blog. There is nothing wrong with writing if you are into it, and yes you can probably do better than Claude, but I can related to engineers who just want to build.
I could spend 100% doing the work with my own Claude, and 0% reading yours. That's a negative-sum outcome. I do think that the 80%/20% split is better (though anything that is mostly human voice is fine for me).
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#8It broke my heart to have a visceral "I'm being slop'd" reaction reading this: it's such good work, and AI's barely used AFAICT, but there's enough odd transitions and copy-pasta'd markdown that you get the subconcious "this is AI" reaction regardless. Many sentences are 3x as long as it normally would be in subtle ways (to wit: "My flash attention is 35x slower than the fused standard at n=4096. Not a little worse.…
You got a lot further than I did.
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#9Why ruin good work by letting Claude write it all? Full of em dashes, riddled with Claudisms.
Why let an obsession with writing style prevent you from learning from a reasonably decent writeup?
Re: Forcing Flash Attention onto a TPU and Learning the Hard Way
#10Why ruin good work by letting Claude write it all? Full of em dashes, riddled with Claudisms.
I personally don't mind letting Claude write about work. You could spend 80% doing the work and 20% writing about it, or 99% doing the work and 1% copy-pasting Claude's writeup about it into a blog. There is nothing wrong with writing if you are into it, and yes you can probably do better than Claude, but I can related to engineers who just want to build.
Because the failures are so frequent and often load-bearing that it makes it a negative sum to even attempt to read stuff that appears generated.