Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

together.ai

Post reply on HN
Print view View original

61–63 of 63 posts

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#61

Post by latchkey »

saagarjha wrote: post #60
latchkey wrote: post #47

Earlier quoted context omitted.

Not a rude question. I'm building public HPC super computers, currently focused on AMD hardware. The one I'm about to deploy is Top 150, which is a pretty good start. The goal is to encourage a developer flywheel. The more developers working with AMD hardware, the more hardware that is needed, the more hardware I can justify buying, the bigger my super computers get. Nvidia has been doing the flywheel for years and i…

Perhaps I phrased my question wrong; I think you answered what are you getting out of this? My question is what the person writing code for you is getting out of it.

FA is open source. They aren't writing code "for me" and you would have to ask them.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#62

Post by namibj »

skyde wrote: post #58
namibj wrote: post #22

Earlier quoted context omitted.

In theory, yes, it's "just" some algebraic properties of the math used that allow for substantial reordering, and then you'd add fairly regular polyhedral loop tiling. Just expensive to do, so you'll have to cache the effort. The area of e-graph optimizers seems well-suited to this, btw. It's not really deployed outside of some niche tooling though, as it's a big paradigm shift in optimizer pass handling (e.g., doesn…

Just discovered e-graph recently and I have a good understanding of compiler from taking compiler class at university. I would like to understand why you say e-graph would need control-flow to be revamped. Do you have anything I could read on it ?

RVSDG (and the like) is needed to cope with shared state like globally addressable memory; plain e-graphs are only suited for pure code.

Happy to chat more btw, feel free to hit me up on discord.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#63

Post by MacsHeadroom »

nyrikki wrote: post #55
refibrillator wrote: post #38

The code has a comment which seems to hint that Tri Dao was working on FA3 as early as April 2022, the month after Hopper/H100 was announced. I find it mildly curious that over 2 years has elapsed before the code was released today. Perhaps it’s because now there’s better solutions in the pipeline? Tri’s publication history has been leaning toward SSM and Mamba style architectures recently. Unlike Flash Attention whi…

Until the strong exponential hypothesis is (dis-)proven, the quadratic cost is required or you have to give something up. Just the cost of exhaustive search. As (dis-)proving SETH will resolve the P vs NP problem, I wouldn't hold my breath. The question is if a particular use case can accept those costs.

What makes you think that the thing you have to give up is related to model quality?
Post reply on HN