Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

together.ai

Post reply on HN
Print view View original

51–60 of 63 posts

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#51

Post by gitfan86 »

latchkey wrote: post #47
saagarjha wrote: post #45

Earlier quoted context omitted.

Not trying to be rude but what is the thinking behind this offer? Why would someone do this port for…free save for access to the hardware? What’s the upside for them?

Not a rude question. I'm building public HPC super computers, currently focused on AMD hardware. The one I'm about to deploy is Top 150, which is a pretty good start. The goal is to encourage a developer flywheel. The more developers working with AMD hardware, the more hardware that is needed, the more hardware I can justify buying, the bigger my super computers get. Nvidia has been doing the flywheel for years and i…

No one person or one company SHOULD have huge control over humanity, I agree.

But practically speaking this is a unique time in history of technology because there are quick feedback loops that cause that flywheel you mentioned to be a insurmountable first mover advantage.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#52

Post by latchkey »

gitfan86 wrote: post #51
latchkey wrote: post #47

Earlier quoted context omitted.

Not a rude question. I'm building public HPC super computers, currently focused on AMD hardware. The one I'm about to deploy is Top 150, which is a pretty good start. The goal is to encourage a developer flywheel. The more developers working with AMD hardware, the more hardware that is needed, the more hardware I can justify buying, the bigger my super computers get. Nvidia has been doing the flywheel for years and i…

No one person or one company SHOULD have huge control over humanity, I agree. But practically speaking this is a unique time in history of technology because there are quick feedback loops that cause that flywheel you mentioned to be a insurmountable first mover advantage.

> But practically speaking this is a unique time in history of technology because there are quick feedback loops that cause that flywheel you mentioned to be a insurmountable first mover advantage.

I'm staking my career and business on you being wrong about the insurmountable part. This is just the beginning of a long road and I'm not the only one who believes this. My partnership with Dell, Advizex and a huge soon to be announced datacenter company, isn't small beans.

Much like how I didn't know how the internet would look when I first joined in 1991. But, what I can see very clearly from my decades of experience in the tech field, is that history is repeating itself with what is happening in AI.

As I'm also prone to say... this isn't a football match where one team needs to "beat" the other. It really is enough to have multiple players in the market and nothing more than that. In fact, I'm more than happy to deploy any type of compute that my customers want me to deploy for them, including Nvidia.

Even Lamini, whom were previously AMD only, just announced [0] that they are partnering with Nvidia. Their software will run equally well on any system. Why? Because it builds a simple bridge from one platform to the next. Reminds me of the Java "write once, run anywhere" slogan. It actually worked pretty well.

[0] https://x.com/realsharonzhou/status/1811439958277927294

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#53

Post by gitfan86 »

latchkey wrote: post #52
gitfan86 wrote: post #51

Earlier quoted context omitted.

No one person or one company SHOULD have huge control over humanity, I agree. But practically speaking this is a unique time in history of technology because there are quick feedback loops that cause that flywheel you mentioned to be a insurmountable first mover advantage.

> But practically speaking this is a unique time in history of technology because there are quick feedback loops that cause that flywheel you mentioned to be a insurmountable first mover advantage. I'm staking my career and business on you being wrong about the insurmountable part. This is just the beginning of a long road and I'm not the only one who believes this. My partnership with Dell, Advizex and a huge soon t…

I'm not saying it is impossible for other companies to build good and profitable products. Google, AMD, Tesla all have good AI systems.

I'm saying NVDA uses their own chips to help build more chips, AND they are intricately involved in the buildout of the 100B data centers and intricately involved in TSMC roadmaps. That with the combination of huge profits that are increasing create even more advantages over competitors.

Obviously this doesn't go on forever, NVDA will never have 100T of profit in a quarter. Years from now the feedback loops will have diminishing returns and there will be commodity AI systems eventually.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#54

Post by latchkey »

gitfan86 wrote: post #53
latchkey wrote: post #52

Earlier quoted context omitted.

> But practically speaking this is a unique time in history of technology because there are quick feedback loops that cause that flywheel you mentioned to be a insurmountable first mover advantage. I'm staking my career and business on you being wrong about the insurmountable part. This is just the beginning of a long road and I'm not the only one who believes this. My partnership with Dell, Advizex and a huge soon t…

I'm not saying it is impossible for other companies to build good and profitable products. Google, AMD, Tesla all have good AI systems. I'm saying NVDA uses their own chips to help build more chips, AND they are intricately involved in the buildout of the 100B data centers and intricately involved in TSMC roadmaps. That with the combination of huge profits that are increasing create even more advantages over competit…

I did not use the word impossible. Nobody is arguing that Nvidia won't be the dominate player for a long time. That does not mean that there isn't a good business in being in the game.

> Years from now the feedback loops will have diminishing returns and there will be commodity AI systems eventually.

Maybe, but the cat is out of the bag. Before it was a question of Moore's law and speed, but nobody talks about that anymore... all they talk about is that the need for raw compute (not even the fastest), is officially boundless.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#55

Post by nyrikki »

refibrillator wrote: post #38

The code has a comment which seems to hint that Tri Dao was working on FA3 as early as April 2022, the month after Hopper/H100 was announced. I find it mildly curious that over 2 years has elapsed before the code was released today. Perhaps it’s because now there’s better solutions in the pipeline? Tri’s publication history has been leaning toward SSM and Mamba style architectures recently. Unlike Flash Attention whi…

Until the strong exponential hypothesis is (dis-)proven, the quadratic cost is required or you have to give something up. Just the cost of exhaustive search.

As (dis-)proving SETH will resolve the P vs NP problem, I wouldn't hold my breath.

The question is if a particular use case can accept those costs.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#56

Post by FL33TW00D »

rfoo wrote: post #17
WanderPanda wrote: post #14

Compiler folks: Is there any chance compilers will be able to find optimizations like FlashAttention on their own? Seems like TVM and tinygrad are working in that direction but I find it hard to believe that that would be feasible

No. Think of it like a different algorithm. You just take the shape of the hardware into consideration when designing the algorithm instead of considering math only. > Seems like TVM Fair enough, though technically they are still about different things but it's indeed very close, but > and tinygrad ?????? what gives you this impression?

Geo has explicitly stated he wants to be able to find FA in the search space of algos eventually. Actually achieving that is another matter.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#57

Post by skyde »

WanderPanda wrote: post #14

Compiler folks: Is there any chance compilers will be able to find optimizations like FlashAttention on their own? Seems like TVM and tinygrad are working in that direction but I find it hard to believe that that would be feasible

https://github.com/uwplse/tensat

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#58

Post by skyde »

namibj wrote: post #22
WanderPanda wrote: post #14

Compiler folks: Is there any chance compilers will be able to find optimizations like FlashAttention on their own? Seems like TVM and tinygrad are working in that direction but I find it hard to believe that that would be feasible

In theory, yes, it's "just" some algebraic properties of the math used that allow for substantial reordering, and then you'd add fairly regular polyhedral loop tiling. Just expensive to do, so you'll have to cache the effort. The area of e-graph optimizers seems well-suited to this, btw. It's not really deployed outside of some niche tooling though, as it's a big paradigm shift in optimizer pass handling (e.g., doesn…

Just discovered e-graph recently and I have a good understanding of compiler from taking compiler class at university.

I would like to understand why you say e-graph would need control-flow to be revamped.

Do you have anything I could read on it ?

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#59

Post by eigenvalue »

LarsDu88 wrote: post #46

I was wondering... this post mentions that ops like sigmoid are very slow. A lot of modern LLMs use activation functions with sigmoid or soft max like SiLU, Swish, and SOLU. Does Relu take less of a performance hit, and if so, maybe it'd be better to go back to good old relu?

Relu is literally just a linear function that gets clamped to zero at some point, so yes, it's much less computationally intensive than anything involving an exponential function. But I doubt you would get competitive results using such a simple activation.

Re: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

#60

Post by saagarjha »

latchkey wrote: post #47
saagarjha wrote: post #45

Earlier quoted context omitted.

Not trying to be rude but what is the thinking behind this offer? Why would someone do this port for…free save for access to the hardware? What’s the upside for them?

Not a rude question. I'm building public HPC super computers, currently focused on AMD hardware. The one I'm about to deploy is Top 150, which is a pretty good start. The goal is to encourage a developer flywheel. The more developers working with AMD hardware, the more hardware that is needed, the more hardware I can justify buying, the bigger my super computers get. Nvidia has been doing the flywheel for years and i…

Perhaps I phrased my question wrong; I think you answered what are you getting out of this? My question is what the person writing code for you is getting out of it.
Post reply on HN