Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern GPU Programming for MLSys

mlc.ai

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Modern GPU Programming for MLSys

#11

Post by reinitctxoffset »

I can't signal boost this enough.

I spent months, months of late nights watching commits to nvfuser and shit, I wrote a SASS decompiler instrumented everything trying to learn Blackwell.

This is the first time I've seen something so clean, just a real work of scholarship on it.

My hat is off to the authors and the contribution it represents.

If I would caution a reader anything it's that the 2CTA (sm_100 sm_110) patterns here are different on 1CTA in important ways and it's not a better / worse thing, they are good for different workloads.

Really outstanding work. I proves q lot of this in lean4 and published but I got lazy short of really doing the pedagogical work.

This is what you should be starting with if you want to max out 2CTA gear, it's immaculate.

Re: Modern GPU Programming for MLSys

#12

Post by dv35z »

mathisfun123 wrote: post #10
throwaw12 wrote: post #9

Earlier quoted context omitted.

I am a product engineer in yet another enterprise SaaS CRUD shop, who wants to learn more about the landscape and find the way to enter it eventually.

> wants to learn more about the landscape and find the way to enter it eventually let's swap roles and let's pretend i'm an ML engineer asking you how to enter CRUD. what would you tell me? my strong suspicion (if i caught you in an honest, frank, moment) is you would say to me "why the fuck would you want to do that - it sucks". i have this suspicion because i did actually used to do CRUD and it does suck! but here'…

Would you (or someone else passionate about this topic) consider answering the question directly? I am curious about this too.

Re: Modern GPU Programming for MLSys

#13

Post by golly_ned »

dv35z wrote: post #12
mathisfun123 wrote: post #10

Earlier quoted context omitted.

> wants to learn more about the landscape and find the way to enter it eventually let's swap roles and let's pretend i'm an ML engineer asking you how to enter CRUD. what would you tell me? my strong suspicion (if i caught you in an honest, frank, moment) is you would say to me "why the fuck would you want to do that - it sucks". i have this suspicion because i did actually used to do CRUD and it does suck! but here'…

Would you (or someone else passionate about this topic) consider answering the question directly? I am curious about this too.

Pytorch is widely accepted as the de facto ML framework in both research and industry. TensorFlow comes second in industry. Jax is hardly used at all, but uses the same backend as TensorFlow.

Triton is a python-like language to define ML math operations that run efficiently on hardware accelerators like GPUs or TPUs. OpenAI open sourced it. If there's a particular math operation you have a unique need for in your model, and it hasn't already been implemented by some other library, and it's important for efficiency, you'd probably write it in triton these days. It'll be compiled to an intermediate representation, then to an efficient runtime.

The course linked deals with "MLSys", or "ml systems". That means using GPUs and other hardware accelerators efficiently to run ML math operations on one or more computers.

95% of working ML engineers will never need to write Triton, and will be more than satisfied with PyTorch. Many more ML engineers will, nevertheless, write Triton code, because it is interesting, fun, easy, and people are impressed when you tell them you did.

Hosting pytorch models efficiently is currently awkward, because there's no clear winner in the ecosystem. ONNX is a way of representing model graphs in a framework-agnostic way. Other systems can interpret ONNX graphs to do inference. So sometimes, when someone wants to host a pytorch model, they turn it into an ONNX model and run it with an efficient runtime on CPUs or GPUs.

Re: Modern GPU Programming for MLSys

#14

Post by golly_ned »

mathisfun123 wrote: post #10
throwaw12 wrote: post #9

Earlier quoted context omitted.

I am a product engineer in yet another enterprise SaaS CRUD shop, who wants to learn more about the landscape and find the way to enter it eventually.

> wants to learn more about the landscape and find the way to enter it eventually let's swap roles and let's pretend i'm an ML engineer asking you how to enter CRUD. what would you tell me? my strong suspicion (if i caught you in an honest, frank, moment) is you would say to me "why the fuck would you want to do that - it sucks". i have this suspicion because i did actually used to do CRUD and it does suck! but here'…

I'm from the ML platforms and systems domain.

I strongly recommend it if one's able. It's a bit more stable than a quickly evolving ML/DL/AI ecosystem or frontend ecosystem. The skills are more durable. It repays deep investment and knowledge.

It allows you to straddle both the distributed systems and services domain and the ML domain.

ML systems problems are extremely interesting since they require extremes of compute, storage, network, and latency, in very different parts of the model lifecycle. Its unique problem is the scarcity and cost of hardware accelerators.

I've worked eleven years in the space and rarely have had the desire to leave.

Re: Modern GPU Programming for MLSys

#15

Post by mathisfun123 »

golly_ned wrote: post #14
mathisfun123 wrote: post #10

Earlier quoted context omitted.

> wants to learn more about the landscape and find the way to enter it eventually let's swap roles and let's pretend i'm an ML engineer asking you how to enter CRUD. what would you tell me? my strong suspicion (if i caught you in an honest, frank, moment) is you would say to me "why the fuck would you want to do that - it sucks". i have this suspicion because i did actually used to do CRUD and it does suck! but here'…

I'm from the ML platforms and systems domain. I strongly recommend it if one's able. It's a bit more stable than a quickly evolving ML/DL/AI ecosystem or frontend ecosystem. The skills are more durable. It repays deep investment and knowledge. It allows you to straddle both the distributed systems and services domain and the ML domain. ML systems problems are extremely interesting since they require extremes of compu…

> rarely have had the desire to leave.

I'm currently a GPU compiler engineer in FAANG specializing in compute (not graphics). So clearly ML systems. Prior I have worked at every level of stack above and during my PhD I worked below (RTL). I hate it and think about leaving every day (I stay because of the money and like wtf else am I gonna do lol).

Re: Modern GPU Programming for MLSys

#16

Post by mathisfun123 »

golly_ned wrote: post #13
dv35z wrote: post #12

Earlier quoted context omitted.

Would you (or someone else passionate about this topic) consider answering the question directly? I am curious about this too.

Pytorch is widely accepted as the de facto ML framework in both research and industry. TensorFlow comes second in industry. Jax is hardly used at all, but uses the same backend as TensorFlow. Triton is a python-like language to define ML math operations that run efficiently on hardware accelerators like GPUs or TPUs. OpenAI open sourced it. If there's a particular math operation you have a unique need for in your mod…

> Triton is a python-like language to define ML math operations that run efficiently on hardware accelerators like GPUs or TPUs. OpenAI open sourced it.

This is incorrect. Triton has literally no path to TPU and it has always been open source because it was Philippe Tillet's PhD project (OAI simply hired Philippe).

> 95% of working ML engineers will never need to write Triton, and will be more than satisfied with PyTorch.

Maybe 95% of hobbyist ML engineers but professional ML engineers are absolutely writing Triton day-to-day (eg FB has an army of such people). Even if you're not writing Triton you're still using Triton through inductor.

> because it is interesting, fun, easy, and people are impressed when you tell them you did

Professionals write Triton not for any of the reasons you mentioned but for the same reason they wrote CUDA kernels prior: it's a path to peak performance for their specific workloads (where stock PyTorch kernels have mediocre performance).

Re: Modern GPU Programming for MLSys

#17

Post by saagarjha »

mathisfun123 wrote: post #15
golly_ned wrote: post #14

Earlier quoted context omitted.

I'm from the ML platforms and systems domain. I strongly recommend it if one's able. It's a bit more stable than a quickly evolving ML/DL/AI ecosystem or frontend ecosystem. The skills are more durable. It repays deep investment and knowledge. It allows you to straddle both the distributed systems and services domain and the ML domain. ML systems problems are extremely interesting since they require extremes of compu…

> rarely have had the desire to leave. I'm currently a GPU compiler engineer in FAANG specializing in compute (not graphics). So clearly ML systems. Prior I have worked at every level of stack above and during my PhD I worked below (RTL). I hate it and think about leaving every day (I stay because of the money and like wtf else am I gonna do lol).

Are you willing to take a pay cut?

Re: Modern GPU Programming for MLSys

#18

Post by mathisfun123 »

saagarjha wrote: post #17
mathisfun123 wrote: post #15

Earlier quoted context omitted.

> rarely have had the desire to leave. I'm currently a GPU compiler engineer in FAANG specializing in compute (not graphics). So clearly ML systems. Prior I have worked at every level of stack above and during my PhD I worked below (RTL). I hate it and think about leaving every day (I stay because of the money and like wtf else am I gonna do lol).

Are you willing to take a pay cut?

No? Why would I? Like to what end?

Re: Modern GPU Programming for MLSys

#20

Post by mathisfun123 »

saagarjha wrote: post #19
mathisfun123 wrote: post #18

Earlier quoted context omitted.

No? Why would I? Like to what end?

> I hate it and think about leaving every day

most people hate their jobs; some people are just lucky enough that their jobs compensate them well enough for the emotional labor as well as the physical labor. also i have worked other SWE jobs (as already mentioned) and i've hated them all.
Post reply on HN