Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Open-Source 1B OLMo Language Models

amd.com

Post reply on HN
Print view View original

21–30 of 35 posts

Re: AMD Open-Source 1B OLMo Language Models

#21

Post by YetAnotherNick »

lumost wrote: post #6
duchenne wrote: post #4

Training a 1B model on 1T tokens is cheaper than people might think. A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model. So you would need around 4,400 hours of GPU training costing only $11k And costs will keep going down.

Is there a handy table for this? My napkin math has either underestimated throughput by 2 orders of magnitude or the above estimate is high.

You require 6 * parameter * token flops[1] to train LLM. Which means (flop/s of H100 * MFU) / (6 * parameter) token per second. Assuming MFU of 40%, it is (1000 * 10^12 * 0.4) / (6 * 10^9) token/sec = 67,000 token/sec.

This repo[2] by Meta achieves 48% MFU, or 80k token/second.

[1]: https://arxiv.org/pdf/2001.08361

[2]: https://github.com/facebookresearch/lingua

Re: AMD Open-Source 1B OLMo Language Models

#22

Post by lumost »

codetrotter wrote: post #7
lumost wrote: post #6

Earlier quoted context omitted.

Is there a handy table for this? My napkin math has either underestimated throughput by 2 orders of magnitude or the above estimate is high.

(1,000,000,000,000/63,000)/(60*60) (1T tokens / 63k tokens per second) / (60 seconds per minute * 60 minutes per hour) Is approx 4400 hours So I guess that’s how the calculation went. Or did you mean a source for the number of tokens per second?

Tokens per second ;) I can do the arithmetic on my own.

Re: AMD Open-Source 1B OLMo Language Models

#23

Post by Havoc »

throwaway888abc wrote: post #2

"Furthermore, AMD OLMo models were also able to run inference on AMD Ryzen™ AI PCs that are equipped with Neural Processing Units (NPUs). Developers can easily run Generative AI models locally by utilizing the AMD Ryzen™ AI Software." Hope these AI PCs will run also something better than 1B model. What is it useful for ? Spellcheck ?

It’s less size of model and more mem throughout and npu tops that’s the limiting factor for this class of device

Which means you can do larger but it’ll become ever slower

Re: AMD Open-Source 1B OLMo Language Models

#24

Post by anon291 »

xtreme wrote: post #17
almostgotcaught wrote: post #16

Earlier quoted context omitted.

> Investing $10-20m of developer time into making ROCm work reliably easily would have paid for itself 100x. I love when outsiders throw around random-ass takes like this. Just curious: how'd you come up with this number? Is it backed by literally any thought/data/roadmap? Let's do some rough back of the envelope calculations: 20MM is 100 engineers working for 1 year. Or maybe it's 5 years of work for 20 engineers? W…

200 mm/year gets you roughly 1000 engineers at 200k salary. Is that not enough to make rocm experience equal to cuda?

Considering that each kernel / kernel size is usually custom tuned on NVIDIA, I'd say no. Working in this field at several different companies, there are likely thousands of hand-tuned variations of a simple GEMM kernel. Each one required an engineer to look at specifically, even if they're all variations on a common theme.

As far as I know (and again, I work in the field of AI compilers), we're still a ways off from complete end-to-end generation of highly optimized kernels. If you want it to go fast, you need to write it by hand [1], and then test and validate.

Moreover, chip makers are constantly adding new features (Tensor Cores in NVIDIA for example), so the compiler is always playing catch up and at some point an engineer has to sit down (likely a team of them) and think 'what's the best way to exploit this hardware functionality for software performance?'. Then they have to test and validate that, and then either write a kernel, or attempt to put that know-how into a compiler.

Multiply this times the number of kernels in a typical suite, and... yeah.

And that was my point about herculean effort on modern chips. Assembly language isn't just the old 'Add register 1 and 2 and dump in R3' anymore. It's 'Use this instruction to access memory in this way, so that it's in a compatible format for the next instruction' and 'oh yeah, make sure your memory synchronization primitives are such that the whole thing is coherent'. Good luck!

Even going one step up into a higher-level language, you have to know how the kernel gets compiled to make it worthwhile. Again, it is trivial to write a correct opencl matrix multiply, but that's never going to be the highest performance. You have to know the hardware intimately. This is where having the software co-designed with hardware is very important. Basically, every AI chipmaker of any importance does this, including the startups, like Groq and Cerebras.

[1] A lot of kernels share basic patterns, so its not as hard as it sounds, but definitely requires engineering effort to get the design right.

Re: AMD Open-Source 1B OLMo Language Models

#25

Post by almostgotcaught »

anon291 wrote: post #24
xtreme wrote: post #17

Earlier quoted context omitted.

200 mm/year gets you roughly 1000 engineers at 200k salary. Is that not enough to make rocm experience equal to cuda?

Considering that each kernel / kernel size is usually custom tuned on NVIDIA, I'd say no. Working in this field at several different companies, there are likely thousands of hand-tuned variations of a simple GEMM kernel. Each one required an engineer to look at specifically, even if they're all variations on a common theme. As far as I know (and again, I work in the field of AI compilers), we're still a ways off from…

> Considering that each kernel / kernel size is usually custom tuned on NVIDIA, I'd say no. Working in this field at several different companies, there are likely thousands of hand-tuned variations of a simple GEMM kernel. Each one required an engineer to look at specifically, even if they're all variations on a common theme.

Lol that's absolutely not true. What you're describing is literally impossible for any company that has more than one product family on the market since each product has different scratch sizes, number of vector registers, data types supported/emulated etc.

Outside of trade show demos, kernels are codegened. What is true is there are recurring "themes/patterns" that are handled by engineers for a class of products. Lately this is flash attention...

> Again, it is trivial to write a correct opencl matrix multiply, but that's never going to be the highest performance.

I guess you work at AMD. The reason AMD ships a whole bunch of binary kernels is not because someone tuned/designed each one but because AMD doesn't have a PTX/SASS equivalent. So each kernel has to be compiled at build time for each device (it's also why they can't have LTS support for architectures).

Re: AMD Open-Source 1B OLMo Language Models

#26

Post by anon291 »

almostgotcaught wrote: post #25
anon291 wrote: post #24

Earlier quoted context omitted.

Considering that each kernel / kernel size is usually custom tuned on NVIDIA, I'd say no. Working in this field at several different companies, there are likely thousands of hand-tuned variations of a simple GEMM kernel. Each one required an engineer to look at specifically, even if they're all variations on a common theme. As far as I know (and again, I work in the field of AI compilers), we're still a ways off from…

> Considering that each kernel / kernel size is usually custom tuned on NVIDIA, I'd say no. Working in this field at several different companies, there are likely thousands of hand-tuned variations of a simple GEMM kernel. Each one required an engineer to look at specifically, even if they're all variations on a common theme. Lol that's absolutely not true. What you're describing is literally impossible for any compa…

> outside of trade show demos, kernels are condegened. What is true is there are recurring "themes/patterns" that are handled by engineers for a class of products. Lately this is flash attention

I never said they weren't using code generation. I said that each one requires a manual tune. You will set various parameters, determine if the generated code does well enough and then if there's performance to squeeze out, you modify the code generator.

> I guess you work at AMD.

Close but not quite

Re: AMD Open-Source 1B OLMo Language Models

#27

Post by almostgotcaught »

anon291 wrote: post #26
almostgotcaught wrote: post #25

Earlier quoted context omitted.

> Considering that each kernel / kernel size is usually custom tuned on NVIDIA, I'd say no. Working in this field at several different companies, there are likely thousands of hand-tuned variations of a simple GEMM kernel. Each one required an engineer to look at specifically, even if they're all variations on a common theme. Lol that's absolutely not true. What you're describing is literally impossible for any compa…

> outside of trade show demos, kernels are condegened. What is true is there are recurring "themes/patterns" that are handled by engineers for a class of products. Lately this is flash attention I never said they weren't using code generation. I said that each one requires a manual tune. You will set various parameters, determine if the generated code does well enough and then if there's performance to squeeze out, y…

> that each one requires a manual tune

Ya definitely not - everyone uses grid search or whatever latest BPO tuning strategy.

Re: AMD Open-Source 1B OLMo Language Models

#28

Post by anon291 »

almostgotcaught wrote: post #27
anon291 wrote: post #26

Earlier quoted context omitted.

> outside of trade show demos, kernels are condegened. What is true is there are recurring "themes/patterns" that are handled by engineers for a class of products. Lately this is flash attention I never said they weren't using code generation. I said that each one requires a manual tune. You will set various parameters, determine if the generated code does well enough and then if there's performance to squeeze out, y…

> that each one requires a manual tune Ya definitely not - everyone uses grid search or whatever latest BPO tuning strategy.

Oh right. Those require no engineering effort because I said so.

Re: AMD Open-Source 1B OLMo Language Models

#29

Post by almostgotcaught »

anon291 wrote: post #28
almostgotcaught wrote: post #27

Earlier quoted context omitted.

> that each one requires a manual tune Ya definitely not - everyone uses grid search or whatever latest BPO tuning strategy.

Oh right. Those require no engineering effort because I said so.

They require one person or team to engineer and then a whole bunch of people to use...? That doesn't resemble in the least what you were describing where each kernel is hand-tuned for each shape and device. But please do continue to insist you're still somehow right

Re: AMD Open-Source 1B OLMo Language Models

#30

Post by cyberax »

almostgotcaught wrote: post #16
cyberax wrote: post #13

Earlier quoted context omitted.

Yes, that's what I mean. NVidia provided easy to use tooling (CUDA), and made sure it JustWorks everywhere. AMD did approximately nothing with ROCm. Investing $10-20m of developer time into making ROCm work reliably easily would have paid for itself 100x.

> Investing $10-20m of developer time into making ROCm work reliably easily would have paid for itself 100x. I love when outsiders throw around random-ass takes like this. Just curious: how'd you come up with this number? Is it backed by literally any thought/data/roadmap? Let's do some rough back of the envelope calculations: 20MM is 100 engineers working for 1 year. Or maybe it's 5 years of work for 20 engineers? W…

> I love when outsiders throw around random-ass takes like this. Just curious: how'd you come up with this number? Is it backed by literally any thought/data/roadmap?

It's a multiple of what the TinyGrad ( https://tinygrad.org/#tinybox ) startup raised in capital. So $10-20m is absolutely reasonable, especially if you add an established HR with a hiring pipeline, established IT dept, offices, etc.

The multiplier is also easy to justify, given the stock price of NVidia and AMD.

> A quick reminder before you answer: whatever you think is actually involved in improving ROCm, unless you work on ROCm, you're almost certainly not considering an entire iceberg of complexity (runtime/driver/firmware).

Oh, I do. I've been following the OpenSource AMD driver development for the last 2 decades.

And I maintain that the total amount of investment that AMD needed to make to rival NVidia in the market cap, would have been around that number.

> Cutting to the chase: you're off by at least two orders of magnitude on your goofy estimate; the real numbers are probably closer to 200MM invested every year for 10 years.

For an entirely new company starting from scratch? Reasonable. But AMD is not a new company, and they already are doing most of the work needed.

Post reply on HN