AMD Open-Source 1B OLMo Language Models
1–10 of 35 posts
Re: AMD Open-Source 1B OLMo Language Models
#2"Furthermore, AMD OLMo models were also able to run inference on AMD Ryzen™ AI PCs that are equipped with Neural Processing Units (NPUs). Developers can easily run Generative AI models locally by utilizing the AMD Ryzen™ AI Software."
Hope these AI PCs will run also something better than 1B model.
What is it useful for ? Spellcheck ?
Re: AMD Open-Source 1B OLMo Language Models
#3"Furthermore, AMD OLMo models were also able to run inference on AMD Ryzen™ AI PCs that are equipped with Neural Processing Units (NPUs). Developers can easily run Generative AI models locally by utilizing the AMD Ryzen™ AI Software." Hope these AI PCs will run also something better than 1B model. What is it useful for ? Spellcheck ?
The point is that AMD is doing the legwork to ensure that AI models can run on their chips. While they could settle for inference workloads (port llama to AMD). It is unlikely that many teams will widely adopt their silicon unless they can be used in the end-end ML stack. Many pure OSS efforts have tried and failed to make AMD work for this use case.
As a chip maker - they will also have some undersold, QA, or otherwise wasted parts available for these training efforts - so the capex is likely less severe for them compared to a random startup betting on AMD.
Re: AMD Open-Source 1B OLMo Language Models
#4Training a 1B model on 1T tokens is cheaper than people might think.
A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model.
So you would need around 4,400 hours of GPU training costing only $11k
And costs will keep going down.
Re: AMD Open-Source 1B OLMo Language Models
#5"Furthermore, AMD OLMo models were also able to run inference on AMD Ryzen™ AI PCs that are equipped with Neural Processing Units (NPUs). Developers can easily run Generative AI models locally by utilizing the AMD Ryzen™ AI Software." Hope these AI PCs will run also something better than 1B model. What is it useful for ? Spellcheck ?
[deleted]
Re: AMD Open-Source 1B OLMo Language Models
#6Training a 1B model on 1T tokens is cheaper than people might think. A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model. So you would need around 4,400 hours of GPU training costing only $11k And costs will keep going down.
Is there a handy table for this? My napkin math has either underestimated throughput by 2 orders of magnitude or the above estimate is high.
Re: AMD Open-Source 1B OLMo Language Models
#7Training a 1B model on 1T tokens is cheaper than people might think. A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model. So you would need around 4,400 hours of GPU training costing only $11k And costs will keep going down.
Is there a handy table for this? My napkin math has either underestimated throughput by 2 orders of magnitude or the above estimate is high.
(1,000,000,000,000/63,000)/(60*60)
(1T tokens / 63k tokens per second) / (60 seconds per minute * 60 minutes per hour)
Is approx 4400 hours
So I guess that’s how the calculation went.
Or did you mean a source for the number of tokens per second?
Re: AMD Open-Source 1B OLMo Language Models
#8"Furthermore, AMD OLMo models were also able to run inference on AMD Ryzen™ AI PCs that are equipped with Neural Processing Units (NPUs). Developers can easily run Generative AI models locally by utilizing the AMD Ryzen™ AI Software." Hope these AI PCs will run also something better than 1B model. What is it useful for ? Spellcheck ?
The point is that AMD is doing the legwork to ensure that AI models can run on their chips. While they could settle for inference workloads (port llama to AMD). It is unlikely that many teams will widely adopt their silicon unless they can be used in the end-end ML stack. Many pure OSS efforts have tried and failed to make AMD work for this use case. As a chip maker - they will also have some undersold, QA, or otherw…
It's amazing how NVidia became worth $3T simply because they have better drivers and CUDA.
AMD has great hardware, but they never could be assed to do anything about their software.
Re: AMD Open-Source 1B OLMo Language Models
#9Earlier quoted context omitted.
The point is that AMD is doing the legwork to ensure that AI models can run on their chips. While they could settle for inference workloads (port llama to AMD). It is unlikely that many teams will widely adopt their silicon unless they can be used in the end-end ML stack. Many pure OSS efforts have tried and failed to make AMD work for this use case. As a chip maker - they will also have some undersold, QA, or otherw…
It's amazing how NVidia became worth $3T simply because they have better drivers and CUDA. AMD has great hardware, but they never could be assed to do anything about their software.
"utilizing the AMD Ryzen™ AI Software* sounds really unappealing! Like when companies don't realise you think their software to leverage hardware is bad and you'd prefer being able to use features via something generic
Re: AMD Open-Source 1B OLMo Language Models
#10Earlier quoted context omitted.
The point is that AMD is doing the legwork to ensure that AI models can run on their chips. While they could settle for inference workloads (port llama to AMD). It is unlikely that many teams will widely adopt their silicon unless they can be used in the end-end ML stack. Many pure OSS efforts have tried and failed to make AMD work for this use case. As a chip maker - they will also have some undersold, QA, or otherw…
It's amazing how NVidia became worth $3T simply because they have better drivers and CUDA. AMD has great hardware, but they never could be assed to do anything about their software.
Never underestimate development eco-system. Ballmer was famously repeatedly shouting developers many times in one of the Microsoft Windows conferences and now he's one of the richest persons. Microsoft also got out of their ways by introducing WSL for running Linux alongside Windows when they realized the majority of OS running their Azure cloud are Linux.