Ways to think about token pricing
ben-evans.com
Ways to think about token pricing
1–10 of 26 posts
Re: Ways to think about token pricing
#21. Slot machine
You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.
And it gets tantalizingly close, but for 100% gotta pull the arm again!
Re: Ways to think about token pricing
#3I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.
Re: Ways to think about token pricing
#4I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.
sure, if there werent capitalists digging their moat. in 1 year local inferwnce doubled cost from ram alone because cloud vendors cornered markets on their circular cash flow expectations.
Re: Ways to think about token pricing
#5I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.
sure, if there werent capitalists digging their moat. in 1 year local inferwnce doubled cost from ram alone because cloud vendors cornered markets on their circular cash flow expectations.
Yeah but Samsung/Micron/SK hynix are building new fabs, and Chinese Ram will be able to fully supply the lower end ddr5 ram market within 2 years. The crunch is temporary and on the other side of it there will be incredible hardware at the $10-20k price point.
Re: Ways to think about token pricing
#6I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.
You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.
Re: Ways to think about token pricing
#7I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.
You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.
DRAM prices have been highly cyclical for a long time. They'll come down again.
Re: Ways to think about token pricing
#8Earlier quoted context omitted.
You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.
DRAM prices have been highly cyclical for a long time. They'll come down again.
From what I'm reading on the shortage, they are very aware of this fact (prices being highly cyclical) and they're trying to avoid overproducing RAM to avoid the oversupply glut. That's probably going to hurt the chance of prices coming down anytime soon.
Re: Ways to think about token pricing
#9[flagged]
Re: Ways to think about token pricing
#10Earlier quoted context omitted.
DRAM prices have been highly cyclical for a long time. They'll come down again.
From what I'm reading on the shortage, they are very aware of this fact (prices being highly cyclical) and they're trying to avoid overproducing RAM to avoid the oversupply glut. That's probably going to hurt the chance of prices coming down anytime soon.
Prices will come down when the stupid money isn't all going to datacenter buildouts.