Earlier quoted context omitted.
Also, electricity isn't free.
With enough solar panels it is!
Not quite.
Free for approximately 8 hours (assuming perfect weather conditions) and excluding unit cost and maintenance cost.
It has a cost.
Memory has grown to nearly two-thirds of AI chip component costs
351–360 of 524 posts
Re: Memory has grown to nearly two-thirds of AI chip component costs
#352Earlier quoted context omitted.
China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.
Not just DRAM market, but the GPU market soon. China is the great equalizer of the world.
> Not just DRAM market, but the GPU market soon
China doesn't have EUV fabs... They've pushed DUV impressively far... but until they get EUV working industrially (and reasonable timelines are at least 2-4 years for that) it shouldn't be possible for them to compete for that market.
> China is the great equalizer of the world.
China is hardly an egalitarian society...
Re: Memory has grown to nearly two-thirds of AI chip component costs
#353Earlier quoted context omitted.
2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively
Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.
This hasn’t been true for the past 2 years
Re: Memory has grown to nearly two-thirds of AI chip component costs
#354Earlier quoted context omitted.
Chip manufacturing has unusually long spin-up times, high capital costs and relatively thin margins for anything but the latest and greatest processes, compared to most industries.
Well, let's remove the sanctions from China then and we'll get a better idea about costs and spin-times. BTW all RAM is severely overpriced, not only the one using the latest process nodes.
You seem more invested in hating RAM manufacturers than interested in the actual economics of the business.
Look up Qiminda, ProMOS, Elpida. They invested in capacity during booms.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#355An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
I wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...
This has been around for quite some time, to the point I had to read this a couple times to understand what you meant. Mighta predated LLMs even.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#356An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
Really wondering what this might mean for local LLMs when RAM costs plummet...
Re: Memory has grown to nearly two-thirds of AI chip component costs
#357Re: Memory has grown to nearly two-thirds of AI chip component costs
#358An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
> either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike
Or the more likely scenario that the AI bubble bursts and the hyperscalars realize they have built too many data centers.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#359An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
I wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...
Notoriously terrible?
Posits do a little better if your numbers are biased enough toward 1, but not much better. A 16 bit posit in a near-ideal situation matches an 18 bit IEEE float, and in a pretty wide range of situations loses to either fp16 or bf16.
Training anything at 8 bits is going to be tough, and it's hard to say if the flexible exponent is worth the precision tradeoffs.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#360[flagged]