241–250 of 389 posts
I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…
I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…
Earlier quoted context omitted.
Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this. Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a mont…
It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer? If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.
If companies working on ML-specific chips was evidence that large transformer models have fully saturated their potential, the field would have been done circa GPT-2.
I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…
> For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. but you trade updatability, which I don't think is worth it yet.
I suspect the answer to both of these questions is yes right now, but I agree it’s borderline.
Story says they're power limited. That's half-true. Actually they're water-limited. To generate power, you need water. To cool chips, you need water. If you try to use less water on one side, you need more water on the other side (it's physics ya'll, making and using energy generates heat which requires dissipation). The world's freshwater is diminishing while also being consumed at an alarming rate. The future AI ol…
For datacenters specifically I've never understood what specifically consumes the water. Arent the water-cooling loops closed, so the water just cycles around and around and around?
- If they use either evaporative cooling or a liquid-cooled heat exchanger, that uses tons of water consistently. This requires less energy (it's mostly passive) so you use more water.
- If they use closed-loop water cooling and/or heat pumps/electric chillers, that uses much less water - at the DC. But it does require more energy to circulate the water, run fans, etc. If you are using more energy, where is the energy coming from? It's coming from power plants, which require... you guessed it... more water (e.g. thermoelectric, hydroelectric, geothermal, concentrated solar). They need water in order to generate the power, and lots of it. Coal, natural gas, nuclear, and concentrated solar, all use steam to generate energy. Nuclear also uses water to cool the reactor. And water is used extensively to extract coal, oil, and natural gas. Geothermal uses water in the ground.
You can't not use a ton of water in one fashion or another. It just depends what method, and on what end the water is used. And the crazy thing is, most new datacenters are being built in places with extremely little water. Guess how that's gonna work out as the planet gets hotter?
I don't know why I got downvoted to hell for stating facts every datacenter architect knows. HN be HN'in.
Earlier quoted context omitted.
> For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. but you trade updatability, which I don't think is worth it yet.
Maybe! (1) Would SOL level intelligence be useful 3 years from now? 5 years? (2) would dedicated chips be the most affordable way to run this model in 3-5 years? I suspect the answer to both of these questions is yes right now, but I agree it’s borderline.
Earlier quoted context omitted.
"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal
It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.
Earlier quoted context omitted.
"640k (token context) should be enough for anyone."
I know what you're saying, but modulo things like losing track of what year it is as time passes by, a current frontier model is going to continue to be useful for many tasks for many years, even moreso if it's 5-10x faster due to the chip architecture. It's not that it would be the best forever, it's that it would be useful for plenty long enough to be worthwhile, even if there was better stuff available. In exactly…
While it’s still too early to tell, I don’t think that’s how intelligence scales. Better models get you better solutions even to trivial problems. The ceiling for getting it done better is very high even if you’re not doing anything complicated. And difficulty isn’t uniformly distributed anyway - it seems to me that “mostly simple” tasks often have annoying 1% tails that low-intelligence models struggle with. I think we’ll see people chasing the top models for quite a while, or indefinitely - depending on the cost curve.
Earlier quoted context omitted.
and amazon shipping used to be free without prime, and uber used to be cheaper than taxis, and airbnb used to be cheaper than hotels. you really don't get it?
almost every pure tech commodity has gone down in price - gpus - retail computers - laptops - ~gpu~ appliances like washing machines - cloud computing i think you don't get how economy usually works in tech
Competition is good for all of us, we will get better and faster chips. Or at least Nvidia GPUs will become slightly cheaper for regular consumers again