151–160 of 169 posts
For those interested, here is an article [1] on glimpse of how it is cooled. Although I still dont understand how this is possible though. 20kW . [1] https://www.eetimes.com/powering-and-cooling-a-wafer-scale-d...
Earlier quoted context omitted.
I don't really agree, at least from a longer term perspective. It's early days yet, but XLA seems to be a promising intermediate representation for allowing the DL frameworks to run on a wider array of hardware without user-facing software changes. It has traction with Google, NVIDIA, IIRC Intel and maybe more (others are definitely using the same approach of compute graph splitting and subgraph scheduling, but I'm n…
Mr/mrs anonymous HN person, please put some info in your profile. You clearly have some deep knowledge of TPUs that I didn’t expect to pop up offhandedly on HN. You’re correct on all counts: dynamic tensor shapes are more or less impossible with XLA, making it more or less impossible to train a model with arbitrary image size inputs, even though the math would allow for that; the pytorch XLA work on TPUs is indeed ki…
Earlier quoted context omitted.
> The philosophy here seems to be “if we build it, they’ll buy it.” Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually. > That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples. Again, based on the companies representations…
I’ve learned to be skeptical of such claims. TPUs made that same claim, but it’s hard to use it for real work, since you don’t have access to the 300GB TPU host memory / cpu, where you infeed training examples. There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) wil…
Cerebras itself commented that DL is not their first use case. They mentioned fluid dynamics instead. (I can find the reference if someone is interested).
Cerebras also has a weird memory/compute ratio, so very hard for some uses.
Somewhat off-topic. If u have standard keras/TF training code, the GPU to TPU is a smooth-ish transition. No?
A re-write will squeeze another 20% but not 200%?
I think I once saw one of the founders with a wafer in an In n' out with a potential investor. Looking at what Apple achieved with their M1A and the demand for "AI" - or training neutral networks, what it really is - they have a lot of potential. At least as long as the AI bubble doesn't burst.
How can the future burst? It's like saying medicine will burst or physics
Most interesting aspect of wafer-scale manufacturing is yield. Even if we have 95% chip yield, as the chip size approaches the wafer-level dimensions, I don't know off top of my head what the math would be but it is going to plummet drastically. My guess is that they're handling this in the chip logic. Building resiliency by turning off cells in the wafer that didn't yield. That begs the question, how are they probin…
Disabling parts of the chip? The secret sauce then is to make it defect-proof
Any article or headline talking about transistor count is just a puff piece. The performance matters, the number of transistors never does, and I have never purchases anything based on the number of transistors.
I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…
> We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.
If AI acceleration chips aren't able to offer similar API standardisation/stability then it's not the same as what happened with GPUs, which was sillysaurusx's point.
Interesting consider that if this "chip" was manufactured with the feature size of the MC68000, the chip would be about 1/4 mile square.
I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…
I don't really agree, at least from a longer term perspective. It's early days yet, but XLA seems to be a promising intermediate representation for allowing the DL frameworks to run on a wider array of hardware without user-facing software changes. It has traction with Google, NVIDIA, IIRC Intel and maybe more (others are definitely using the same approach of compute graph splitting and subgraph scheduling, but I'm n…
Yes.
Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training.
Recent progress suggests this will happen. See, for example:
https://arxiv.org/abs/2006.04182
https://arxiv.org/abs/2103.03725
https://arxiv.org/abs/2010.01047
I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense connections than present state-of-the-art models.