This is the first I'm coming across Cerebras and interested here to understand more. Are these already in use integrated in some pc/servers or is it some dedicated server which you need to integrate into some motherboard, ...
Cerebras’ new monster AI chip adds 1.4T transistors
111–120 of 169 posts
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#112> the size ... [is] ... 46,225 square millimeters According to google calculations, this is just under 6 square feet.
I think it might be closer to 215mm x 215mm or 0.49 square feet.
Though that is unimaginably large to me for a silicon chip. I can't comprehend it - the engineering behind it would be incredible.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#113Earlier quoted context omitted.
> The philosophy here seems to be “if we build it, they’ll buy it.” Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually. > That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples. Again, based on the companies representations…
I’ve learned to be skeptical of such claims. TPUs made that same claim, but it’s hard to use it for real work, since you don’t have access to the 300GB TPU host memory / cpu, where you infeed training examples. There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) wil…
I agree.
I have high hope for Graphcore. They stated "We will be participating in MLPerf in 2021, starting with the first training submission in the spring".
https://www.graphcore.ai/posts/graphcore-sets-new-ai-perform...
(I would ignore all benchmarks in that post except the sentence quoted above however.)
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#114I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…
Thr Anandtech article from a other comment here indicates that both PyTorch and Tensorflow are already running on it.
>A key to the design is the custom graph compiler, that takes pyTorch or TensorFlow and maps each layer to a physical part of the chip, allowing for asynchronous compute as the data flows through.
https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#115Earlier quoted context omitted.
Graphics APIs and deep learning frameworks are really nothing alike. In graphics, save for a few new features, the goal every year is the same: draw more and faster. Deep learning is evolving much faster than that. The answer isn’t an API; it’s powerful and mostly transparent automatic differentiation. Anything that is just an API will forever be behind and is much less useful for active research. There are a few pro…
The answer is an API, like NNAPI. AD is a frontend concern and doesn't really matter to accelerators. For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/
I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants that even C exposes, which makes codegen and optimization harder.
An API is always going to be necessarily behind state-of-the-art because often research depends on inventing things that don’t fit within an existing API.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#116I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…
> We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements.
You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#117Earlier quoted context omitted.
The answer is an API, like NNAPI. AD is a frontend concern and doesn't really matter to accelerators. For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/
I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants th…
I think we are talking past each other. sillysaurusx is talking about OpenGL/Direct3D as a portability layer for hardwares. API vs AD is not a hardware portability layer concern. The proposal is to standardize OpenGL/Direct3D-like hardware portability layer defined by API. API vs AD you are talking about happens far above that and it is irrelevant.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#118Polyhedral LLVM compiler optimization > Quantum Computing Technology
Well at least on relevance* to active problems.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#119Why it's called as "AI"?
Designed for deep learning
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#120Earlier quoted context omitted.
How does this fusing work? I assume there are a bunch of wires than are either hot or ground and that determines if part of a chip gets run?
It might actually use a traditional fuse block, where at some point in the packaging/testing process, you literally apply a sufficiently high voltage that you can permanently 'set' some part of it (whether that's actually melting a tiny wire, I'm not sure). But that's basically just programming a ROM that gets read in at boot time, and sets a bunch of logic on the chip to route around the bad parts. You could just us…
20 years ago people were overclocking AMDs by drawing on them with pencil: http://www.ocmelbourne.com/tutorials/PencilTrick/
This last decade people soldered resistors onto cheaper NVIDIA cards, to make them behave as more expensive NVIDIA cards: https://www.eevblog.com/forum/general-computing/hacking-nvid...
So chipmakers have an incentive to bury the process in the chip and make it irreversible.