Who said Moore’s law is dead?
Cerebras’ new monster AI chip adds 1.4T transistors
71–80 of 169 posts
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#72Earlier quoted context omitted.
You do realize that AI crossed human expert performance in NLP / Vision tasks ?
Exactly how does one outperform a human expert in natural language processing?
They don't really outperform human experts on real tasks yet (no matter what some superGLUE or other benchmark shows); but in general, once a system can solve a particualr task well, it would be plausible to outperform human experts simply by not making random errors.
If we have multiple human experts annotate a NLP task and measure inter-annotator agreement, it will be far from 100%; part of that will be genuine disagreements or fuzzy gray area, but part of the identified differences will be simply obviously wrong answers given by the experts - everyone makes mistakes. The same applies for many other domains - business process automation, data entry, etc; no employee will produce error-free output in a manual process, no matter how simple and unambiguous the task is.
And for simpler tasks the computer can easily make less mistakes than a human - especially if you measure the human reliability not for a few minutes of focus, but for a whole tedious working day.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#73This thing needs a GraphBLAS[1] implementation yesterday. 100 billion edge graphs and up are the new norm. This monster could smoke the competition if the implementation was tuned right! [1] http://graphblas.org
Creating the ecosystem of both software and adjacent hardware for wafers this size is the real challenge for a company like Cerebras (which is doing amazing work). At first, they thought they just needed to make a chip 56x the size of its predecessor, and somehow get around the issue of defects and yield. After they solved those problems (which blocked Gene Amdahl, among others), they found they needed to bring an en…
Agreed, that's why I think the GraphBLAS would be such a great fit for this hardware. The ecosystem is growing pretty fast. There are, for example Python bindings, you can do sparse 'A @ B' on millions of elements in parallel with this wafer-chip. MATLAB 2021a now has GraphBLAS built in, you could drive this thing directly from your notebooks.
I'm sure there's a compiler and low level primitives to really get the maximum performance out of it, but the trade-off maybe worth it in many cases to do it using an abstraction like the linear algebra approach.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#74This thing is awesome and, as someone working for a competitor, kind of scary. I applaud their approach though. I think we're a couple years off from it, but we'll probably see wider adoption of larger silicon, with more specialized functional units, which are used with a lower duty cycle to manage heat. If nothing else, they're probably developing some good IP and techniques to handle other sorts of ultra-mega-insan…
LSI, VLSI, UMISI
I like it!
In previous articles they've gone into some detail about how they deal with reticle limits, jumping over the scribe line area, and other n stuff. Between that, chiplets, HBM-style die stacks, etc... the developments here have been more interesting than I expected.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#75I think I once saw one of the founders with a wafer in an In n' out with a potential investor. Looking at what Apple achieved with their M1A and the demand for "AI" - or training neutral networks, what it really is - they have a lot of potential. At least as long as the AI bubble doesn't burst.
How can the future burst? It's like saying medicine will burst or physics
Personalized medicine, as an example, absolutely burst.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#76Much better article from anandtech at https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
I laughed at the "arm+leg" official cost of the CPU.
EDIT: Or GPU, or whatever it is.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#77Earlier quoted context omitted.
They probably aren't bothering to. The extreme economics for producing this type of chip are likely acceptable to the stakeholders. Also, there is no reason they cant have some redundancy throughout the design so you can fuse off bad parts. It all really depends on the nature of the anticipated vs actual defects, which is an extraordinarily deep rabbit hole to climb into.
How does this fusing work? I assume there are a bunch of wires than are either hot or ground and that determines if part of a chip gets run?
It might actually use a traditional fuse block, where at some point in the packaging/testing process, you literally apply a sufficiently high voltage that you can permanently 'set' some part of it (whether that's actually melting a tiny wire, I'm not sure). But that's basically just programming a ROM that gets read in at boot time, and sets a bunch of logic on the chip to route around the bad parts. You could just use an external EEPROM to track that info too, and it would basically work the same.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#78I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way.
The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.
And that’s just for gpt — you lose all the other libraries people have written. This matters more in GAN training, since for example you can find someone else’s FID implementation and drop it in without too much hassle. But with this specialized chip, you’d have to write it from scratch.
We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. But, for all its flaws, GL made these improvements “drop-in” —- just opt in to the new extension, and keep writing your gl code as you have been.
Ditto for direct3d, except they took the attitude of “limit to a specific API, not arbitrary extensions.” (Pixel shader 2.0 was an awesome upgrade from 1.1.)
AI has no such standards, and it hurts. The M1 GPU in my new Air is supposedly ready to do AI training. Imagine my surprise when I loaded up tensorflow and saw that it doesn’t support any GPU devices whatsoever. They seem to transparently rewrite the cpu ops to run on the gpu automatically, which isn’t the expected behavior.
So I dig into Apple’s actual api for doing training, and holy cow, that looks miserable to write in swift. I like how much control it gives you over allocation patterns, but I can’t imagine trying to do serious work in it on a daily basis.
What we need is a unified API that can easily support multiple backends — something like “pytorch, but just enough pytorch to trick everybody” since supporting the full api seems to be beyond hardware vendors’ capabilities at the moment. (Lookin’ at you, google. Love ya though.)
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#79Earlier quoted context omitted.
How does this fusing work? I assume there are a bunch of wires than are either hot or ground and that determines if part of a chip gets run?
I believe you design special fuses on the chip that can be "blown" with a laser after testing and before putting the silicon in the protective packaging.
Fuses aren't blown with a laser, it's purely electrical. You apply a sufficiently high voltage to some port on the part from a source that can drive a high enough current and then tell the digital block of the chip which address to fuse and if you want it high or low. Repeat for the entire fuse bank and you're done.
Re: Cerebras’ new monster AI chip adds 1.4T transistors
#80Who said Moore’s law is dead?
It’s not dead, but melting.