Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras’ new monster AI chip adds 1.4T transistors

spectrum.ieee.org

Post reply on HN
Print view View original

121–130 of 169 posts

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#121

Post by AnimalMuppet »

pradn wrote: post #22
systemvoltage wrote: post #3

Most interesting aspect of wafer-scale manufacturing is yield. Even if we have 95% chip yield, as the chip size approaches the wafer-level dimensions, I don't know off top of my head what the math would be but it is going to plummet drastically. My guess is that they're handling this in the chip logic. Building resiliency by turning off cells in the wafer that didn't yield. That begs the question, how are they probin…

The first Cerebras Wafer Scale Engine used "... breakthrough techniques in cross-reticle patterning, but with the level of redundancy built into the design, ensured a yield of 100%, every time." I'm unsure what to think of this. "When we spoke to Cerebras last year, the company stated that they already had orders in the ‘strong double digits’." And they cost 2- 2.5 million each! https://www.anandtech.com/show/15838/c…

So, less than 100 orders? That doesn't blow me away. Though at 2 million or more each, that adds up to real money...

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#122

Post by sillysaurusx »

durst wrote: post #106
sillysaurusx wrote: post #102

Earlier quoted context omitted.

Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today. Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosion…

Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?

Sadly I have nothing to offer yet, other than to DM me on Twitter with questions. Always happy to answer basic ones; I love this stuff.

But you’re right, I should write up something. That guide is pretty good, but it doesn’t walk you through anything specific; it shows you a map, but doesn’t take you on a trip, so to speak.

Some tips:

Use tf.name_scopes! You probably use them for variables, but there’s a different one for operations. If you make an @op_scope decorator, you can use it on all your ML functions and immediately get lots of insights as to where the XLA ops end up mapping to in your source code.

As much as it pains me to say this, avoid tensorflow 2 style code like the plague. Pretend that if you use eager execution, someone will jump out of the bushes and shoot you. I technically use TF2.4 now, but it’s still session / graph-based, not the new tf.function magic. The new magic pipeline seems to be slower, harder to use, harder to debug, and very likely to explode when you do anything even slightly different than the tutorial examples. YMMV, and maybe things are better now (or in a future release).

The profiler is magical; leverage it whenever you can. My workflow is to start up a TPU run, then ssh into my server and fire up a Tensorboard to that model dir. Then I manually navigate to /#profile (because the “profile” button doesn’t seem to show up in the menu anymore) which then lets you “capture profile”.

Make sure your TPU version matches your Tensorboard version. At this point we use TPU version 2.4.0, Tensorboard 2.4.1. If you get mysterious errors, this is likely the reason things are going wrong. Even TPU version 2.3.0 wasn’t enough.

Happily, the new profiling tooling is totally badass. The trace viewer is great, the op profiling is ok (though I wish it would show me all the damn ops, instead of “helpfully” hiding all but the top N ops), and the memory viewer is incredible. You can see exactly where in your pipeline is causing “peak memory usage”, what the peak is (down to the kilobyte), and have at least some idea of what’s causing it.

It’s not effortless though. The XLA fusion ops sort of make it harder to track down what’s doing what. (TF compiler is very powerful, but the trade off is that you almost never have manual control over memory usage, which can be frustrating).

All in all (or all-to-all, ha) it’s a lot of fun if you like seeing expensive hardware go brrrr, as I do. It makes it all worth it when your loss drops from 11 to 3 overnight on a 430m parameter gpt model. :)

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#123

Post by DesiLurker »

habibur wrote: post #91
sillysaurusx wrote: post #78

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…

So we are looking for OpenCUDA for AI. Like OpenGL in the 90s.

So basically OpenCL?

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#124

Post by oivey »

sanxiyn wrote: post #117
oivey wrote: post #115

Earlier quoted context omitted.

I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants th…

I think we are talking past each other. sillysaurusx is talking about OpenGL/Direct3D as a portability layer for hardwares. API vs AD is not a hardware portability layer concern. The proposal is to standardize OpenGL/Direct3D-like hardware portability layer defined by API. API vs AD you are talking about happens far above that and it is irrelevant.

Today, things like PyTorch are AD systems, optimization systems, libraries of high performance interoperable common blocks, and more. This massive complexity is why there is no standard. If you have a compiler front end that can AD arbitrary code, then the framework no longer has to worry about that. If you have a compiler backend that can target multiple devices, then the framework doesn’t have to do that, either. The size and complexity of PyTorch/TensorFlow is exactly why there isn’t a standard hardware portable NN API. A sane API of hardware independent common blocks can’t happen without all the front and backend compiler stuff first.

A PyTorch/TensorFlow-like API is always going to be massive, complicated, and hard to port to new hardware targets. Additionally, the complexity and reliance on C alone will make integrating exotic new concepts like say, differential equation solvers, extremely laborious.

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#125

Post by TFortunato »

sillysaurusx wrote: post #78

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…

The article mentions that developers use standard PyTorch and Tensorflow.

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#126

Post by durst »

sillysaurusx wrote: post #122
durst wrote: post #106

Earlier quoted context omitted.

Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?

Sadly I have nothing to offer yet, other than to DM me on Twitter with questions. Always happy to answer basic ones; I love this stuff. But you’re right, I should write up something. That guide is pretty good, but it doesn’t walk you through anything specific; it shows you a map, but doesn’t take you on a trip, so to speak. Some tips: Use tf.name_scopes! You probably use them for variables, but there’s a different on…

Thank you for the tips! Whenever I hear about new accelerators, my first question is: "how do people in the real world run fast code on this"? Because (related to your above points), you don't use an accelerator for things to just run, they have to run fast. Otherwise, you'd just use a CPU.

Detailed Question 1: do you have examples of fusion making things hard? Is there a way to nudge the compiler to not fuse or create a symbol table tracking fusion? Does fusion cause issues with the name_scopes?

Detailed Question 2: isn't Pytorch eager execution? Do you know how it compares to Tensorflow's eager execution?

General Request: I'm in a more theoretical position, writing papers on programming languages for accelerators https://aetherling.org/ and TAing courses on accelerators http://cs149.stanford.edu/fall20. So, I'm excited to see people's practical experiences using these accelerators in industry. It would be very enlightening (if you have time) to write up a comparison of tuning a model for an A100's tensor cores vs a TPU. This seems like the key trade-off in comparing architectures.

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#127

Post by docfort »

sillysaurusx wrote: post #78

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…

You might be interested in this for your M1 MBA: https://github.com/apple/tensorflow_macos

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#128

Post by foobiekr »

AnimalMuppet wrote: post #121
pradn wrote: post #22

Earlier quoted context omitted.

The first Cerebras Wafer Scale Engine used "... breakthrough techniques in cross-reticle patterning, but with the level of redundancy built into the design, ensured a yield of 100%, every time." I'm unsure what to think of this. "When we spoke to Cerebras last year, the company stated that they already had orders in the ‘strong double digits’." And they cost 2- 2.5 million each! https://www.anandtech.com/show/15838/c…

So, less than 100 orders? That doesn't blow me away. Though at 2 million or more each, that adds up to real money...

200M would be great revenue in that space.

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#129

Post by sillysaurusx »

docfort wrote: post #127
sillysaurusx wrote: post #78

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…

You might be interested in this for your M1 MBA: https://github.com/apple/tensorflow_macos

That was actually what I was referring to! It's super weird. I like it, and it's what I use primarily, but only because no other version of tensorflow will install.

Their compiler is closed source, so it's almost impossible to tell what's going on. But, when I connect using tf.Session(), then .list_devices() shows only CPU available.

However, when I enable the TF_MLC_LOGGING=1 magic variable, it does seem to be printing out messages that indicates it's doing some kind of graph substitution under the hood. Therefore, I assume that this is the intended usage mode.

In other words, there seems to be zero difference between the "CPU" and the "GPU". Normally you can say "Do this on the CPU" while "do that on the GPU." But not with this.

Hopefully they'll open source the code sometime this century so that it's clearer what the heck it's doing. For now, though, it's reasonably fast in whatever this "CPU" mode is -- I only need to run unit tests on my laptop anyway, since all training happens on TPUs. So I ended up happy.

(For the first day or so, I was panicking that I was going to have no working tensorflow whatsoever on my M1 laptop, which would've necessitated a swift return + substitution.)

Re: Cerebras’ new monster AI chip adds 1.4T transistors

#130

Post by blihp »

sillysaurusx wrote: post #78

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and w…

The philosophy is working as it often does in the early days/years of a technology. The customers for this have problems at an entirely different scale. It's worth it for them to drop millions on hardware and potentially millions more to (re)write the software as needed. Sure, they'd prefer to be buying commodity hardware with standardized APIs. But since that doesn't currently exist for this class of performance, they'll pay up. People did the same back in the days of Cray/SGI/Lisp Machines and so on. These custom solutions can be quite profitable until the competition gets to the 'good enough' stage and then orders start drying up.

If you're looking for standardized hardware and software for AI, Apple is the wrong platform to be on. I'm fairly confident it won't be happening there. (I don't say that to be a hater: see the direction they've gone re: GPU APIs... it's just Apple being Apple)

Post reply on HN