Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras’s giant chip will smash deep learning’s speed barrier

spectrum.ieee.org

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#101

Post by Veedrac »

typon wrote: post #74
Veedrac wrote: post #32

Earlier quoted context omitted.

They have customers already, one (Argonne National Labs) is given explicitly. The issue with using ‘industry standard’ benchmarks is that it's like measuring a bus' efficiency by shuttling around a single person at a time. The CS-1 is just bigger than that; the workloads that it provides the most value on are ones that are sized to fit, and specifically built for the device. This does make it hard to evaluate as outs…

They could always release figures for larger networks - they don't have to target Resnet50 (which is the MLPerf standard). I don't think anyone would hold it against them if they show massive improvements in something like GPT-2 training time (a network 37000x the size of Resnet)

GPT-2 uses attention, which is very memory hungry to train, so probably won't work well. But I agree with your overall point.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#102

Post by dsign »

mark_l_watson wrote: post #36

I don’t know if this mega-chip will be successful, but I like the idea. Before I retired I managed a deep learning team that had a very cool internal product for running distributed TensorFlow. Now in retirement I get by with a single 1070 GPU for experiments - not bad but having something much cheaper, much more memory, and much faster would help so much. I tend to be optimistic, so take my prediction with a grain o…

Deep learning is not going to get us to AGI. But the hardware techniques definitely are going to get us a bit closer.

I did the numbers a while ago and honestly I don't think we need smaller transistors to get the computation volume of our mushy brains -- although ofc, more and smaller transistors is always very nice. I believe the only thing stopping AGI at this point is architecture -- we really have no idea how to connect and structure something as complex as our brains -- and cognitive maturity. The last part is my way of saying "two weeks for training a NN? Wait until you have a kid and have to work on training the little human for decades....".

TBH, the ethical implications of AGI seem insurmountable to me. Life is a game --meaning the universe doesn't care about us, nor we owe anything to it--, and for now, it's our game. So, I would rather we put all that computing to improving human life -- including mind upload,-- and put AGI right there with nuclear weapons.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#103

Post by ThrowawayR2 »

dnautics wrote: post #54
ThrowawayR2 wrote: post #16

Earlier quoted context omitted.

That article is utter balderdash. Yes, it's obvious that you can fit more transistors on a "chip" if you make the chip be much, much larger than what we ordinarily think of as a chip. No, it does not mean that Moore's Law has been invalidated or some new "AI Moore’s Law" (quoting from the post) has come into being.

> Yes, it's obvious that you can fit more transistors on a "chip" if you make the chip be much, much larger than what we ordinarily think of as a chip. Without defending the article, it is however the case that simply scaling a chip size has nontrivial problems. For example, Will the piece of silicon warp or shatter if one side happens to get hotter than the other?

Possibly. Wafer scale integration has been investigated before though and there were even a couple of attempts at commercial products; it's not a brand new technology. Nevertheless, it might be interesting to examine Cerebras' patents to see if anything of significance relating to WSI is there.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#104

Post by rckoepke »

mantap wrote: post #96
corporate_shi11 wrote: post #53

Earlier quoted context omitted.

It's also my impression - from my modest exposure to DL over the past two years as a student taking courses - that deep learning must be overcome to reach AGI. Specifically gradient descent is a post hoc approach to network tuning, while human neural connections are reinforced simultaneously as they fire together. The post hoc approach restricts the scope of the latent representations a network learns because such re…

Don't forget the human brain takes about 7 to 8 hours off every day to rejiggle itself, to use a scientific term. The brain's architecture is better than having a training stage but it's by no means able to continually learn without stops and starts.

You see this in young puppies (3-6 month old) a lot as well. They get irritable/exhausted after 15-30 minutes of training, and usually dont seem to learn anything at all during the training activity itself. Then they pass out ("nap") for 30 minutes and when they wake up they do the trick/skill perfectly.

Same thing as humans, just more obvious/visible.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#105

Post by Veedrac »

IshKebab wrote: post #99
Veedrac wrote: post #60

Earlier quoted context omitted.

It's model parallel, so the first thing you do is lay out your floorplan for the model, which looks like this. https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpuplo... Then you put your data next to the core that uses it. Simples. (Optimal placement is tricky, but approximate techniques work fine.)

When you consider the things that that diagram doesn't show, it doesn't look at all simple. Does that graph even have training? It'll have to be pipelined too. Probably will have to use recomputation due to the shortage of memory. What about within the boxes? You can't nicely separate a matmul into pieces like that. I work on something similar but less ambitious, trust me it is crazy complicated.

Could you be more explicit? What about the naïve approach to training (same graph but backwards, computing gradients) is going to fail?

Wrt. matmul, if you couldn't split them up, today's AI accelerators wouldn't work full stop. But regardless, even if it was much more complex on CS-1 than on all the other sea-of-multipliers accelerators, it's obviously a problem they've solved and so irrelevant to the compilation issue.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#106

Post by justicezyx »

someguyorother wrote: post #64
justicezyx wrote: post #59

Earlier quoted context omitted.

Well, if a better algorithm cannot beat DNN in a realistic product setting, then how can you say its better after all? If the algorithm is indeed better, how can DNN dominates and turn into a dystonia...

What economists call path dependence. The alternative algorithm would be better than DNN if the same amount of effort was put into creating special-purpose hardware, libraries, and so on; but in the dystopia, it's not fully refined DNN vs fully refined alternative algorithm, but fully refined DNN vs alternative algorithm with hardware and software optimized for DNN. The alternative algorithm always looks unappealing…

You are referring back to OP's own reasoning fallacy...

DNN emerges out from being an underdog. Its superiority was proven by technology and economy.

What you said is of course not wrong, but they can never be proven right. As immediately you switch the role, your argument then favors the other one.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#107

Post by MaxBarraclough »

mark_l_watson wrote: post #36

I don’t know if this mega-chip will be successful, but I like the idea. Before I retired I managed a deep learning team that had a very cool internal product for running distributed TensorFlow. Now in retirement I get by with a single 1070 GPU for experiments - not bad but having something much cheaper, much more memory, and much faster would help so much. I tend to be optimistic, so take my prediction with a grain o…

> I get by with a single 1070 GPU

Amazon/Google/Microsoft will gladly take your money for time on their nVidia GPU instances, but they charge tens of cents per hour.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#108

Post by jhj »

Veedrac wrote: post #84
jhj wrote: post #82

Earlier quoted context omitted.

It's not like there is one SRAM, there are many SRAMs, so you get the same problem as NUMA but a thousand fold. Some computations you can map to a regular grid/hypercube/whatever quite easily, but it is unclear what the interconnect between the PEs is here, or what this thing has for a NOC or NOCs, how routing is handled, etc., and further complicating the issue is compensating for any damaged PEs or damaged routes.

No, you don't have all the issues with traditional NUMA because you aren't doing the same sort of heterogeneous workloads. You're always working on local data, and streaming your outputs to the next layer. This isn't a request-response architecture; such a thing wouldn't scale.

It is more or less the same, it's just that in NUMA you have a limited number of localities, except here it is in the thousands. The issue is one of scheduling that locality. Some process still needs to determine what data is actually local and where it should "flow". Because it can't all fit in one place, the computation needs to be tiled (potentially in multiple ways) and the tiles need to be scheduled to move around in an efficient manner.
Post reply on HN