Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras’s giant chip will smash deep learning’s speed barrier

spectrum.ieee.org

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#91

Post by wbl »

luma wrote: post #55
tedivm wrote: post #48

Earlier quoted context omitted.

I think it's a little more complicated than running more cables. Most datacenters have a total capacity they can handle, based on how many connections they have to their local grid (or grids, as datacenter places like Santa Clara have multiple power grids to give datacenter redundancy). You need to make sure your internal power distribution systems can actually handle the amount you want to push through, and you need…

Getting more power into a datacenter is a different problem than getting more (already available) power into a rack. I suppose I could have added "if your existing power distribution system can handle the extra power capacity". That includes service entrance, transfer switching, standby and backup power sources, and distribution to the rack level. The point I'm trying to make is that, all things being equal, it's _mu…

I'm waiting for the high pressure helium filled datacenter.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#92

Post by Veedrac »

tachyonbeam wrote: post #90
Veedrac wrote: post #72

Earlier quoted context omitted.

Remember that you aren't compiling arbitrary programs. Neural nets don't really have any local looping control flow, in the sense that data goes in one end and comes out the other. You'll have large-scale loops over the whole network, and each core might have a loop over small, local arrays of data, but you shouldn't have any sort of internal looping involving different parts of the model.

It's pretty common to have neural networks that have both recurrent nets processing text input and convolutional layers. A classic example would be visual question answering (is there a duck in this picture?). That would be a simple example involving looping over one part of the model. Ideally you want that looping to be done as locally as possible to avoid wasting time having a program on a CPU dispatching, waiting…

I would be surprised if Cerebras was trying to handle any recurrence inside the overall forward/backward passes. It seems like a lot of difficulty (as mentioned) for peanuts.

I don't get your point about training. Yes, it's backwards rather than forwards, and yes it often has fancy stuff intermixed (dropout, Adam, ...), but these are CPUs, they can do that as long as it fits the memory model.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#93

Post by desertrider12 »

Google234 wrote: post #78
bcatanzaro wrote: post #5

Reminds me of that other great prediction of a GPU killer from IEEE Spectrum back in 2009: https://spectrum.ieee.org/computing/software/winner-multicor...

What did go wrong with Intel's MIC (Xeon Phi) project? I can't find a compressive account of this from HPC people. The idea seemed pretty sound: large die, simpler circuit, and much more parallelism, in the x86 line..

They didn't really go that wrong, Cori and Trinity are still useful machines. But I can say for computational science, programming models have gotten way better in the last few years. Now it's as easy to get a new sparse algorithm to high occupancy on a GPU as it is to scale on a manycore CPU. So GPUs just look better now considering cost, power efficiency and software support.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#94

Post by jamesblonde »

mark_l_watson wrote: post #36

I don’t know if this mega-chip will be successful, but I like the idea. Before I retired I managed a deep learning team that had a very cool internal product for running distributed TensorFlow. Now in retirement I get by with a single 1070 GPU for experiments - not bad but having something much cheaper, much more memory, and much faster would help so much. I tend to be optimistic, so take my prediction with a grain o…

Commodity deep learning might be a lot closer than you think. Nvidia won't bring us there (without kicking and screaming), but AMD might. You can pick up a Radeon VII for about 600 dollars and use it in your data center without licensing issues (16 GB, about the training speed of a 2080Ti for ImageNet ConvNets). AMD use ROCm (now fully upstreamed into TensorFlow instead of Cuda - https://www.logicalclocks.com/blog/welcoming-amd-rocm-to-hop... ). Disclaimer: I worked on getting ROCm into YARN for Hopsworks.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#95

Post by jamesblonde »

foota wrote: post #71
varelse wrote: post #68

I am far more excited by the underlying Wafer Scale Integration moonshot than I am by any AI benchmarks here. I know it's trendy to think there can only be one w/r to the AI Iron Throne but nope, not the case, everyone is writing bespoke code in production where the money is made. Well, almost everyone, Amazon seems to be the odd duck but they're a bunch of cheapskate thought leaders anyway (except for their offers t…

Isn't that similar to what AMD is doing with infinity fabric? Obviously not at such a large scale.

Infinity fabric is closer to Nvidia's NVLink - much lower interconnect B/W. PCI 4.0 will be interesting as a commodity alternative, particularly when paired with AMD Rome chips with huge numbers of I/O lanes - for distributed training. https://wccftech.com/amd-epyc-rome-zen-2-7nm-server-cpu-162-...

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#96

Post by mantap »

corporate_shi11 wrote: post #53
mark_l_watson wrote: post #36

I don’t know if this mega-chip will be successful, but I like the idea. Before I retired I managed a deep learning team that had a very cool internal product for running distributed TensorFlow. Now in retirement I get by with a single 1070 GPU for experiments - not bad but having something much cheaper, much more memory, and much faster would help so much. I tend to be optimistic, so take my prediction with a grain o…

It's also my impression - from my modest exposure to DL over the past two years as a student taking courses - that deep learning must be overcome to reach AGI. Specifically gradient descent is a post hoc approach to network tuning, while human neural connections are reinforced simultaneously as they fire together. The post hoc approach restricts the scope of the latent representations a network learns because such re…

Don't forget the human brain takes about 7 to 8 hours off every day to rejiggle itself, to use a scientific term. The brain's architecture is better than having a training stage but it's by no means able to continually learn without stops and starts.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#97

Post by Zenst »

kingosticks wrote: post #80
01100011 wrote: post #79

Earlier quoted context omitted.

Did they come up with an architecture which can route around any defect? Probably not. Now, granted, 90% of their chip is probably dedicated to compute, but I'd bet there's some management infrastructure where they absolutely cannot tolerate a defect.

They'll simply have redundant copies of that logic. And they'll be physically located at areas of the wafer that yield well - some areas are much worse than others and I would imagine they'll make use of that.

Interesting so on a die, there are area's which are more prone to faults and they are able to factor that into the design?

Though if there are known hotspots, wouldn't that point to the process node inducing them over silicon quality? Or is it a case of silicon production produces known hotspots that are predictable? FWIW, I'm currently learning towards process node over the silicon being the source of hotspots, given what I know about silicon production.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#98

Post by Zenst »

Accujack wrote: post #43
Zenst wrote: post #2

A chip that size, imagine the yield. Equally, cooling - has to be water based as a heatsink that size would be on par to a small anvil and the weight factor would be some serious issues. Though unsure as no pictures of it in-play alas and all they say is - "20 kilowatts being consumed by each blew out into the Silicon Valley streets through a hole cut into the wall", which does somewhat beg for a picture as just rais…

>which does somewhat beg for a picture as just raises more questions. There's a picture in the article. >Why would they make a chip this big Did you read the article? >this article mentions TSMC. Another article I read that they used a 16nm node Yes, 16nm/TSMC.

> There's a picture in the article.

Yes - hardly helpful ones as you get a picture of a wafer and a box, not breakdown beyond that - hence had look and found other articles with much more detail upon this that answers the questions I raised in relation to the lack of pictures - like the cooling aspect in which you snipped my quote and removed that lovely thing we call context.

>Did you read the article?

Yes and had you read what I said you would see that the article does not answer the aspects I was asking - see what you did there.

>Yes, 16nm/TSMC

Yes - I found that in another article - which I also linked, you're welcome.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#99

Post by IshKebab »

Veedrac wrote: post #60
IshKebab wrote: post #57

Earlier quoted context omitted.

I'm afraid recursivecaveat is right. This is an insanely difficult compilation target. I think you're possibly talking about a different kind of "compilation" - i.e. the Clang/GCC bit that converts C++ to machine code. That is indeed trivial. But "compilation" for these chips includes much more than that. The really complicated bit is converting the tensorflow model to some kind of computation plan. Where do you put…

It's model parallel, so the first thing you do is lay out your floorplan for the model, which looks like this. https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpuplo... Then you put your data next to the core that uses it. Simples. (Optimal placement is tricky, but approximate techniques work fine.)

When you consider the things that that diagram doesn't show, it doesn't look at all simple. Does that graph even have training? It'll have to be pipelined too. Probably will have to use recomputation due to the shortage of memory. What about within the boxes? You can't nicely separate a matmul into pieces like that.

I work on something similar but less ambitious, trust me it is crazy complicated.

Re: Cerebras’s giant chip will smash deep learning’s speed barrier

#100

Post by kingosticks »

Zenst wrote: post #97
kingosticks wrote: post #80

Earlier quoted context omitted.

They'll simply have redundant copies of that logic. And they'll be physically located at areas of the wafer that yield well - some areas are much worse than others and I would imagine they'll make use of that.

Interesting so on a die, there are area's which are more prone to faults and they are able to factor that into the design? Though if there are known hotspots, wouldn't that point to the process node inducing them over silicon quality? Or is it a case of silicon production produces known hotspots that are predictable? FWIW, I'm currently learning towards process node over the silicon being the source of hotspots, give…

With normal-sized dies, at the die-level, I've not seen people design around this; other than the more obvious places e.g. the corners (bad power delivery, prone to mechanical issues, normally left vacant) and the middle (gets hotter, also sometimes bad power). However, there are many test structures placed across the die to measure/check variations and also design rules that constrain the relative placements of certain things. That also goes towards increasing yield.

But at the wafer-level, yes.

> wouldn't that point to the process node inducing them over silicon quality?

I don't see why. I would only vaguely guess it's related to the manufacturing process they follow at that particular node. Maybe it's not even directly silicon related but something else.

I'm not convinced it's worthwhile separating out the process node and the silicon quality, they are entwined when looking across a large sample size.

Unfortunately, someone that actually knows why probably isn't allowed to share why.

Post reply on HN