Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning and Its Implications for Computer Architecture and Chip Design

arxiv.org

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#11

Post by buboard »

quotemstr wrote: post #8
buboard wrote: post #6

brain floating point ... cool name i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778 > A particularly interesting research direction puts these three trends together, with a system running on large-scale ML accelerator hardware, with a goal of being able to train a model that can perform thousands or millions of tasks in a single mod…

> i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778 Thanks for the link. Artificial neural networks all the way down to binary weights [1] although this approach doesn't seem like the most efficient one. It's interesting how we're still seeing a ton of variability in ML architectures: it suggests we haven't stumbled on the right area…

A formal theory of deep learning is proving to be much more elusive that avionics. Interesting times though

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#12

Post by gnode »

buboard wrote: post #6

brain floating point ... cool name i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778 > A particularly interesting research direction puts these three trends together, with a system running on large-scale ML accelerator hardware, with a goal of being able to train a model that can perform thousands or millions of tasks in a single mod…

> many different components of different structures

> yup, he is building a brain

The limbic system of the brain is made of many different structures. It handles much of the ancient fixed function instinctive operation of the brain -- heavily involved in sleep, reflexes, appetite, and motivation, for example.

However, our general intelligence and learning capability is mostly due to our neocortex, which has a highly regular structure. The neocortex also subsumes the roles of much of the limbic system as development progresses, overriding the specific structures.

This suggests to me that intelligence / learning doesn't benefit from specific structures, but from general structures, capable of encoding behaviour as data (by long-term potentiation of synaptic weights in the brain's case).

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#13

Post by acollins1331 »

sanxiyn wrote: post #7
londons_explore wrote: post #3

Earlier quoted context omitted.

Read it. Worth a read, especially for those not closely following the machine learning world. The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most. I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve.…

Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538 . My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.

Capsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures.

Source: my thesis using capsule networks for semantic segmentation of aerial imagery

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#14

Post by fizixer »

londons_explore wrote: post #2

Badly formatted paper with handwavey abstract with no real focus and dropping buzzwords aplenty... I'll pass... Oh - it's written by Jeff Dean, inventor of Mapreduce, Bigtable, tensorflow, and practically a god... Yeah, I'll read it!

Short of TeX/LaTeX, I can put up with a Google docs paper.

I absolutely cannot stand a MS-Word paper. It's crippling.

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#15

Post by zwieback »

fizixer wrote: post #14
londons_explore wrote: post #2

Badly formatted paper with handwavey abstract with no real focus and dropping buzzwords aplenty... I'll pass... Oh - it's written by Jeff Dean, inventor of Mapreduce, Bigtable, tensorflow, and practically a god... Yeah, I'll read it!

Short of TeX/LaTeX, I can put up with a Google docs paper. I absolutely cannot stand a MS-Word paper. It's crippling.

Are you saying it's impossible to format a paper in Word to please you? Seems like there's a lot of options to get the look you want.

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#16

Post by fizixer »

zwieback wrote: post #15
fizixer wrote: post #14

Earlier quoted context omitted.

Short of TeX/LaTeX, I can put up with a Google docs paper. I absolutely cannot stand a MS-Word paper. It's crippling.

Are you saying it's impossible to format a paper in Word to please you? Seems like there's a lot of options to get the look you want.

Well folks who prepare documents in Word, esp. formal documents, more often than not, pick the default rendering of Times New Roman (and I know the default in MS-Word is not TNR).

And a standard MS-Word TNR document looks like crap. I'm sorry.

But I have to admit I have an over-reaction to seeing an MS-Word document after spending more than a decade working with TeX/LaTeX exclusively. So part of the blame goes to me, and actually hurts my ability to keep up with the literature when I start avoid MS-Word papers.

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#17

Post by sorenn111 »

Potentially noob question, with Moore's law slowing down, are there enough specializations/hardware modifications available like those mentioned in the paper such that progress in ML will continue to progress rapidly? or will these advancements simply forestall an inevitable asymptote.

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#18

Post by retrac »

sorenn111 wrote: post #17

Potentially noob question, with Moore's law slowing down, are there enough specializations/hardware modifications available like those mentioned in the paper such that progress in ML will continue to progress rapidly? or will these advancements simply forestall an inevitable asymptote.

It's little more than an educated guess on my part, but I figure there's about two orders of magnitude in improvements in processing speed exploitable with current processes, if a big-budget chip were designed specifically for ML training. GPUs are architecturally not very optimal for the task.

You want something like a chip with a huge mesh of small independent cores with their own local storage, quite possibly with non-digital circuits that can very quickly approximate the functions with analog electronics, rather than actually doing all of the calculations digitally. Some variation on that is the approach both Intel and IBM have taken with their "neural chips" in the last few years.

It seems that analog computers are finally getting their revenge.

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#19

Post by zwieback »

fizixer wrote: post #16
zwieback wrote: post #15

Earlier quoted context omitted.

Are you saying it's impossible to format a paper in Word to please you? Seems like there's a lot of options to get the look you want.

Well folks who prepare documents in Word, esp. formal documents, more often than not, pick the default rendering of Times New Roman (and I know the default in MS-Word is not TNR). And a standard MS-Word TNR document looks like crap. I'm sorry. But I have to admit I have an over-reaction to seeing an MS-Word document after spending more than a decade working with TeX/LaTeX exclusively. So part of the blame goes to me,…

Yeah, the default Word style definitely isn't anything like a typeset document. For fun I downloaded the "Latex.dot" template and some computer modern fonts, with those you can make a first approximation of the Latex look in Word but it's still not the same.

I've never been a fan of the Computer Modern font but the page layout and formating in Latex is certainly nice and just having a standard for scientific papers is a plus.

Re: Deep Learning and Its Implications for Computer Architecture and Chip Design

#20

Post by XuMiao »

acollins1331 wrote: post #13
sanxiyn wrote: post #7

Earlier quoted context omitted.

Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538 . My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.

Capsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures. Source: my thesis using capsule networks for semantic segmentation of aerial imagery

I like the Capsule idea too. In some way, capsule network is very similar to sparse attention network. It's just the attention normalization is different. Attention is normalized on the inputs, the capsule is normalized on the output. Potentially capsule can yield much cleaner patterns, while patterns generated by attention networks can be overlapping. It's just that capsule is much harder to solve.
Post reply on HN