Deep Learning and Its Implications for Computer Architecture and Chip Design
1–10 of 26 posts
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#2Badly formatted paper with handwavey abstract with no real focus and dropping buzzwords aplenty... I'll pass...
Oh - it's written by Jeff Dean, inventor of Mapreduce, Bigtable, tensorflow, and practically a god... Yeah, I'll read it!
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#3Badly formatted paper with handwavey abstract with no real focus and dropping buzzwords aplenty... I'll pass... Oh - it's written by Jeff Dean, inventor of Mapreduce, Bigtable, tensorflow, and practically a god... Yeah, I'll read it!
Read it. Worth a read, especially for those not closely following the machine learning world.
The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most.
I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve. If someone like Google, with far more different problems to solve, can get this type of model to work and demonstrate its effectiveness, I think it would be a big step towards solving artificial general intelligence.
Jeff Dean has a lot of respect and influence inside Google, and his ideas tend to get implemented. I'm looking forward to it!
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#4Does anyone happen to have a link to a paper or book describing the state of the art in placement and routing algorithms? I'd like to read up on that topic.
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#5[deleted]
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#6brain floating point ... cool name
i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778
> A particularly interesting research direction puts these three trends together, with a system running on large-scale ML accelerator hardware, with a goal of being able to train a model that can perform thousands or millions of tasks in a single model. Such a model might be made up of many different components of different structures,
yup, he is building a brain
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#7Badly formatted paper with handwavey abstract with no real focus and dropping buzzwords aplenty... I'll pass... Oh - it's written by Jeff Dean, inventor of Mapreduce, Bigtable, tensorflow, and practically a god... Yeah, I'll read it!
Read it. Worth a read, especially for those not closely following the machine learning world. The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most. I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve.…
Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#8brain floating point ... cool name i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778 > A particularly interesting research direction puts these three trends together, with a system running on large-scale ML accelerator hardware, with a goal of being able to train a model that can perform thousands or millions of tasks in a single mod…
> i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778
Thanks for the link. Artificial neural networks all the way down to binary weights [1] although this approach doesn't seem like the most efficient one. It's interesting how we're still seeing a ton of variability in ML architectures: it suggests we haven't stumbled on the right area yet. It reminds me how early aviation had a huge diversity of aircraft plans, but now, after a lot of optimization, we've settled on that one standard airliner shape everyone uses everywhere.
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#9Does anyone happen to have a link to a paper or book describing the state of the art in placement and routing algorithms? I'd like to read up on that topic.
These two books might be useful,
1) https://www.oreilly.com/library/view/electronic-design-autom... 2) https://www.crcpress.com/Electronic-Design-Automation-for-IC...
On the first book, see chapter 10 - 12 (on floorplanning, placement, and routing). End of chapter 11 points you to some literature survey as well. But the book itself is somewhat dated (published in 2009)
I haven't read the second book but it's much more recent (published in 2018), it also has chapters on placement, and routing.
Re: Deep Learning and Its Implications for Computer Architecture and Chip Design
#10brain floating point ... cool name i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778 > A particularly interesting research direction puts these three trends together, with a system running on large-scale ML accelerator hardware, with a goal of being able to train a model that can perform thousands or millions of tasks in a single mod…
> yup, he is building a brain
Plus applying ML to improving the underlying hardware and software. On the other hand it's not recursive yet and moore's law ending throws a wrench into exponential self-improvement, but it's still a little concerning.