Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

youtube.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#21

Post by imakecomments »

agibsonccc wrote: post #13
minimaxir wrote: post #2

Although TensorFlow/Keras allows anyone to implement Deep Learning easily, that doesn't mean that they will get hired for a relevant job position without a PhD. Most Deep Learning jobs, and even relatively mundane Data Scientist jobs nowadays, want a PhD from my experience. There is a surplus of Statistics/CS PhDs, why would a company hire someone without one if they do not have to? Without a relevant job position, k…

I have seen this from multiple angles. I used to teach at a data science bootcamp where many of the students got hired by big companies. I've also been running a deep learning startup for the last few years and have hired quite a few people. Many of our team don't have phds but can still write backprop code for even complex modules like inception among other things. A lot of my students didn't have phds either. A few…

Regarding your book, have you expanded on the math section? I saw somewhere a draft of the material and the math review seemed to be broken up into short paragraphs. These short paragraphs lacked examples and appeared to assume previous background knowledge in the subject, which seems contradictory to the book's title and aim. For example.. I believe you mentioned somewhere "The Jacobian is a m x n matrix containing the 1st order partial derivatives of vectors with respect to vectors." -- Since I have a math background I can understand what you write. But for someone with little to no math background (e.g. a software practitioner) this may throw them off.

I am hesitant to recommend your book to a true practitioner due to the assumed knowledge presented within the math section. I think a better treatment of mathematics would assume the reader has little to no background but is intelligent enough to learn ground up the specific use cases of the mathematics for the deep learning techniques presented in the book. See: http://www.deeplearningbook.org/ for better treatment of the math review. It seems more thorough and makes less assumptions about the math background of the reader.

I would love to recommend your book to a practitioner but I'm afraid the math section (the version I reviewed) would scare them off/they would get little out of it.

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#22

Post by agibsonccc »

imakecomments wrote: post #21
agibsonccc wrote: post #13

Earlier quoted context omitted.

I have seen this from multiple angles. I used to teach at a data science bootcamp where many of the students got hired by big companies. I've also been running a deep learning startup for the last few years and have hired quite a few people. Many of our team don't have phds but can still write backprop code for even complex modules like inception among other things. A lot of my students didn't have phds either. A few…

Regarding your book, have you expanded on the math section? I saw somewhere a draft of the material and the math review seemed to be broken up into short paragraphs. These short paragraphs lacked examples and appeared to assume previous background knowledge in the subject, which seems contradictory to the book's title and aim. For example.. I believe you mentioned somewhere "The Jacobian is a m x n matrix containing…

We have appendixes covering the basics there. I actually recommend deeplearningbook.org myself for a reference.

The book is meant to contain simple examples oriented towards engineers building applications rather than deriving backprop.

The book isnt called the definitive guide for a reason ;)

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#23

Post by imakecomments »

agibsonccc wrote: post #22
imakecomments wrote: post #21

Earlier quoted context omitted.

Regarding your book, have you expanded on the math section? I saw somewhere a draft of the material and the math review seemed to be broken up into short paragraphs. These short paragraphs lacked examples and appeared to assume previous background knowledge in the subject, which seems contradictory to the book's title and aim. For example.. I believe you mentioned somewhere "The Jacobian is a m x n matrix containing…

We have appendixes covering the basics there. I actually recommend deeplearningbook.org myself for a reference. The book is meant to contain simple examples oriented towards engineers building applications rather than deriving backprop. The book isnt called the definitive guide for a reason ;)

I didn't notice the appendices in the draft. This may help, but the version I reviewed has a brief review of Linear Algebra and Statistics in Chapter 1. The language is written in a way that assumes mathematical familiarity. Putting this level of 'sophistication' in a "practitioners" book early on may turn off a certain demographic of readers. I say 'sophistication' because it's relative to the reader-- to someone with a math degree this section isn't sophisticated (in fact they'd find it completely trivial) and can be skipped. To a software engineer that hasn't taken a Linear Algebra/Statistics class in 10+ years this may appear too much for them. You risk losing the readers entirely or having them skip those sections. Again, this review is not friendly for the beginner and the 'practitioner' title is misleading here.

In my opinion it wouldn't be too difficult or much effort to define what these mathematical objects are and show basic examples with basic computations to solidify the concepts. The notion of gradient descendent & derivates (or partial derivatives) isn't that difficult to understand and could be easily explained in a page or less.

For example when you discuss the Outer Product:

"This is known as the “tensor product” of two input vectors. We take each element of a column vector and multiply it by all of the elements in a row vector creating a new row in the resultant matrix."

It would be nice for the beginner to see an example of this and as stated it wouldn't take much space in the book to provide one. I think these sort of things would differentiate your book from others. If you made it more friendly more 'practitioners' would be willing to read/use it end-to-end.

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#24

Post by agibsonccc »

imakecomments wrote: post #23
agibsonccc wrote: post #22

Earlier quoted context omitted.

We have appendixes covering the basics there. I actually recommend deeplearningbook.org myself for a reference. The book is meant to contain simple examples oriented towards engineers building applications rather than deriving backprop. The book isnt called the definitive guide for a reason ;)

I didn't notice the appendices in the draft. This may help, but the version I reviewed has a brief review of Linear Algebra and Statistics in Chapter 1. The language is written in a way that assumes mathematical familiarity. Putting this level of 'sophistication' in a "practitioners" book early on may turn off a certain demographic of readers. I say 'sophistication' because it's relative to the reader-- to someone wi…

Thanks for the feedback! I'll share this with my editor and see what we can do.

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#25

Post by drvdevd »

lightcycle wrote: post #9
minimaxir wrote: post #2

Although TensorFlow/Keras allows anyone to implement Deep Learning easily, that doesn't mean that they will get hired for a relevant job position without a PhD. Most Deep Learning jobs, and even relatively mundane Data Scientist jobs nowadays, want a PhD from my experience. There is a surplus of Statistics/CS PhDs, why would a company hire someone without one if they do not have to? Without a relevant job position, k…

Getting hired isn't always the point. TensorFlow and other tools make deep learning a fairly accessible solution for ordinary developers to offer to customers. It lets me say "yes" (or "maybe") to requests that would not have been feasible previously.

This. For me the whole point of being self taught has always been so I could get down to work and build things :)

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#26

Post by nl »

Or you could just implement this (A submission from Google Brain to ICLR 2017):

In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.84, which is only 0.1 percent worse and 1.2x faster than the current state-of-the-art model. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art.[1]

To translate that, they built and train a RNN to design neural networks. These machine designed networks are almost equal to the best human designed network on an image-recognition benchmark, and outperform the best human-designed systems on a text understanding benchmark.

[1] http://openreview.net/forum?id=r1Ue8Hcxg

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#27

Post by mamon »

danieltillett wrote: post #11
zeratul wrote: post #8

99.3% accuracy is not that great. Currently the best performer on this data set has 99.77% accuracy: http://yann.lecun.com/exdb/mnist/

What is the human accuracy on this data set?

more interesting question: if some of those "digits" are so hard to recognize even by humans then how can we ever label them with the one true "correct" answer?

Do we know a person who draw those digits and ask "what artist had in mind when making this masterpiece" ? And even then someone might have been trying to draw the "2" but end effect looks more like "3".

I think that some of the test cases simply don't have definitive answer and trying to reach 100% accuracy is just misguided effort.

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#28

Post by mamon »

gajomi wrote: post #12
zeratul wrote: post #8

99.3% accuracy is not that great. Currently the best performer on this data set has 99.77% accuracy: http://yann.lecun.com/exdb/mnist/

I wonder, is there a good reason why these accuracies are reported on a logscale by convention (say, as power of 2 from 50%)?

To make even small difference look bigger, cause even a 0.1% improvement over previous best result is considered big deal.

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#29

Post by eshvk »

minimaxir wrote: post #2

Although TensorFlow/Keras allows anyone to implement Deep Learning easily, that doesn't mean that they will get hired for a relevant job position without a PhD. Most Deep Learning jobs, and even relatively mundane Data Scientist jobs nowadays, want a PhD from my experience. There is a surplus of Statistics/CS PhDs, why would a company hire someone without one if they do not have to? Without a relevant job position, k…

> Most Deep Learning jobs, and even relatively mundane Data Scientist jobs nowadays, want a PhD from my experience.

Uhh..No. I have been doing this for 5 years or so. I don't have a PhD; most people we have hired don't have PhD. Some who write NIPS papers (very different from a "Medium thought piece") on their spare time. Now what we optimize for is relevant experience and the ability to not just throw a framework at something. That is highly correlated with having worked on this for a while or have strong math skills. Guess, what? Some of those who have these skills have a PhD. Some, not all.

Re: Tensorflow and Deep Learning, Without a PhD, Martin Gorner, Google [video]

#30

Post by danieltillett »

mamon wrote: post #27
danieltillett wrote: post #11

Earlier quoted context omitted.

What is the human accuracy on this data set?

more interesting question: if some of those "digits" are so hard to recognize even by humans then how can we ever label them with the one true "correct" answer? Do we know a person who draw those digits and ask "what artist had in mind when making this masterpiece" ? And even then someone might have been trying to draw the "2" but end effect looks more like "3". I think that some of the test cases simply don't have d…

Another interesting question is which approach most closely matches the errors made by humans.
Post reply on HN