Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A week-long programming retreat

facebook.com

Post reply on HN
Print view View original

141–150 of 158 posts

Re: A week-long programming retreat

#141

Post by IamNotAtWork »

arnioxux wrote: post #19

I second his recommendation of CS231N: http://cs231n.stanford.edu/ You can probably go through the whole thing including assignments in under a week full-time.

Are these courses being taught by graduate students? The three main instructors seem like they are students themselves, with an army of under and grad TAs.

Pity that you pay so much money to attend Stanford only to be taught by your peers. Not knocking on Stanford as this is how is being done much everywhere in the undergrad level now.

Re: A week-long programming retreat

#142

Post by cobalt »

mxschumacher wrote: post #127
munificent wrote: post #109

Earlier quoted context omitted.

I think the subtext of your comment here is something like, "Look how productive he is without mastering the real tools we Unix hackers use! He can get by with second rate Windows stuff!" Maybe I'm reading you wrong. But if I am right, it's good to take a look outside of the Unix bubble. Visual Studio is literally the world's most sophisticated developer tool. More human hours of engineering have been poured into it…

does it beat intellj's IDEs?

yes

Re: A week-long programming retreat

#143

Post by jorgemf »

Amezarak wrote: post #128
jorgemf wrote: post #94

Earlier quoted context omitted.

> not any sort of fundamental algorithmic paradigm shift I think I cannot agree with this. There has been a lot of improvements to the algorithms to solve problems and the pace has speed up thanks to GPUs. You just cannot make a neural network from 15 years ago bigger and think it is going to work with modern GPUs, it is not going to work at all. Moreover, new techniques have appeared to solve other type of problems.…

Do you have any suggestions on reading material?

I have read a lot of papers and usually, you end up in the original one if you check the references. But you have to check papers in specific areas. I am not aware of any good document where everything is there.

Re: A week-long programming retreat

#144

Post by jorgemf »

neiled wrote: post #125
jorgemf wrote: post #123

Earlier quoted context omitted.

> his first impulse was to write backprop from scratch backprop is a very simple algorithm, nothing to fear there. The problems are to calculate the derivates if you want to be flexible building your model. But for feedforward networks with sigmoid activation, the equations to update the weights are a joke.

I'm not sure we agree on the definition of very simple. It maybe very simple if you already know it...

Backpropagation is based on linear optimization (aka calculate the maximum or minimum of a function based on the derivades of that function, this is taught before university in my country). And also in the chain rule to calculate the derivades of functions (first year of university).

But I meant, if you see the equations and the steps without understanding completely the insights, it is a joke of an algorithm. It just does some multiplications and applies the new gradients, move to the previous layer and repeat.

Re: A week-long programming retreat

#145

Post by jorgemf »

test6554 wrote: post #139
jorgemf wrote: post #123

Earlier quoted context omitted.

> his first impulse was to write backprop from scratch backprop is a very simple algorithm, nothing to fear there. The problems are to calculate the derivates if you want to be flexible building your model. But for feedforward networks with sigmoid activation, the equations to update the weights are a joke.

It's been a while, but I remember backprop starting at the end of the neural net, and working backwards. Each weight that contributed to a wrong answer had its weight value weakened or even reversed by some small factor. And each weight that contributed to a correct answer had its weight value strengthened by some small factor. So it's probably as simple as newWeight = oldWeight +/- (stepValue * someFactor)

exactly, the someFactor is the error of the next layer calculated with the derivades of the functions (as you would calculate the minimum of a funtion using the derivades). The tricky part is to calculate the derivades, but since auto differentiation we can do a lot of cool stuff.

Re: A week-long programming retreat

#146

Post by tostitos1979 »

jorgemf wrote: post #145
test6554 wrote: post #139

Earlier quoted context omitted.

It's been a while, but I remember backprop starting at the end of the neural net, and working backwards. Each weight that contributed to a wrong answer had its weight value weakened or even reversed by some small factor. And each weight that contributed to a correct answer had its weight value strengthened by some small factor. So it's probably as simple as newWeight = oldWeight +/- (stepValue * someFactor)

exactly, the someFactor is the error of the next layer calculated with the derivades of the functions (as you would calculate the minimum of a funtion using the derivades). The tricky part is to calculate the derivades, but since auto differentiation we can do a lot of cool stuff.

The differentiation is probably why I wouldn't have bothered to hack it myself. Curious how you/others would tackle it? What do you mean by auto differentiation?

Re: A week-long programming retreat

#147

Post by hota_mazi »

BenoitEssiambre wrote: post #88

"I initially got backprop wrong both times, comparison with numerical differentiation was critical! It is interesting that things still train even when various parts are pretty wrong — as long as the sign is right most of the time, progress is often made." That is the bane of doing probabilistic code. Errors show up not as clear cut wrong values or crashes but as subtle biases. You are always wondering, even when it…

It's actually not very different from graphic programming, where a simple rounding error can cause all kinds of troubles, from very small (surfaces or ray not reflecting perfectly where they should be) or very big (completely messing your entire rendering).

Both activities very much resemble chaotic systems and they are both very challenging to debug.

Re: A week-long programming retreat

#148

Post by shbm »

neiled wrote: post #125
jorgemf wrote: post #123

Earlier quoted context omitted.

> his first impulse was to write backprop from scratch backprop is a very simple algorithm, nothing to fear there. The problems are to calculate the derivates if you want to be flexible building your model. But for feedforward networks with sigmoid activation, the equations to update the weights are a joke.

I'm not sure we agree on the definition of very simple. It maybe very simple if you already know it...

https://m.youtube.com/watch?v=i94OvYb6noo&feature=youtu.be&t...

You won't regret. One of the best explanation of backprop on the internet.

Re: A week-long programming retreat

#149

Post by jorgemf »

tostitos1979 wrote: post #146
jorgemf wrote: post #145

Earlier quoted context omitted.

exactly, the someFactor is the error of the next layer calculated with the derivades of the functions (as you would calculate the minimum of a funtion using the derivades). The tricky part is to calculate the derivades, but since auto differentiation we can do a lot of cool stuff.

The differentiation is probably why I wouldn't have bothered to hack it myself. Curious how you/others would tackle it? What do you mean by auto differentiation?

https://en.wikipedia.org/wiki/Automatic_differentiation

Re: A week-long programming retreat

#150

Post by arnioxux »

IamNotAtWork wrote: post #141
arnioxux wrote: post #19

I second his recommendation of CS231N: http://cs231n.stanford.edu/ You can probably go through the whole thing including assignments in under a week full-time.

Are these courses being taught by graduate students? The three main instructors seem like they are students themselves, with an army of under and grad TAs. Pity that you pay so much money to attend Stanford only to be taught by your peers. Not knocking on Stanford as this is how is being done much everywhere in the undergrad level now.

In the first few lectures of the course you get a pretty good history of deep learning and you'll see it didn't really take off until around ~2012. And the reasons for it taking off is mostly because people are getting better at the black magic of training a deep network.

So these grad students are exactly the people you want to learn from because they have done the dirty work of fiddling with parameters to know what tricks work and what doesn't. It's probably preferable to a more theory-heavy course because very few people (not even the more experienced professors) understand why those tricks work.

Note: I took an older version of the course which was started by Andrej Karpathy who was a grad student at the time but is now the Director of Artificial Intelligence at Tesla.

Post reply on HN