Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

drive.google.com

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#91

Post by dekhn »

I guess it's not totally surprising that Dean's undergrad thesis was on training neural networks and the main choice was between or in-graph replication. This is still one of the big issues with TensorFlow today.

One thing most people don't get is that Dean is basically a computer scientist with expertise in compiler optimizations, and TF is basically an attempt at turning neural network speedups into problems related to compiler optimization.

I'd like to thank my undergrad university for hosting my undergrad thesis for 25 years with only 1-2 URL changes. Some interesting details include: Latex2Html held up, mostly, for 25 years and several URL changes. The underlying topic is still relevant (training the weight coefficients of a binary classifier to maximize performance) to my work today, even if I didn't understand gradient descent or softmax at the time.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#92

Post by dekhn »

mcilai wrote: post #2

Quite incredible that he was interested in NNs back in 1990. He closed this thread very well.

The early 90s were an interesting time for NNs and other machine learning systems. I remember getting really interested, but being told that "NNs with more than 1 layer can't really be trained", so I went into simulation rather than training. It's really great that GPUs and deep backprop arose to recover the stature of NNs.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#93

Post by sigjuice »

pknerd wrote: post #53

Interesting coding style with too much whitespace. Is it some standardized pattern? I found something similar in the code written by John Carmack.

This is the usual amount of whitespace. A couple of examples

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

https://github.com/apple/darwin-xnu/blob/master/libsyscall/m...

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#94

Post by phkahler »

Certhas wrote: post #90
phkahler wrote: post #60

Earlier quoted context omitted.

That seems to be a key difference between science and engineering. One likes to survey the field and insist that their paper offers something novel - no matter how big or small. The other just wants to do some solid work and get the results.

If you want to be snarky: The one is research the other is documentation.

I'm not really trying to be snarky, just point out the difference in approach. Research is in fact documentation - of what others have done.

Papers that are entirely surveys or comparisons of different approaches can be excellent and would make good citations in any practical work.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#95

Post by Certhas »

phkahler wrote: post #94
Certhas wrote: post #90

Earlier quoted context omitted.

If you want to be snarky: The one is research the other is documentation.

I'm not really trying to be snarky, just point out the difference in approach. Research is in fact documentation - of what others have done. Papers that are entirely surveys or comparisons of different approaches can be excellent and would make good citations in any practical work.

Not really. The point is that in research we want to generate and further knowledge. This is distinct from generating and documenting facts. If you don't link into the web of knowledge there is (implicitly: leave that task up to the reader) you are just documenting facts.

This is not academic. What did reading this Master thesis teach me? That two approaches perform reasonably (by what standard?) with a size trade-off. That's an excellent start but also leaves open many questions: Why these two approaches? Are there reasons to expect they are better suited than other approaches in the literature? Were these results expected? Can I expect them to generalize? Do they paint a coherent picture on the performance of different designs in various contexts or are they surprising?

A lot of this is about generality of the knowledge gained. As a mere fact ("Two implementations of two algorithms that solve one problem perform slightly differently") it's not very interesting unless I have that exact specific problem myself. If I do, I would still need to find the paper. But if it is linked into a wider web of knowledge ("In paper [X] it was found that this algorithm performs well on tasks that have something in common with our problem, paper [Y] and [Z] suggest that we should expect a trade off for small sizes. Generally nothing is known about what should be algorithms well suited to the problem at hand.") it allows me to reason about situations.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#96

Post by silverlake »

mcilai wrote: post #2

Quite incredible that he was interested in NNs back in 1990. He closed this thread very well.

I’m almost Dean’s age. My undergrad project was evolving NN with genetic algorithms. AI was popular, but funding died abruptly soon after.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#97

Post by bipson »

fjsolwmv wrote: post #66
bipson wrote: post #51

Earlier quoted context omitted.

I have in the past been subadvisor to various bacc. theses. I value conciseness dearly, and prefer quality over quantity in scientific writing, i.e. I would accept incredibly short theses, if the content is sufficiently presented (reproducible and comprehensive), and most of all, contains a valuable contribution. The reason I typically have to request "more verbiage" and an own section on the state of the art, is bec…

sitcom? It's strange to expect an undergrad to do new and interesting work when they haven't even finished their basic education in the field. Solve problems that are easy but not important enough for professional academics, sure. Do an application of a standard idea in a specific environment (like porting an pp to Android), sure. But not new approaches to the field.

There are various shades between scientific breakthrough and "yet another app/Server Tool", of which hundreds were implemented in the past, which can be solved by following random blog-posts, and (in the worst case) make no sense, even besides academic rigor.

Scientific work should fulfill at least some standards, and IMHO this includes undergrad theses.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#98

Post by GuiA »

nerdponx wrote: post #74
GuiA wrote: post #30

Earlier quoted context omitted.

I put a lot of effort in my undergraduate thesis, but none of the professors on my committee had much interest in advising me; and after my defense, the only professor who really gave me his undivided attention came to me and said “I’m glad you’re not staying here for grad school; you’re way too good for this place”. ¯\_(ツ)_/¯ Comparing your journey to others’ is pointless.

Sorry you're being downvoted. I don't think you're trying to detract from the accomplishment here, but you are raising an important point: at an age like this, good mentorship, leadership, and guidance is essential. There is a very small number of truly gifted people who end up discovering things on their own at a young age. Most gifted people who discover something do so with the benefit of a mentor who can work wit…

Thanks for your comment. This was not to detract from Jeff Dean's work in any way - obviously he is a talented, hard working engineer.

This was more to share with 'halflings that comparing one's achievements to someone else's is rarely something that bears much fruit.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#99

Post by tdb7893 »

akhilcacharya wrote: post #33
wenc wrote: post #17

Earlier quoted context omitted.

In the U.S., one's undergraduate institution does not correlate to success as much as it does in certain countries like France or Japan, where universities are a pipeline for elite selection and grooming. Also, not all intelligent American kids can or want to go to elite schools, even if they are academically qualified. In the U.S., you often hear stories of kids turning down really good schools for ones they felt we…

> Owing to its population and economy, the U.S. has a large enough talent pool that the top percentile students at large, well-funded state schools (of which UMN is an example) are plenty smart. If you were to meet the really smart top-5-percentile kids from such state colleges (I have), you'd have no doubt that many of them could have attended MIT or CMU. > To be sure, good colleges can give you a headstart in life…

I graduated from a school that I think was #40 when I was there and got a job at one of the top tier companies but each person has their own experiences. Now people couldn't care less about where I went to college (also fun fact my GPA was 3.2 so it wasn't even that good but luckily no one cares about that either).

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#100

Post by mlthoughts2018 »

busyant wrote: post #84
mlthoughts2018 wrote: post #67

Earlier quoted context omitted.

Of course people want you to write less when it means less work for them. But think of an academic review committee for a senior thesis. They aren’t going to heavily read your paper, just decide if it meets compliance standards for some preserved artifact for your graduation. But they can veto what you’ve written and send it back to someone else (your advisor likely) and have that person request edits. So this puts t…

I guess my experience has been different than yours. I wrote an undergrad thesis and I felt like my advisor and two readers cared about what I was saying and how I said it. Same is true for my Ph.D. My committee members seemed to care deeply about the work (I think they just didn't want to be associated w/ crap research).

Yes, I agree the two experiences you describe sound very rare from my own experience and my colleagues’ and friends’ experiences in undergrad, grad school and authoring papers in academic and industrial settings.
Post reply on HN