Live data from Hacker News

Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

drive.google.com

81–90 of 102 posts

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#81
post #62
post #49

Earlier quoted context omitted.

It's kind of remarkable. There really is no literature review in this paper. As a supervisor I would have no problem with a content part of this length, but I would also insist on doing the scholarly work that is not demonstrated here. Don't just throw out some code and describe it but put it into the context of what exists. Give credit to where ideas originate. That shouldn't add too much. No more than a few pages.…

From the readers perspective, what would be the point?

As a reader, I'm often left trying to figure out whether the author is presenting something new, or is rehashing old ideas (which, btw, is a good thing sometimes, but I want to know that's someone tried this once before). I want to know exactly what the new contribution of the paper is, and how it fits in the universe of contributions.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#82
post #47

An underappreciated aspect of this is finding an academic department that would allow you to submit something this concise as a senior thesis. My experience, mostly in grad school, was that anyone editing my work wanted more verbiage. If you only needed a short, one-sentence paragraph to say something, it just wasn’t accepted. There had to be more. Jeff Dean is an uncommonly good communicator. But he also benefited f…

I had to write a 10k word undergrad final paper for law school (in Europe, law school is a regular university study, with a 3 year LLB and 1 or 2 year LLM). At around 8k or so, I went to my supervisor and said 'look, I've said everything I wanted to say, and in a drawn out way already which I' not happy about. If I have to add more, I will need to start another topic, and I'd rather keep this paper focused and contin…

Reminds me of page count requirements in middle school. I'd write 4 pages, and in order to get to 5 I'd make every period and comma a higher font size. This was after they figured out my line spacing and margin tricks.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#83
post #30

As always, Jeff Dean doesn't fail to inspire respect. Tackling a complex problem (still relevant today) at an early age, getting great results and describing the solution clearly/concisely. My master thesis was ~60 pages long, and was probably about 1/1000 as useful as this one.

I put a lot of effort in my undergraduate thesis, but none of the professors on my committee had much interest in advising me; and after my defense, the only professor who really gave me his undivided attention came to me and said “I’m glad you’re not staying here for grad school; you’re way too good for this place”. ¯\_(ツ)_/¯ Comparing your journey to others’ is pointless.

I had a bitter experience while pursuing my master's thesis at a pretty big firm. The people responsible for advising me were shameless in accepting that no one really cares about how well I have done my thesis (they were trying to defend as to why no one cared enough to spend time and guide me), all that matters is finishing it and moving on. If you happen to be a perfectionist, such experiences break you.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#84
post #55

Earlier quoted context omitted.

> anyone editing my work wanted more verbiage I _never_ got that feedback. My mentors all emphasized economy of language and nobody cared how "thick" my thesis was. This is a pretty amusing story about verbiage. Back in the old days, you would send a manuscript/research article to colleagues/friends by _snail-mail_ to get their feedback. You'd wait a month, and maybe they would mail a 'red-inked' copy of your manuscr…

Of course people want you to write less when it means less work for them. But think of an academic review committee for a senior thesis. They aren’t going to heavily read your paper, just decide if it meets compliance standards for some preserved artifact for your graduation. But they can veto what you’ve written and send it back to someone else (your advisor likely) and have that person request edits. So this puts t…

I guess my experience has been different than yours.

I wrote an undergrad thesis and I felt like my advisor and two readers cared about what I was saying and how I said it.

Same is true for my Ph.D. My committee members seemed to care deeply about the work (I think they just didn't want to be associated w/ crap research).

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#85

Earlier quoted context omitted.

Since you clearly like math, suppose there are 20 million students applying to college every year. One million of them are very bright (2-sigma intelligence). Suppose that every Ivy League accepts around two thousand students every year. Can the sum of Ivy Leagues fit all of the smart people? Bonus question: What if we change our constraints to only account for 3-sigma people? Does our conclusion change?

Of course there's a distribution of those folks at all other universities, especially at top public ivies like GTech and UCB. But the cumulative effect of being at a lower tier institution can be pretty significant when searching for future opportunities. One thing I've noticed among people at my school is that even if they are very capable, they often don't apply to top internships, top programs for grad school etc,…

I think you made an astute observation: top people at lower ranked schools sometimes don't apply to top internships or grad schools because they don't see it as an option.

This speaks to their lack of confidence more than their capability. Perhaps that is one of the advantages a good school, a good peer group, or a good network can confer: the confidence to aim higher.

People don't think they're good enough... which may be true, but no one can truly know until they try. Self-limiting thinking is particularly prevalent in rust-belt cities and regions where knowledge or achievement is not prized, so people in knowledge-intensive fields have no models to emulate.

And sometimes when they try (it's not unusual for graduates of lower ranked universities to send out 300+ resumes only to get single digit responses), they get demoralized when they don't succeed on their first few tries, when in fact there's more than one path in life -- if one doesn't have natural advantages, one might have to embrace the more circuitous path(s). This can mean joining a startup, going to a better grad school than one's undergrad, moving to a better city to upgrade one's peer groups (this is more important than most people think [1]), etc.

Life can surprise you if you keep trying and pivoting (ugh cliche, but there it is). There's an element of randomness and stochasticity in a free market, and I've seen enough counterexamples to distrust a static conception of how things "should be". (except for some stodgy areas like investment banking that only hire from certain schools; but even then there are backdoors)

* You're a new graduate, and the hardest hurdle you have to overcome is to get in the door. If you manage to do that and are able to prove yourself, your undergrad degree will become less and less and important. If you google Fortune 500 company CEOs, especially in non-tech companies, (you can do this exercise for yourself) you will learn that many of them went to non-elite schools for undergrad. For all its elite colleges, America is not really an academic-technocratic society (unlike countries like Germany where most CEOs have Ph.D.s). There are elements of William James' pragmatic philosophy that still influence the thinking in this country -- getting results is more important than academic knowledge.

[1] http://www.paulgraham.com/cities.html

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#86
post #80
post #46

Earlier quoted context omitted.

Pretty interesting, reading his short bio I learned for the first time about AHPCRC ( https://ahpcrc.stanford.edu ) of which prof. Kumar was head of for about 7 years, the US military is indeed involved almost everywhere in SV.

Read Steve Blank's "The Secret History of Silicon Valley". The US military created Silicon Valley.

I've always wondered what the "secret" in the title refers to. This is pretty common knowledge among people who are from here, is it only secret from the people rushing in to work at FAANG?

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#87
post #2

Quite incredible that he was interested in NNs back in 1990. He closed this thread very well.

As all the other responses point out, NNs were red hot back then.

The interest in NNs was ignited (in part) by this double volume collection of essays called "Parallel Distributed Processing" edited by Rumelhart and McClelland.

Dean even cites them. And, if you read the contributors, it contains many (though not all) of the heavy hitters.

Reading back on it, it will sound very familiar. All the amazing breakthroughs: object recognition, handwriting recognition etc all seemed to be there. But all that rapid progress just seemed to stop. There was this quantum leap and then you were back to grinding out for even 0.1% improvement.

For those who stuck through the second winter, things obviously paid off.

The intro essay is online:

https://stanford.edu/~jlmcc/papers/PDP/Chapter1.pdf

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#88
Really interesting and innovative early work, and I think it also explains why tensorflow does not support within layer model parallelism. It's amazing how much our early experiences shape us down the road.

My entire career has consisted of reimplementing bits and pieces of things I've previously built all the way back to high school and then reimplementing whatever was new on the previous round in the next one.

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#89
post #54

Earlier quoted context omitted.

Thanks for the list but I am not entirely sure about Sergey Brin and Larry Page’s alma mater.

Interesting fact: those two went on to create the search engine Google. You should check it out.

I meant I am not sure about their alma mater not being among the top reputed universities in USA, meaning I implied they were indeed from one of the top reputed universities in USA.

Apologies for not making myself clear enough :(

Re: Google’s Jeff Dean’s undergrad senior thesis on neural networks (1990) [pdf]

#90
post #49

Earlier quoted context omitted.

It's kind of remarkable. There really is no literature review in this paper. As a supervisor I would have no problem with a content part of this length, but I would also insist on doing the scholarly work that is not demonstrated here. Don't just throw out some code and describe it but put it into the context of what exists. Give credit to where ideas originate. That shouldn't add too much. No more than a few pages.…

That seems to be a key difference between science and engineering. One likes to survey the field and insist that their paper offers something novel - no matter how big or small. The other just wants to do some solid work and get the results.

If you want to be snarky: The one is research the other is documentation.
Post reply on HN