Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just know stuff (or, how to achieve success in a machine learning PhD)

kidger.site

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#71

Post by patrickkidger »

albertzeyer wrote: post #53

I have learned many of these things at some point. Unfortunately, I have also forgotten most of it. Some of it I still remember to some degree, or just the basic idea, and can probably reconstruct it given some time, or easily look it up and remember. However, many of the things I have completely forgotten, and it would take more time to understand it again. I am not sure I can keep so many things active in my memory…

Haha, so I actually have an atrocious memory! Famously so amongst my friends, I never remember what we've discussed.

When I wrote this list I certainly wasn't expecting/recommending all of this to stay in the reader's head forever.

Rather: if you've worked with something deeply at one time, then -- even if you've forgotten the details -- you still can pattern-match on it later. And then look up whatever you've forgotten!

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#72

Post by patrickkidger »

steppi wrote: post #31

Knowing things is good, but I think the real benefit of a PhD is in developing the skill to learn new things quickly and fill in any gaps as needed. It’s important to get practice learning many things in depth but the goal isn’t to become a storehouse of knowledge, it’s to develop the ability assimilate existing knowledge and apply it to figuring new things out. The amount of topics I’ve studied in depth dwarfs this…

Completely agreed!

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#73

Post by burnished »

mochomocha wrote: post #58
burnished wrote: post #48

Earlier quoted context omitted.

Should they be? In this case wouldnt it end up being false modesty? Like, if this person cant say "Look at me, I am UNUSUALLY INTELLIGENT!" then who can?!

> Like, if this person cant say "Look at me, I am UNUSUALLY INTELLIGENT!" then who can?! No one, that's my point. If academia has not taught to the author that his intelligence isn't unusual, the workforce of his new employer certainly will. Listing github stars and twitter followers in the second paragraph as an achievement to me transpires lack of maturity and a need for external validation. On the bright side, bei…

I see, you read it as a form of self aggrandizement. I think your reading was wrong (in addition to being unkind), but wouldnt disagree with the core value under discussion.

I think a fair test to apply is whether or not the thing is a fact - if it is a fact, and you think it is impressive, that might say more about you than the author. I feel like otherwise you are asking people to self censor too strongly for fear of betraying some perverse sense of modesty that does now allow for anyone to have done anything worth noticing at all.

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#74

Post by gautamcgoel »

I just wrapped up a machine learning PhD at Caltech (now doing a postdoc in ML at Berkeley) and I disagree strongly with this article. What matters isn't knowing a bunch of random stuff, but rather writing/speaking skills, a willingness to learn new things, perseverance in the face of setbacks, creativity, having enough EQ to navigate the advisor-advisee relationship and departmental politics, and most of all, an ability to follow through and actually get things done. These "intangible" skills are far more important than having any specific knowledge.

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#75

Post by bitL »

qsort wrote: post #4

"Just know stuff" deeply resonates. Even much below the author's level, as is the case for me, technical knowledge dominates everything else by orders of magnitude. I'm still appalled at how people can manage to gather the courage to utter they don't need math. A great list, too. I guess I have stuff to brush up on for the next 10 to 20 years?

...and then you meet your new boss who has no clue about anything and makes your life hell when his imagination doesn't match the "stuff you know".

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#76

Post by burnished »

patrickkidger wrote: post #67
mochomocha wrote: post #58

Earlier quoted context omitted.

> Like, if this person cant say "Look at me, I am UNUSUALLY INTELLIGENT!" then who can?! No one, that's my point. If academia has not taught to the author that his intelligence isn't unusual, the workforce of his new employer certainly will. Listing github stars and twitter followers in the second paragraph as an achievement to me transpires lack of maturity and a need for external validation. On the bright side, bei…

I'm sorry it came across this way for you! Rather, I'm just outlining why folks seem to keep asking me this question. :)

I think your post combined with the responses to it are a great example of how messages become stripped of all nuance, even by otherwise very educated/intelligent people. A lot of the critique here ignores the nuance and specificity you give just in the opening paragraphs.

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#77

Post by uoaei »

This list is great for moving things from the "unknown unknown" (U-U) to the "known unknown" (K-U) bucket. It's relatively easy to move the things from the K-U to the K-K bucket just by virtue of knowing enough search terms and places to start.

I think all of the topics in TFA's list could come into play at some point (I have explored something to do with the majority of these concepts during my work in private research) and it is important to know how compiler optimizations are done, e.g. XLA and Jacobian accumulation techniques to design fast models. I don't think it matters that you don't master all of them, but being able to quickly spin back up to comprehension upon a relatively brief refresh is pretty important when it comes to algorithm design and prototyping.

Re: Just know stuff (or, how to achieve success in a machine learning PhD)

#78

Post by patrickkidger »

shihab wrote: post #2

I didn't expect this to be as helpful as it actually was. Great list. Can anyone suggest me something similar for HPC domain?

So HPC means like ten different things.

For example another commentor mentions low-latency concerns in finance, and that's something I have zero experience with.

HPC has often also meant writing a lot of C++ to do e.g. MD or something.

These days, I consider myself HPC-adjacent -- I write scientific ML software, often for use on pretty beefy hardware (TPU pods etc.) So at least for that, here's an off-the-cuff list of a few items that come to mind:

- Know JAX. Really, really well: its internals, how its transforms work. It's definitely a bit bumpy in places, but it's still one of the best things we have for easily scaling programs, e.g. through `jax.pmap`, being able to test on CPU and then run on TPU, etc.

- Triton! New(-ish) kid on the block for GPU programming.

- How CPUs work: L1/L2/L3 caches, branch prediction, etc. Parallelism via OpenMP.

- How GPUs work: warps etc.

- How BLAS works (e.g. tiling)..

- Compiler theory. Inlining functions, argment aliasing, NRVO, ...

- Know autodiff well. E.g. have a read of the Dex paper, and the concerns with doing autodiff through index operations. Modern scientific computing is moving towards a ubiquitously autodifferentiable future.

- ... plus loads more, haha. Probably I'm still missing ten different things that another reader considers crucial.

Post reply on HN