Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

161–170 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#161

Post by molf »

BoredomIsFun wrote: post #131
molf wrote: post #109

Earlier quoted context omitted.

> unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring" That's what I would call exploring. I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.

> That's what I would call exploring. Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing. > Fooling around and getting results is what made it fun. Understanding came later. It is not how LLMs are "built from scratch".

No it's not useful, no you don't know what you're doing, and no it is not 'proper'.

But you're learning.

Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.

Re: I were 17, I'd learn how to build LLMs from scratch

#162

Post by bmacho »

yobbo wrote: post #140
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age. It is viable as a toy project, but there are vanishingly few career opportunities.

It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.

Re: I were 17, I'd learn how to build LLMs from scratch

#163

Post by mike_hearn »

kevmo314 wrote: post #136
DanielHB wrote: post #113

Earlier quoted context omitted.

But can you _sell_ that? If you can't you can't get a job doing it.

I suspect most who worked at Google did not work on Google infrastructure before getting there.

No, but the skills Google needed (back then) were just normal programming and sysadmin skills scaled up. I had eight interviews that covered Linux sysadmin, programming, debugging, networking, maths and more. If Google hadn't wanted me there'd have been plenty of other companies who needed those skills.

If you look at model training jobs a lot of the work at this point is creating RL gyms (normal programming work), but most people still think the work is all neural architecture research. Doing the former is fine but won't teach you much about how to build LLMs, whatever that means now. Doing the latter is a very hard market to get into: not many jobs and requirements are often like, "you must have published at one of the following conferences". Prior experience is assumed. Most of them seem to treat Google as ML university and source of new recruits. It's understandable given the cost of training runs.

Re: I were 17, I'd learn how to build LLMs from scratch

#164

Post by mike_hearn »

embedding-shape wrote: post #126
ballooney wrote: post #122

Earlier quoted context omitted.

This only proves the original point which is that there is not much demand for actual machine learning expertise because that is only carried out in a small number of places and what demands there is is for the more basic software carpentry like infrastructure and operations rather than the actual technology and Engineering side of things

What parent says about "there are very few available positions" for "engineers that can do real LLM machine-learning" is fair, yeah, I'd agree with this. I don't think the "incredibly small minority of companies in the world do any real training or optimisation" part is necessarily as true, as some parts of the work I do get is about helping them optimize training and infrastructure around training. Mind you, none of…

I think Paul lives in Britain these days.

Re: I were 17, I'd learn how to build LLMs from scratch

#165

Post by keiferski »

epolanski wrote: post #160
borski wrote: post #154

Earlier quoted context omitted.

If you’re talking purely financial benefit: yes, it is still good advice, as most white collar jobs still require a degree. If you’re talking about a place to mature, around others who are at a similar phase of life, also yes. It is where most people meet their cofounders, for example, even if they don’t found anything until much later.

But it is not true financially, it really depends. Plenty of blue collar workers make more than white collar ones, and have a huge debt free head start in life. A plumber or electrician will make significantly more than the average bank employee or translator or nurse or teacher. And they will also have an easier time starting a business as many trade workers are self employed and make much more than hired ones.

Going to college is worth it for basically everyone other than those that are choosing between a mediocre college / mediocre degree and a blue collar route. AND if college is going to put you into debt.

Financially, intellectually, socially, it’s a good idea - college is a formative period of life in American culture. That is more and more true the better the college gets. At the upper tier (Ivy League, etc.) you don’t really have to pay anything if your family isn’t already wealthy, and the connections and degree you’ll make more than pay for themselves.

Re: I were 17, I'd learn how to build LLMs from scratch

#166

Post by latexr »

embedding-shape wrote: post #141
latexr wrote: post #127

Earlier quoted context omitted.

> I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hard disagree. This submission is still being highly upvoted, while another recent post[1] on the harms caused by Graham’s fellows[2], with a fairly…

> Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote [...] It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that. I guess it depends on what submission you look at, previous comment of mine solely based on memory. Now I went to https://news.ycombinator.com/from?site=twitter.com/paulg , clicked "More" a bunch of times…

> Just one example

That post has barely any points and comments. It’s not a good indicator of general sentiment, it’s just an indicator of people who were on HN at that time.

> Maybe you need to enable "Show Dead" or something?

I have it enabled.

> Pgs articles on HN definitely never was free of any critique in the HN comments, just like any article.

Of course. I very explicitly wrote “in general (certainly more than any other website)”. That does not mean “always”, or “never”. HN is not a hive mind, there’s never going to be 100% agreement. The general trend is what’s being discussed, and we both agree that in general the sentiment on Graham used to be higher. We’re just disagreeing (we may be able to find ourselves agreeing through tough thorough thought, though[1]) on where exactly it is now and how to interpret it.

[1]: Sorry, can’t believe there was a real organic opportunity to use that sentence, had to take it.

Re: I were 17, I'd learn how to build LLMs from scratch

#168

Post by Ampersander »

wy35 wrote: post #132
loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

What would be better advice for a 17 year old?

Stay in school, get good grades.

Re: I were 17, I'd learn how to build LLMs from scratch

#170

Post by qsera »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

>Would you advise a 17 y.o. to learn how transistors work..

Hell yeah. Transistors are pretty awesome.

Post reply on HN