Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

91–100 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#91

Post by jillesvangurp »

weinzierl wrote: post #42

What are the best resources to learn how to build LLMs from scratch for 17 year olds? I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing: If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.

Andrej Karpathy has a great Youtube series on how to build LLMs from scratch. Perfect for somebody who just learned a lot of high school math. Would start there and get busy with some handson python coding.

Re: I were 17, I'd learn how to build LLMs from scratch

#93

Post by BoredomIsFun »

embedding-shape wrote: post #76
BoredomIsFun wrote: post #64

Earlier quoted context omitted.

I just voiced my opinion. I just think buiding an LLM from the scratch for 17 y.o. is pointless exercise, advising a teenager to do so is borderline irresponsible, and frankly PG is simply virtue signalling here, as LLMs are still trendy, esp. in his circles. There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IM…

> I just voiced my opinion. Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X. > youngsters who'd be able to extract tremensous value from such an excercise But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding…

> Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).

This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.

Re: I were 17, I'd learn how to build LLMs from scratch

#94

Post by livinglist »

muragekibicho wrote: post #66
livinglist wrote: post #55

When I was 17 I was building Windows Phone apps, bad decision on my part.

Incredible counterexample, but oddly relatable. I'd probably have achieved techbro 'post-economic' status earlier if I focused on Android dev instead of the shiny (and new at that time) Xamarin for Windows phones.

God I almost invested in Xamarin after Windows Phone got aborted, I did spend a lil time on UWP, but thank god Flutter came out not so long after that. After all these years I learned to stay away from Microsoft tech stack.

Re: I were 17, I'd learn how to build LLMs from scratch

#95

Post by otherme123 »

IshKebab wrote: post #41
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

Seek advice from people who have had a normal level of success. Not a one-in-a-million level.

Veritasium has a good piece for this kind of bias:

https://www.youtube.com/watch?v=3LopI4YeC4I

An advantage that is not "advisable", like being born in january, in a rich country, in an above average family, or just having luck, might have more influence on the outcome than any conscious action. It is almost sure that one-in-a-million level people only edge over the other 999,999 they competed with is just "have more luck".

Re: I were 17, I'd learn how to build LLMs from scratch

#96

Post by jbs789 »

danpalmer wrote: post #27
oersted wrote: post #19

Earlier quoted context omitted.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

Agreed. It's hard to learn unless you have access to quite high end hardware, and even paying by the hour is expensive. There's a low ceiling on what you can learn without doing training runs. You can however learn everything you need to know to get on the career ladder as a software engineer on a regular home PC.

While the topic here is narrow, the concept is broader.

Do you take the first step or rule it out because you don’t yet see the complete picture.

As a teenager I never hesitated to try things out. As a young adult I wanted the whole picture. Now I’m back to playing / trying things out. I kinda wish I’d not given it up. PG being a bit older and reminiscing - I bet he’s in that bucket too, whereas someone trying to establish themselves professionally probably (aka me early 20s) wants to see the path.

Re: I were 17, I'd learn how to build LLMs from scratch

#97

Post by keybored »

chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> Do people think learning new tech is somehow wasted effort?

No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.

But why are people cynical? they ask.

Re: I were 17, I'd learn how to build LLMs from scratch

#98

Post by jasonkester »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> Personally I do both

That’s what you get from listening to “successful people”. You get to learn about all the things they tried that failed, then the things that did work on that 24th try, which was successful.

The “survivorship bias” people always seem to assume that the “survivor” lucked into his fortune on his first try ever, so he can’t have learned anything, so we don’t have to listen to him. But that’s seldom the case.

I’ve written about this before:

https://expatsoftware.com/Articles/survivorship-bias.html

Re: I were 17, I'd learn how to build LLMs from scratch

#99

Post by Zylokloto »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

We finetune LLMs. Small ones like Gemma 4 for semantic tasks.

There are plenty of areas were we need people to do this for insurances, banks etc.

AI/ML exists on many levels.

Re: I were 17, I'd learn how to build LLMs from scratch

#100

Post by trentor »

embedding-shape wrote: post #81
trentor wrote: post #45

Earlier quoted context omitted.

Because at some point in life everyone gets tired of fairytales. He started mending the anecdotes to his content which always rubs people the wrong way.

So, this comment of yours obviously isn't in the "knee-jerk reaction" category of comments, I suppose? What exactly from the linked tweet(s) are fairytales here? There is hardly any text at all, so strikes me as a comment about previous pg content, but then this would be one of those comments I talk about? Very confusing.

Your hand waving doesn't make it knee-jerk. It's just what happened to his writing since COVID. He goes for more of a shock and awe style and not everybody likes it. He's been writing for over 20 years now, hasn't he? His style has clearly changed, and an changing style attracts a different audience so it's no surprise his original readers might not connect with his newer work...
Post reply on HN