Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

41–50 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#41

Post by IshKebab »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

Seek advice from people who have had a normal level of success. Not a one-in-a-million level.

Re: I were 17, I'd learn how to build LLMs from scratch

#42

Post by weinzierl »

What are the best resources to learn how to build LLMs from scratch for 17 year olds?

I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:

If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.

Re: I were 17, I'd learn how to build LLMs from scratch

#43

Post by felixrieseberg »

I'll use this post as a shameless opportunity to tell more people about a little side project, I made:

http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.

Re: I were 17, I'd learn how to build LLMs from scratch

#44

Post by embedding-shape »

InsideOutSanta wrote: post #40
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> seek advice from unsuccessful people Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.

Intuitively, that'd make sense if that unsuccessful person eventually found success, otherwise who knows if they actually picked up what made them unsuccessful in the first place?

Re: I were 17, I'd learn how to build LLMs from scratch

#45

Post by trentor »

embedding-shape wrote: post #32
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> I am kind of amazed how negative the comments are here, especially on HN. I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hazarding a guess, I think the moment Altman became the CEO and later durin…

Because at some point in life everyone gets tired of fairytales. He started mending the anecdotes to his content which always rubs people the wrong way.

Re: I were 17, I'd learn how to build LLMs from scratch

#46

Post by armcat »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you, and then take advice from people that are amazing in that specific field. In mid 2000s in Australia all the "top people" were telling me not to get into a software engineering career because it was dead. It's certainly challenged right now, but it took off during those 15+ years.

Re: I were 17, I'd learn how to build LLMs from scratch

#47

Post by oersted »

spwa4 wrote: post #20
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

That's not true, because everyone, everyone, everyone seems to want to do training. Which results in a 50 person company training, say, a voice model that then fails, because it's just not good enough. In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% bette…

Everyone says they want to do training, because it's sexy and an easy way to justify raising mad funding rounds. Some manage, most don't.

I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if you try to train, you cannot get anything amazing unless you burn millions.

Very few people can afford to play the long game and cross that dessert. And, sure, you will not get far without good engineering, but good engineering is definitely not sufficient and is not the primary bottleneck.

Re: I were 17, I'd learn how to build LLMs from scratch

#48

Post by tayo42 »

I don't think individuals have the resources to build an interesting llm. The l stands for large. You need a dataset too. Llms are only interesting because theyre large

And it's basically a weekend project to put transformers together in a ML library and train it.

The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?

Re: I were 17, I'd learn how to build LLMs from scratch

#49

Post by willtemperley »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues.

1. Both training and optimisation will get significantly cheaper and easier quickly.

2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029.

3. The big AI firms will become part of the surveillance capitalism network, if they're not already.

So I think for self-protection a lot of companies will be looking near to medium term AI independence.

Re: I were 17, I'd learn how to build LLMs from scratch

#50

Post by vintermann »

Cheyana wrote: post #3

He bases this decision on all of the experience he has amassed, as a 61 year old man in the tech industry. An actual 17 year old, with 17 years of experience, would not think like this, nor should they.

Yes. And they almost certainly have a better understanding of their own situation that him. This is not a dig at Paul Graham, the closer anyone is in age, the better they understand what they have to deal with. I'm roughly in the middle between Paul G and the 17 year old, and even though I'm really quite fascinated with zoomer culture and probably come more in touch with it than most (due to relatives in the age range etc.) I realize I have very little idea what it's like to grow up in the world they grow up in.
Post reply on HN