Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

371–380 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#371

Post by willis936 »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

I would encourage young people to be born rich. It's the best time in 100 years to be advantaged. Why waste your potential by having your labor stolen?

Re: I were 17, I'd learn how to build LLMs from scratch

#372

Post by weatherlite »

spolitry wrote: post #368
weatherlite wrote: post #262

Earlier quoted context omitted.

A lot of startup companies are not training frontier models but help solve and optimize pain points of LLMs: cyber security, token usage, harnesses etc. These jobs don't require a PHD in machine learning but it does help if you understand LLMs at a deeper level.

A lot of startups succeed by lying to themselves about the quality of their solutions and focusing on selling a marketing story.

True, I'm tempted to put Antrhopic under the same category of lying startups so it enforces my point that understanding LLMs deeply could be useful

Re: I were 17, I'd learn how to build LLMs from scratch

#373

Post by beemboy »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

Yes and no. I believe the point he is making is simply that there is no substitute for fundamentals and first-principles thinking.

We had scores of students study how microprocessors work and compilers work over decades, yet we have 3 or 4 major processor companies and a handful of programming languages. Yet, what they learned was probably crucial in their development as engineers.

We are also so early right now that even 2-3 years from now who knows how many LLMs and model firms survive (esp. given the "snake eating its tail" venture/investor funding situation)

Re: I were 17, I'd learn how to build LLMs from scratch

#374

Post by Tomlmmrs »

This essentially comes down to choosing the search for substance (here in the form of technical depth) over short term gratification and quick wins in life.

I think this kind of mindset should be taught way more in school so that people really appreciate learning a subject deeply.

Re: I were 17, I'd learn how to build LLMs from scratch

#375

Post by topham »

ignoramous wrote: post #299
nextaccountic wrote: post #273

Earlier quoted context omitted.

If someone writes a browser in their teens, they will probably learn more about the web than if they were just writing web pages

A few did. [Blake Ross] worked as an intern at Netscape at the age of 16 ... Ross became disenchanted with the browser he was working on and the direction given to it by America Online, which had recently purchased Netscape. Ross and Hyatt envisioned a smaller, easy-to-use browser that could have mass appeal, and Firefox was born from that idea ... in 2003 all of Mozilla's resources were devoted to the Firefox and Th…

Sure, you had to pick the browser I despise worse than Netscape 4.

Re: I were 17, I'd learn how to build LLMs from scratch

#376

Post by spolitry »

khalic wrote: post #201
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

lol unnecessary and inefficient… I just finished fine tuning Gemma e2b for local code completion on my local machine. This comment just reinforces what the post actually means. We need people that are LLM natives, computing solves itself with time and with scale adjustments

That’s great but the industry has moved on to code generation and editing, not code completion.

Re: I were 17, I'd learn how to build LLMs from scratch

#377

Post by oldsklgdfth »

I appreciate the sentiment and I'm pretty curious how I could train an LLM, even are really basic one from 5yrs ago without nuts hardware.

That said, I a good starting point for a 17yo is reading about perceptrons[0], then the basics of neural networks[1] (ex. 3-layer perceptron) then writing a program to train a 3-layer perceptron and classifying the MNIST dataset[2] - a dataset of characters.

This can anywhere between a day and a week and you will demystify the basics of neural networks and work your way forward with more advanced contemporary concepts.

Fun fact: any multi-layer perceptron neural net can be reduced to a 3-layer perceptron network.

[0] https://en.wikipedia.org/wiki/Perceptron [1] http://geeksforgeeks.org/deep-learning/neural-networks-a-beg... [2] https://www.kaggle.com/datasets/hojjatk/mnist-dataset/data

Re: I were 17, I'd learn how to build LLMs from scratch

#378

Post by smcg »

paulcole wrote: post #278
yobbo wrote: post #140

Earlier quoted context omitted.

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age. It is viable as a toy project, but there are vanishingly few career opportunities.

But he’s not saying that it’s a career opportunity. It seems to me like he’s saying that doing this thing would be 1) fun and 2) a great way to become employable in the future. I don’t believe he’s saying that this project would be some kind of job training exercise.

sounds like a great way to not establish social skills

Re: I were 17, I'd learn how to build LLMs from scratch

#379

Post by keybored »

eleventhborn wrote: post #326
keybored wrote: post #92

YC Combinator guy says that with a time machine he would learn to build the currently trillions-valued or whatever technology. Okay.

No, that isn't what he said. He said, if he is 17 _now_, with no family, no commitments, no pressure to start a career, he'd invest his time to learn to build LLMs from scratch instead of trying to start a company.

Okay sure it’s not a “time machine” in that sense. He wouldn’t go back forty years or whatever. And he’s still saying the same thing that I was alluding to.
Post reply on HN