Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

681–690 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#681

Post by chickenuggies69 »

chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

This is essentially most discussion here. I close the HN tab so often after seeing the people that just came to disagree with someone on the internet. There are too few discussions here that come from a place of "That's a great perspective and I can see it being useful when applied in X, just trying to solicit a discussion here, I wonder how this could work or break down in condition Y?"

Re: I were 17, I'd learn how to build LLMs from scratch

#682

Post by cbondurant »

Had to reread this a few times to properly parse what the hypothetical was.

My first read was "if you could redo your life from age 17 what would you do". To which "figure out how to make an LLM" would be an insane answer.

I think his advice here is maybe... a year or two too late to be good advice. I can't foresee the job market for machine learning experts being better than it is now in 5-6 years time when said hypothetical 17 year old would be most ready to start career hunting. Either the bubble is gonna pop and the market will be flooded with laid off AI talent.

Even if I'm wrong about there being a bubble at all, I still think that in 5 years time the tech will just have matured to the point of diminishing returns on refining existing architectures. Plateauing until some PHD comes up with something as ground breaking as attention.

Re: I were 17, I'd learn how to build LLMs from scratch

#683

Post by wao0uuno »

conartist6 wrote: post #658
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

No, but there's a WHOLE ENTIRE WORLD of tech that isn't LLMs. LLMs aren't the future or anything. Sure they'll keep existing. Right now though all the opportunities exist for people looking literally anywhere else

So you're saying that LLMs are a dead end technology? Do you have anything to back up that claim?

Re: I were 17, I'd learn how to build LLMs from scratch

#684

Post by TheOtherHobbes »

BodyCulture wrote: post #588
mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

The next class of problems to solve for future generations is how to survive on a hot planet and how to recover from the damage done by reckless lunatic profiteering assholes.

The next wave is already in view and it's going to surprise and horrify the survivors in equal amounts.

Some people will be delighted. Most are going to have a very hard time adjusting.

YC and adjacent isn't it.

Re: I were 17, I'd learn how to build LLMs from scratch

#686

Post by handoflixue »

LarsDu88 wrote: post #653
lern_too_spel wrote: post #604

Earlier quoted context omitted.

Yann has been consistently wrong about the limits of LLMs.

He has, but I think he's right in the sense that there will be better models. It would be premature to think that the current frontier model architectures will last into the future. It would be like making DVD engineering a standard part of high school curricula

> It would be like making DVD engineering a standard part of high school curricula

I fail to see how PG suggesting that a 17 year old, today, should study something... is in any way a parallel to formalizing it as part of the curriculum.

I would, in fact, expect PG to recommend something completely different in 10 years.

The whole point of hacking is to understand the world around you, today

Re: I were 17, I'd learn how to build LLMs from scratch

#687

Post by canyon289 »

rdedev wrote: post #235
kalms wrote: post #87

Earlier quoted context omitted.

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful. The above is, after all, the wh…

How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasona…

A lot of different ways. Every guide on this guidebook runs on a ~$500 gpu or a couple of dollars on a cloud instance with a bigger GPU

https://ravinkumar.com/GenAiGuidebook/book_intro.html

This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware.

If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi.

https://developers.googleblog.com/en/introducing-gemma-3-270...

The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here.

And FWIW I'm a staff researcher at Deepmind (opinions here are my own) so I want to specifically encourage all people out there, you can learn a lot about how these LLMs work at home, for (mostly free), using resources like colabs or spot pricing on accelerator providers. There's many great resources out there and I encourage anyone willing to learn to go for it!

Re: I were 17, I'd learn how to build LLMs from scratch

#688

Post by danielmarkbruce »

auggierose wrote: post #678
danielmarkbruce wrote: post #642

Earlier quoted context omitted.

This is a silly point. Just because the size of these things is too large for our tiny brains, it doesn't mean we have no idea why it does what it does. If you run a physics simulation of a weather system, you have a situation that is unpredictable for a human - but it's not fair to say "we have no idea why the outputs are what they are!!"

That is not a silly point at all. You confuse understanding the mechanics of it with having a theory of why it works. For the physics simulation, physics provides us with the theories which give us the equations underlying the physics simulation. For neural networks, why is next word prediction giving us AI that can do math? We don't really know!

There is no "theory" of tomorrow's weather. We understand the math of every single individual equation of an LLM (for example), just like we understand every single equation in the physics simulation. It's the entirety of the system that we don't understand (and hence can't predict) in our heads.

From a biology perspective - we have a good understanding of the carbon atom and how forces influence it. We really don't know much about a cell.

Re: I were 17, I'd learn how to build LLMs from scratch

#689

Post by dominotw »

megaloblasto wrote: post #626
dominotw wrote: post #617

Earlier quoted context omitted.

i read all that i still dont see what your intuition is about how llm work. visualizign 100 dim matrix will not tell you how llm work. so what you even talking about.

Visualizing large dimensional matrices is a way of starting to develop intuition about llms. If you can't visualize high dimensional spaces, and you haven't studied llms, you're not going to have any intuition about how they work. Depending on your knowledge of math, I recommend starting with linear algebra, building an understanding of the equations and try to visualize more and more complex systems, then study llms…

you are just dancing around the question without addressing it directly.

> Visualizing large dimensional matrices is a way of starting to develop intuition about llms.

one last time before i disengage. how do you know this and what intuitions have you personally developed.

Re: I were 17, I'd learn how to build LLMs from scratch

#690

Post by [deleted] »

s1artibartfast wrote: post #624
tech_ken wrote: post #550

Earlier quoted context omitted.

Developing an intuition about high dimensional systems is pretty different from understanding the character of some specific point on a 1e9+ dimensional manifold of parameters, in my professional opinion (setting aside all the degrees of freedom that come from the structure of the thing). Sure one can understand generic principles like the curse of dimensionality, but truly groking how an LLM works is basically an op…

Seems like a silly goalpost. As you say, there's a lot of value to unlock with understanding the generic principles, and I would add specific application. Lots of people are building fantastic Tools or pulling down million dollar salaries without groking the precise representation of a single weight.

[deleted]
Post reply on HN