Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

211–220 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#211

Post by user43928 »

reacharavindh wrote: post #103
oersted wrote: post #19

Earlier quoted context omitted.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes. On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources envir…

Well, that's not how it works. You don't just put some old laptops into a rack.

Maybe with a decent consumer GPU like a 4090, you could do experiments like distilling and fine tuning a small image model for edge deployment for specific tasks.

Even there, many use cases might require renting compute for $10/hour and investing a few hundred.

A LLM from scratch? Forget it. You can do theoretical experiments, but not build anything remotely useful with that kind of budget.

If you're talented enough to come up with revolutionary methods, maybe an university or AI lab would be the place to be.

Re: I were 17, I'd learn how to build LLMs from scratch

#212

Post by deepllm »

As someone who's at a similar age and was interested in learning how to do this, there just aren't enough resources to do so. Most LLM research is in the form of academic papers, and there isn't any 'popular' way to learn these things, and besides that all said research assumes you have a B200 cluster ready to go. If you have weaker hardware (say an 8GB nVidia GPU, which is what I have) you're going to be limited to fine tuning small models or torturing yourself working the GPU for days per iteration trying to run things like https://github.com/karpathy/nanochat, which is hardly an educational experience. Renting cloud GPUs is expensive, and at this age the most I could muster up for experimentation is probably $100 or so, which only gets me 25 or so hours on a B200 which just isn't enough. So why would I bother myself with this if I'm already at a disadvantage because of not having access to the right hardware and when surely there are better ways to spend my time? I concluded the only way to learn and be competitive is by finding work at an AI lab somehow (not happening at 17), or studying ML at the right university.

Putting this in contrast with programming, I learned coding when I was 8, and it was incredibly stimulating to learn because you can quickly iterate and there were thousands of books and YouTube tutorials that dumb everything down and teach you fundamentals. All you needed was a $300 computer, and you can learn nearly anything you want, without being gatekept from this or that because you don't have enough vRAM / an sm_100 GPU.

Re: I were 17, I'd learn how to build LLMs from scratch

#213

Post by afro88 »

bmacho wrote: post #162
yobbo wrote: post #140

Earlier quoted context omitted.

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age. It is viable as a toy project, but there are vanishingly few career opportunities.

It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.

I mean... that's actually amazing advice. Not because they would grow up to create browser startups. But because they would grow up to create web startups that succeed because of very fundamental of how the web is rendered.

Which is paulg's point really

Re: I were 17, I'd learn how to build LLMs from scratch

#217

Post by larodi »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

Right - like advising people to learn nuclear power back in 1992. Perhaps a good idea, but super specific already. My gut is ML/AI is even more complex in 2026… I can’t even remember all the abbreviations and the new ones emerging. And every sub component, such as attention or embeddings, are actually a discipline of its own already.

Re: I were 17, I'd learn how to build LLMs from scratch

#218

Post by kenjackson »

shep101 wrote: post #191
kenjackson wrote: post #153

Earlier quoted context omitted.

Except knowing how LLMs work don’t actually provide much understanding for using them. People don’t use LLMs the way we’ve built on most other tons or platforms. It’s more learning Unity hoping to be a better gamer.

this is just wrong idk why u came to that like obviously it does make u better at using them…

How so? I’ve been working on DNNs for over a decade. Not sure it helps me in any almost non-trivial way when it comes to using them.

Re: I were 17, I'd learn how to build LLMs from scratch

#219

Post by graemep »

mike_hearn wrote: post #184
amelius wrote: post #174

Earlier quoted context omitted.

Well, there's a lot more to learn from the former than the latter ...

Really? The latter was immediately useful to lots of people which is motivating, and it had a nice smooth learning curve (html -> js -> php -> databases -> apps -> backend). Learning HTML is the first step to learning how to make full blown apps. Making a browser at 17 is like trying to climb Everest as your first hike. The expected outcome is burnout and demotivating failure. At best you'll learn some C++ or Rust. 1…

Historically people lived very different lives, required different skills, were poorer, had different opportunities etc.

A lot of people will make better decisions with a few yeas more maturity, and spending a few years developing themselves.

University will help a lot of people, and for some it will help.

There is a lot more to life than making a living.

Re: I were 17, I'd learn how to build LLMs from scratch

#220

Post by sho_hn »

mike_hearn wrote: post #184
amelius wrote: post #174

Earlier quoted context omitted.

Well, there's a lot more to learn from the former than the latter ...

Really? The latter was immediately useful to lots of people which is motivating, and it had a nice smooth learning curve (html -> js -> php -> databases -> apps -> backend). Learning HTML is the first step to learning how to make full blown apps. Making a browser at 17 is like trying to climb Everest as your first hike. The expected outcome is burnout and demotivating failure. At best you'll learn some C++ or Rust. 1…

Anecdotally, I think it's great advice.

I contributed to a browser engine around that age (KHTML, which later became WebKit and Blink), and while I don't work in browsers right now, much of that knowledge, mindset and of course the professional network have done much to shape my life. And a fairly successful career, for that matter.

Post reply on HN