I were 17, I'd learn how to build LLMs from scratch • Hacker News Board

Re: I were 17, I'd learn how to build LLMs from scratch

#431

Post by Aissen » Mon, Aug 24, 2026, 2:13 PM UTC

femto wrote: post #294

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good l…

I was there at the start of the smartphone boom. I built a demo Android device that was capable of telephony/data, 3D rendering, etc. all before Google open-sourced the OS, for a SoC vendor that wasn't in Google's inner circle. Yet the industry was not interested in my junior profile during the subprime crisis.

Re: I were 17, I'd learn how to build LLMs from scratch

#432

Post by sur_lal » Mon, Aug 24, 2026, 2:15 PM UTC

This is the ultimate builder’s mindset. Tinkering with the hardest technical problems—even for frivolous things like games—always yields the highest return on curiosity. Time to go back to the fundamentals.

Re: I were 17, I'd learn how to build LLMs from scratch

#433

Post by stellamariesays » Mon, Aug 24, 2026, 2:15 PM UTC

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

[flagged]

Re: I were 17, I'd learn how to build LLMs from scratch

#434

Post by vibe_that_works » Mon, Aug 24, 2026, 2:16 PM UTC

I really don't understand Paul's reasoning here. Does he predict more scarcity on the model-level? That layer seems to be almost a commodity now + training is damn expensive.

If you are really 17, my advice is to identify use-case for AI (ideally relevant for businesses) that work most of the time and find ways to make them work pretty much every time. AI reliability is the scarcity right now.

Re: I were 17, I'd learn how to build LLMs from scratch

#435

Post by chasd00 » Mon, Aug 24, 2026, 2:17 PM UTC

enos_feedler wrote: post #293

serbuvlad wrote: post #190

Earlier quoted context omitted.

If you learnt to build a browser in 2000 you're probably doing well for yourself lol. Not a lot of demand, but also probably not a lot of supply.

For this example, if you built a browser in 2000 it would put you in a great position to launch android in 2008. I think this is pg’s point. You spend all this time learning the interesting bits and when an inflection comes along that makes a new product or service possible you could be the one to likely launch it

That’s like saying if you can dunk in 8th grade you could be the next LeBron James. A LOT more things have to fall perfectly in place at the perfect time for that end result to materialize.

Re: I were 17, I'd learn how to build LLMs from scratch

#436

Post by gpjt » Mon, Aug 24, 2026, 2:18 PM UTC

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

But you can train a small LLM with a gaming graphics card -- I managed one on a GTX 1660. I don't think pg is suggesting that you try to chase the frontier. It's more like building your own OS in the 80s, or web server in the 90s -- sure, you'll never match the commercial offerings or the big OS projects, but building something from scratch within the limits of the hardware you can afford is amazing educationally.

Re: I were 17, I'd learn how to build LLMs from scratch

#437

Post by mike_hearn » Mon, Aug 24, 2026, 2:21 PM UTC

stymaar wrote: post #414

inciampati wrote: post #356

Earlier quoted context omitted.

There is no reason to believe that generalist models need to be transformer (or transformer hybrids) it's just that people are risk averse and not spending their resources on the many scaled risky bets needed to explore the possibilities. There is more there.

True. I'm a big fan of Cactus's work on the needle family of Simple Attention Network: https://github.com/cactus-compute/needle

That looks like a really nice and well put together library.

But it feels like it sort of backs up my point about there being good models at every size class. Fine tuning Needle looks automated. Yes, you need to know basics like what validation loss means and how to use Python, but otherwise it's all about creating the dataset.

Re: I were 17, I'd learn how to build LLMs from scratch

#438

Post by elorant » Mon, Aug 24, 2026, 2:21 PM UTC

I built from scratch a sparse text embedding model trained on a 13T token corpus. Not the same as an LLM because there’s no transformer in the mix, but still I learned a shit load of things in order to solve all sorts of problems that emerge when you try to access big datasets and daily update tables with billions of rows. But if it wasn’t for a specific use case that I tried to solve I don’t think that whatever knowledge I gained could be utilized in the market. Sparse models are a very small niche and most people I’ve come across with similar knowledge are in academic circles, not business related ones. So even if LLMs are all the rage these days I doubt the demand for people who know how to build them is that high. Someone who knows how to setup an open weight model and expose an API might be more valuable to a company these days.

Re: I were 17, I'd learn how to build LLMs from scratch

#439

Post by gensym » Mon, Aug 24, 2026, 2:22 PM UTC

kingofalldevs wrote: post #352

markb139 wrote: post #75

Earlier quoted context omitted.

As a 17 y.o (way back in the last century) I didn’t need to be advised to learn about transistors. I just had a thirst for the knowledge. I would encourage everyone to learn something about transistors. They are one of mankind’s most useful discoveries.

Hey, I'm 19 CS undergrad but i don't really know what to do but I really wanna build something that can shape the world. Can you tell me more about why you would advice someone to learn about transistors? what are the possible career paths?

Not the original commenter, but in my view as a crusty old software engineer, the reason to learn how transistors is because they're interesting and the key to how all computation is implemented. Once you learn how transistors work, you can learn how logic gates work. And then how CPUs and memory work. Bit by bit, computers stop being magic boxes and start being something built from technologies you understand.

Now, for most software careers (including my own), that knowledge isn't directly practical. BUT, the most important things for a successful, fulfilling career are curiosity and a willingness to dive into subjects _without_ necessarily knowing how (or even if) that knowledge will yield practical applications. That drive is what's going to lead to insights and breakthroughs over your career that wouldn't happen if you just let abstractions be abstractions.

Re: I were 17, I'd learn how to build LLMs from scratch

#440

Post by mondainx » Mon, Aug 24, 2026, 2:23 PM UTC

On first glance, this is good advice and in general I'd give the same for this age group. With age, you'll see more of these scenarios come up and if you have experience and foresight, you can provide direction that may prove fruitful to young people. My son and his friend asked "what should we look into and learn?", about 15 years ago, I told them Python and Java; Python because of versatility and cryptocoins; Java for the long term stability in the job market. I despise Python personally, but I could see its potential and still do; especially for AI. At the end of the day, kids have more time than money and its great experience for them to get their hands dirty and find out what they might be interested in; its a long life.