Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

471–480 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#471

Post by Atouraya »

Cheyana wrote: post #3

He bases this decision on all of the experience he has amassed, as a 61 year old man in the tech industry. An actual 17 year old, with 17 years of experience, would not think like this, nor should they.

Naturally 17 year olds don't think long-term like this which is why PG's advice is so useful. It gives them a pathway to follow that they likely wouldn't have reasoned otherwise.

I told my much younger brother when he was 12 what programming was and it'd be a great career. He looked into it and within months was writing CLI games. Eventually releasing his own unity 3d game on steam as a teen.

Eventually he got into CS and did really well because none of it was scary and new. He parlayed that into role at Meta out of university.

My point being, 17 year olds have time to learn new skills and guidance can go a long way.

Re: I were 17, I'd learn how to build LLMs from scratch

#472

Post by tejohnso »

fcatalan wrote: post #276
yoz-y wrote: post #253

Earlier quoted context omitted.

17 is a weird age but ymmv. I left home at 16 alone to study abroad. I had tons of free time due to dorm curfews and such. Unfortunately, I was too poor to have a computer and the computers we had access to were completely locked up. (Naturally we waltzed past the locks to play some games but it was also under surveillance) Paradoxically I coded way more between ages 12-14, I regret my wasted late teens.

Me too: ages 8-15: lots of great 8-bit computer fun. ages 15-20: girls, booze, motorcycles. 20 onwards: get a PC, back to computers, realize how much I've been missing. To be honest, judging by my own kids and their friends, late teens seem to generally be an era of hard to avoid stupidity.

If your focus on girls, booze, and motorcycles ends at 20 years old that's a pretty good sign.

Re: I were 17, I'd learn how to build LLMs from scratch

#473

Post by nbardy »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

This is a wildly incorrect and myopic view on the world.

Finetuning model is cheap and incredibly useful for deployment. You don't need to pre-train a frontier llm from scratch to make useful models.

There is tons of domains where you and fine-tune llms and deploy them for value in companies and for your own entrepreneurship ambitions. I have made this a big part of my career for the last few years and now I'm working on finetuning models for starting my own companies.

Re: I were 17, I'd learn how to build LLMs from scratch

#474

Post by dylanowen »

paulryanrogers wrote: post #332
nextaccountic wrote: post #273

Earlier quoted context omitted.

If someone writes a browser in their teens, they will probably learn more about the web than if they were just writing web pages

They'll learn so much more that won't transfer to as many job opportunities. For ex, say more about C++ and less about cutting edge CSS (because modern browser tech is an ocean). I suppose they might luck into other adjacent or unrelated roles with the same skills.

I have a hard time imagining anything where writing a browser wouldn't be an excellent transferable foundation. Even your example: writing css will never teach you as much as writing a css engine.

Re: I were 17, I'd learn how to build LLMs from scratch

#475

Post by QuercusMax »

ryandrake wrote: post #455
bigfishrunning wrote: post #420

Earlier quoted context omitted.

Are you asserting that C++ is not a marketable skill? If anyone reading this has this un-marketable skill, Carnegie robotics in Pittsburgh is hiring carnegie-robotics.breezy.hr/p/2d85f5321cc7-software-engineer

I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you…

But building a web browser isn't a niche skill - it requires a whole bunch of them. You need to be a generalist to write a web browser.

Re: I were 17, I'd learn how to build LLMs from scratch

#476

Post by andy_ppp »

megaloblasto wrote: post #449
andy_ppp wrote: post #419

Earlier quoted context omitted.

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I disagree. It's absolutely possible to develop an intuition about extremely complex mathematical ideas, including llms or high dimensional systems. Learning to build an llm is a great way to start building that intuition.

What provable conclusions have you intuited around how LLMs work? Give me some examples to prove your point? I'm absolutely happy to change my view with enough data.

Re: I were 17, I'd learn how to build LLMs from scratch

#477

Post by bigfishrunning »

ryandrake wrote: post #455
bigfishrunning wrote: post #420

Earlier quoted context omitted.

Are you asserting that C++ is not a marketable skill? If anyone reading this has this un-marketable skill, Carnegie robotics in Pittsburgh is hiring carnegie-robotics.breezy.hr/p/2d85f5321cc7-software-engineer

I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you…

I would argue that in a post-LLM world, becoming as specialized as possible is the only way to survive. If you have the serious systems programming skills required to write graphics drivers, your talent would be wasted writing CRUD apps anyway. I hope you eventually found/will find something more appropriate for your skillset!

Re: I were 17, I'd learn how to build LLMs from scratch

#478

Post by PoRidg3 »

nbardy wrote: post #473
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

This is a wildly incorrect and myopic view on the world. Finetuning model is cheap and incredibly useful for deployment. You don't need to pre-train a frontier llm from scratch to make useful models. There is tons of domains where you and fine-tune llms and deploy them for value in companies and for your own entrepreneurship ambitions. I have made this a big part of my career for the last few years and now I'm workin…

I find the fine tune approach more interesting than straight to RAG and MCP.

End of the day they're all customized data stores and protocols to interact with them. May as well stick to a uniform toolkit with fine-tunes.

Not that other tools aren't useful. But reaching straight for a bunch of infrastructure reliant services is like jumping in with k8s when you're still at a stage where basic mocks in code are sufficient.

I won't roll my own encryption or UI lib but want to stay focused on the incompleteness of the project I have to ship not all the buttons and knobs of some dependency or framework. Same old manage context switch problem.

Re: I were 17, I'd learn how to build LLMs from scratch

#479

Post by MaxHoppersGhost »

Uptrenda wrote: post #309

I genuinely don't think telling young people to do anything tech-related is good career advice. We don't even know if entry level roles will ever come back. The situation couldn't be worse for these roles. PG thinks that some random teenager will build a startup and get rich from it, or some shit. Like get real, man. Completely out of touch, tech bro who hasn't worked a real tech job for the last like 20 years... Rem…

These guys (VCs) entire livelihoods rely on tens of thousands of young people throwing away their early years of lives trying to get super rich. Only .01% even see moderate success but PG and co don't care and continue to pump the dream and urge folks to waste their lives to try and make them rich.

Re: I were 17, I'd learn how to build LLMs from scratch

#480

Post by bentt »

felixrieseberg wrote: post #43

I'll use this post as a shameless opportunity to tell more people about a little side project, I made: http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.

Beautiful work! What prompted you to build this? Pun not intended.
Post reply on HN