Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

711–720 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#711

Post by zem »

jmaw wrote: post #709
zem wrote: post #638

Earlier quoted context omitted.

"learn to code because it will get you a good career" perhaps, but "learn to code at at least a high school level, because it will give you a very good mental model for how computers do things" is still very useful.

Or maybe generalized to "get good at problem solving." I've found building software systems has a lot of parallels to working on physical things such as carpentry/DIY and working on small engines. The concepts of breaking down a problem into a bunch of smaller problems, and identifying what can be independently tested/built remain the same.

the problem solving aspect is great too, but I mean specifically "this is what it means for a computer to know something, and this is what it means for it to do something" expressed in terms of data stored in memory, and algorithms that read to and write from that memory, and "this is how computers think" in terms of control flow and if tests, is a form of literacy that everyone should at least be exposed to.

Re: I were 17, I'd learn how to build LLMs from scratch

#712

Post by enos_feedler »

chasd00 wrote: post #435
enos_feedler wrote: post #293

Earlier quoted context omitted.

For this example, if you built a browser in 2000 it would put you in a great position to launch android in 2008. I think this is pg’s point. You spend all this time learning the interesting bits and when an inflection comes along that makes a new product or service possible you could be the one to likely launch it

That’s like saying if you can dunk in 8th grade you could be the next LeBron James. A LOT more things have to fall perfectly in place at the perfect time for that end result to materialize.

Building browsers is not learning to dunk, its more like mastering some game. Your example basically sucks. Hah

Re: I were 17, I'd learn how to build LLMs from scratch

#713

Post by fancyfredbot »

firemelt wrote: post #487
fancyfredbot wrote: post #85

I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka. I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier conc…

so can u explain to me how they are works?

Yes

Re: I were 17, I'd learn how to build LLMs from scratch

#714

Post by d0mine »

loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

The point of courses like "Nand to Tetris" is not to become CPU designer. It is not useless just because you can't build a chip factory.

Watching Karpathy's "Zero to Hero" won't make you AI specialist. It will improve your understanding of the basics.

Re: I were 17, I'd learn how to build LLMs from scratch

#715

Post by rchaud »

potatoking wrote: post #590
BodyCulture wrote: post #588

Earlier quoted context omitted.

The next class of problems to solve for future generations is how to survive on a hot planet and how to recover from the damage done by reckless lunatic profiteering assholes.

Maybe people want to take it as cynicism but I think you are factually correct that personal cooling, infectious disease, atypical weather and fire breaks are all more realistic topics for people too young to already have patents, connections or their own oil well.

35 years ago, the enormously popular animated series Captain Planet taught kids about protecting the ozone layer, using biodegradable materials and renewable energy.

Absolutely none of that mattered in the long run because capital markets think environmental destruction is a small price to pay as long as some people get rich.

What kids learn about personal cooling tech and what not won't matter when the supply of the raw materials required to make them is reserved for some VC-funded trillion-dollar 'startup' trying to synthesize an undetectable chemical weapon to get past the bureaucratic red tape of international treaties.

Re: I were 17, I'd learn how to build LLMs from scratch

#716

Post by nomel »

pjerem wrote: post #208
wy35 wrote: post #132

Earlier quoted context omitted.

What would be better advice for a 17 year old?

Throw the computer and the smartphone out of the window. Or more reasonably, the same old thing : use Linux, hack a little, why not learn programming basics. But learn to own your technology, fight against centralization of technology. The same old RMS story. IDK where the tech industry is going, if there will be jobs anymore or not, but what I'm sure (and what have been the case for the last 10-15 years anyway) is t…

As someone who very much enjoys self learning, I can't comprehend what you wrote. It doesn't make any sense. In fact, I don't believe you've played in the same sandbox at all.

> why not learn programming basics

Do you think implementing an LLM from scratch would not require programming!?

A generic project based goal like this is the best approach to self learning, because you fight your way to it, picking up what you need to know on the way. And, once it's all working, and you get your first meaningful sign of success (in this case, some tokens out), after your long path, you fucking celebrate!

Re: I were 17, I'd learn how to build LLMs from scratch

#717

Post by [deleted] »

embedding-shape wrote: post #32
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> I am kind of amazed how negative the comments are here, especially on HN. I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hazarding a guess, I think the moment Altman became the CEO and later durin…

[deleted]

Re: I were 17, I'd learn how to build LLMs from scratch

#718

Post by aryehof »

chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> I am kind of amazed how negative the comments are here, especially on HN.

Sadly, I attribute this to a decline in having “good will” towards others, and an increase in cynicism, prejudice, and meanness.

Re: I were 17, I'd learn how to build LLMs from scratch

#719

Post by tomhow »

delis-thumbs-7e wrote: post #663
tomhow wrote: post #611

Earlier quoted context omitted.

Sure, but isn't this a bit of a sneer? “I feel slightly bad for people who didn’t” has an air of superiority about it, and seems like a veiled swipe at those who lacked whatever qualities your 17-year-old self had. The guidelines have several different ways of asking us to avoid commenting like this. As to the substance of your comment, there's nothing stopping a 17-year-old from writing poetry, learning guitar, and…

> there's nothing stopping a 17-year-old from writing poetry, learning guitar, and also learning about how LLMs work, if they're so inspired. I assume the people at HN are intelligent enough to understand this without me having to explain the most obvious banalities to them. The point is why you would do anything at 17: > if they're so inspired You said it yourself. The Twitter post, to me, reads like career advice o…

It’s not career advice, it’s life advice: don’t limit what you try and learn or accomplish to what your peers are doing or what your teachers tell you is sufficient. Follow your curiosity as far as it goes. Don’t stop just because it seems like it should be beyond the ability of “someone like you”. You might be more capable than you thought.

I was incredibly lucky to have had a father who worked that out for himself then impressed it on me. The people I feel sad for are the people who don’t have people around to make them feel it’s ok to try things that seem impossibly advanced for your age or social status.

Re: I were 17, I'd learn how to build LLMs from scratch

#720

Post by floro »

throwaway1183 wrote: post #664
floro wrote: post #602

Earlier quoted context omitted.

> shouldn’t have robbed others then. a richer world for you comes at poorer world for someone. we are playing infinite game in finite world. Young people are not responsible for the crimes of their ancestors centuries ago.

say that to visa laws that were designed by your ancestors that carry around the world. Say the same to currency arbitrage, freedom of mobility and everything that you enjoy for granted in protected walls. The reason remote worker can out compete you is because of this currency arbitrage you designed. Your pennies goes a long way in this so called third world. Your one day dinner at nice place is a months salary for…

Once again *I* didn't design and of it. Leftists don't like hearing that.

The fix would be obviously to control and limit foreign workers by regulating companies.

Post reply on HN