Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

71–80 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#71

Post by jbstack »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> Moreover I am not sure it is even good advice?

I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.

Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.

Re: I were 17, I'd learn how to build LLMs from scratch

#72

Post by embedding-shape »

armcat wrote: post #46
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you, and then take advice from people that are amazing in that specific field. In mid 2000s in Australia all the "top people" were telling me not to get into a software engineering career because it was dead. It's certainly challenged right now, but it took off during those 15+ years.

> It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you,

Agreed, my previous stated "ignore both and do whatever the fuck I want" approach has worked out very well for me in life, people should probably focus on identifying better what their gut tells them, rather than what randoms on the internet thinks and writes.

Re: I were 17, I'd learn how to build LLMs from scratch

#74

Post by haunter »

If I were 17 again I'd prepare to go for volunteering overseas after high school for 1-2 years (plenty of free options in the EU where you might only need to cover the plane ticket). See the world, you learn a new language, help others and then think about what you want to do.

Re: I were 17, I'd learn how to build LLMs from scratch

#75

Post by markb139 »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

As a 17 y.o (way back in the last century) I didn’t need to be advised to learn about transistors. I just had a thirst for the knowledge. I would encourage everyone to learn something about transistors. They are one of mankind’s most useful discoveries.

Re: I were 17, I'd learn how to build LLMs from scratch

#76

Post by embedding-shape »

BoredomIsFun wrote: post #64
embedding-shape wrote: post #22

Earlier quoted context omitted.

> I do not think it is a proper thing to do for 17 y.o If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this. Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is o…

I just voiced my opinion. I just think buiding an LLM from the scratch for 17 y.o. is pointless exercise, advising a teenager to do so is borderline irresponsible, and frankly PG is simply virtue signalling here, as LLMs are still trendy, esp. in his circles. There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IM…

> I just voiced my opinion.

Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.

> youngsters who'd be able to extract tremensous value from such an excercise

But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).

Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.

Re: I were 17, I'd learn how to build LLMs from scratch

#77

Post by loveparade »

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions.

But of course, 10 years ago this wasn't obvious.

Re: I were 17, I'd learn how to build LLMs from scratch

#78

Post by dakolli »

What the fuck does this guy know about? I'm sure if we went back through similar statements he's said over the years he's said the same thing about various technologies that are no longer relevant. The guy is a talentless hack who larps as a blogger and his only "redeeming" quality is having lots of money.

Owner of Golf Club Company says I should dedicate my life to golf lmfao.

Re: I were 17, I'd learn how to build LLMs from scratch

#79

Post by dakolli »

loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

Yeah he's a moron with a lot of money, that's about it. I'm sure he's said the same thing about various other bags he had bets on throughout the years.

Re: I were 17, I'd learn how to build LLMs from scratch

#80

Post by JuniperMesos »

mcmoor wrote: post #38
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM? It seems like another thing that needs big investment to work properly, unlike those other techs where results can be shown even with little materials.

Yeah, there are retrocomputing hobbyists who mess around with sometimes-physically-large computers that were important many decades ago. I don't know if anyone is hacking on vacuum tubes of the kind that you could in principle build a computer with - there's a reason they became obsolete for digital computation almost as soon as the transistor was invented. On the other hand, I personally think it would be neat to try to build a CRT in a garage, which is of course a type of vacuum tube. I don't think this would be an easy garage project, but it does seem like might be tractable for someone who understand physical manufacturing and electronics well, has access to glassblowing equipment, etc.
Post reply on HN