Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

131–140 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#131

Post by BoredomIsFun »

molf wrote: post #109
BoredomIsFun wrote: post #54

Earlier quoted context omitted.

> Why would you tell people that the correct order Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17? > For some (many?) people, a 'proper' understanding develops _after_ the exploration. I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be una…

> unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring" That's what I would call exploring. I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.

> That's what I would call exploring.

Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.

> Fooling around and getting results is what made it fun. Understanding came later.

It is not how LLMs are "built from scratch".

Re: I were 17, I'd learn how to build LLMs from scratch

#132

Post by wy35 »

loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

What would be better advice for a 17 year old?

Re: I were 17, I'd learn how to build LLMs from scratch

#133

Post by kevmo314 »

teaearlgraycold wrote: post #69
kevmo314 wrote: post #16

Earlier quoted context omitted.

That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.

Interesting/capable diffusion models are much smaller than similarly interesting language models. But yes you could always scale things down to learn the fundamentals.

There are plenty of similarly tiny language models in the realm of tts too. Gatekeeping what’s interesting misses the forest for the trees.

Re: I were 17, I'd learn how to build LLMs from scratch

#134

Post by the_gipsy »

oersted wrote: post #47
spwa4 wrote: post #20

Earlier quoted context omitted.

That's not true, because everyone, everyone, everyone seems to want to do training. Which results in a 50 person company training, say, a voice model that then fails, because it's just not good enough. In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% bette…

Everyone says they want to do training, because it's sexy and an easy way to justify raising mad funding rounds. Some manage, most don't. I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if…

It took a long time to cross that desert, and no sane company would want to get stuck in a desert, unless it's specifically an R+D "desert crossing" company.

Re: I were 17, I'd learn how to build LLMs from scratch

#135

Post by rrr_oh_man »

reacharavindh wrote: post #103
oersted wrote: post #19

Earlier quoted context omitted.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes. On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources envir…

> On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….

10000%.

Re: I were 17, I'd learn how to build LLMs from scratch

#136

Post by kevmo314 »

DanielHB wrote: post #113
kevmo314 wrote: post #16

Earlier quoted context omitted.

That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.

But can you _sell_ that? If you can't you can't get a job doing it.

I suspect most who worked at Google did not work on Google infrastructure before getting there.

Re: I were 17, I'd learn how to build LLMs from scratch

#137

Post by mdp2021 »

livinglist wrote: post #55

When I was 17 I was building Windows Phone apps, bad decision on my part.

I also did, and had written code for/on other mobile devices before, and did write a large number of even very ambitious software for other mobile devices later.

I do not see any past constructive experience as a waste of time.

Re: I were 17, I'd learn how to build LLMs from scratch

#138

Post by [deleted] »

embedding-shape wrote: post #118
epolanski wrote: post #111

Earlier quoted context omitted.

> seek advice from unsuccessful people? That seems worse :) Not sure why would you think so. Inverse reasoning is very powerful, and unsuccessful people can give you plenty of "don't do this mistake", which the survivors would not even think about.

> unsuccessful people can tell you teach you plenty of "don't do this mistake", But how can I know for sure that that particular mistake is actually why they were unsuccessful? Has exactly the same issue as listening only to successful people as they hardly know what actually made them successful most of the time, but they still compose large blog posts with their reasoning for why. Again, I still think my approach o…

[deleted]

Re: I were 17, I'd learn how to build LLMs from scratch

#140

Post by yobbo »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age.

It is viable as a toy project, but there are vanishingly few career opportunities.

Post reply on HN