Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

51–60 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#51

Post by hresvelgr »

BoredomIsFun wrote: post #17

I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc. I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into matur…

> I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted

I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.

Re: I were 17, I'd learn how to build LLMs from scratch

#52

Post by ff10 »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

I would seek advice from people who have a theory of why or why not they were successful. A lot of those results were happening in very specific contexts and usually should not be regarded as a blueprint, but as inspiration to whatever I do.

Re: I were 17, I'd learn how to build LLMs from scratch

#53

Post by gf000 »

InsideOutSanta wrote: post #40
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> seek advice from unsuccessful people Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.

Yep, many successful people greatly underappreciate the effect of simple dumb luck in their lives. And often they just make up complex reasoning chains, even wholly believing them, that more of it was in their control/talent, etc.

Nonetheless, there are many successful people I would gladly listen to for advice, though they are often successful in a different meaning than what venture capitalists would use (e.g. parents with great kids, managing to keep a healthy work-life balance, happiness, and maybe even having time to spend on some cool hobby project -- you are heros!)

Re: I were 17, I'd learn how to build LLMs from scratch

#54

Post by BoredomIsFun »

molf wrote: post #34
BoredomIsFun wrote: post #17

I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc. I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into matur…

Why would you tell people that the correct order is to build foundational knowledge before exploring a subject? For some (many?) people, a 'proper' understanding develops _after_ the exploration.

> Why would you tell people that the correct order

Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?

> For some (many?) people, a 'proper' understanding develops _after_ the exploration.

I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".

If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.

Re: I were 17, I'd learn how to build LLMs from scratch

#56

Post by mickeyp »

Writing, supervising and training LLMs are now the purview of... even larger LLMs. Optimising CUDA kernels; hand-writing SIMD assembly to speed up data loading; tinkering with your particular brand of DRAM to see if there's anything to gain from optimising for its memory topology and NUMA --- these are now the job of AI.

There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.

Re: I were 17, I'd learn how to build LLMs from scratch

#57

Post by oersted »

willtemperley wrote: post #49
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues. 1. Both training and optimisation will get significantly cheaper and easier quickly. 2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029. 3. The big AI firms will become part of the surveillance capitalism network, i…

The argument is sound, but the maths don't math for now, and it's unclear when/if they will.

For the time being, unless you truly have millions, the outcome from training will be very net negative, while focusing on building on top of existing AI will yield amazing things if you apply the same talent and effort.

When it does get cheaper, then it will be easier to acquire the skills and experience too, and the struggle you went through by trying to do it now will be somewhat wasted.

Besides, I am well versed in this field, and it is not rocket science. There are plenty of software engineering domains that are a lot more challenging, like high-end graphics, large-scale data engineering or kernel programming. People will learn to train LLMs when people want them to.

Re: I were 17, I'd learn how to build LLMs from scratch

#59

Post by cultofmetatron »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> Would you advise a 17 y.o. to learn how transistors work or how to code

how many of us out here are doing work directly in what we got a degree in? I majored in economics and now I'm a CTO.

I would absolutely advise a 17 yo to learn how to code, understand how transitors work and how to code an llm. even if he never works on llms, you basically end up with a kid with applied knowlege of statistics, math, physics hardware, logic and a whole lot of practice in critical thinking.

Post reply on HN