Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

621–630 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#621

Post by ta93754829 »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

I think you're conflating complex and complicated. An OS is very complicated but is predictable. An LLM is complex and not necessarily understandable.

Re: I were 17, I'd learn how to build LLMs from scratch

#623

Post by pwdisswordfishs »

andy_ppp wrote: post #419
mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

So computer science actually involves science now.

Re: I were 17, I'd learn how to build LLMs from scratch

#624

Post by s1artibartfast »

tech_ken wrote: post #550
megaloblasto wrote: post #449

Earlier quoted context omitted.

I disagree. It's absolutely possible to develop an intuition about extremely complex mathematical ideas, including llms or high dimensional systems. Learning to build an llm is a great way to start building that intuition.

Developing an intuition about high dimensional systems is pretty different from understanding the character of some specific point on a 1e9+ dimensional manifold of parameters, in my professional opinion (setting aside all the degrees of freedom that come from the structure of the thing). Sure one can understand generic principles like the curse of dimensionality, but truly groking how an LLM works is basically an op…

Seems like a silly goalpost.

As you say, there's a lot of value to unlock with understanding the generic principles, and I would add specific application.

Lots of people are building fantastic Tools or pulling down million dollar salaries without groking the precise representation of a single weight.

Re: I were 17, I'd learn how to build LLMs from scratch

#625

Post by CamperBob2 »

ryandrake wrote: post #464
virtualritz wrote: post #246

Earlier quoted context omitted.

AI is the subtrate the future runs on. And so I think the idea is more to understand tomorrow ... from first principles. In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me. I haven't written assembly in years. But whatever I did in my career: it helped me and g…

> AI is the subtrate the future runs on. We don't know this. So many people are simply claiming this confidently, and a lot of them are betting their careers on it, but nobody has a crystal ball. Whenever someone tells you confidently, and without any doubt or qualifications, that something "is the future," be skeptical. I remember when the Segway was definitely going to change urban planning worldwide.

Segways failed because they were a solution looking for a problem. You can't say that about current-gen LLMs, much less AI in general.

Re: I were 17, I'd learn how to build LLMs from scratch

#626

Post by megaloblasto »

dominotw wrote: post #617
megaloblasto wrote: post #596

Earlier quoted context omitted.

I think provable conclusions and building an intuition are different things to be fair. In my opinion it is entirely possible to build intuition about seemingly impossible to understand topics like infinitesimal areas, infinite limits, function mappings, and high dimensional spaces. For example, 3D spaces are easy to intuit if you can visualize vectors in your head. Some people claim it is impossible to understand di…

i read all that i still dont see what your intuition is about how llm work. visualizign 100 dim matrix will not tell you how llm work. so what you even talking about.

Visualizing large dimensional matrices is a way of starting to develop intuition about llms. If you can't visualize high dimensional spaces, and you haven't studied llms, you're not going to have any intuition about how they work.

Depending on your knowledge of math, I recommend starting with linear algebra, building an understanding of the equations and try to visualize more and more complex systems, then study llms to see how you can apply your linear algebra intuition to your understanding of llms.

VTK is a great toolkit for visualizing complex systems. 3 blue one brown on YouTube has other visuals that might help you.

It takes time but it's possible. Good luck.

Re: I were 17, I'd learn how to build LLMs from scratch

#628

Post by Grombobulous »

khalic wrote: post #347
Grombobulous wrote: post #325

Earlier quoted context omitted.

Maybe this is a dumb analogy but is that kind of like saying “well EVs will still use lubricant! Cars that use oil are here to stay!”? The point being that the way we heavily rely on LLMs specifically right now is likely to cease to be the case, and that in the future they’ll be almost like an irrelevant component.

There's far more shared concepts between them than you think. You should check the code of both world models (JEPA class for example) and compare to GPT. Many of the tricks stay the same, representation is still embeddings, there is a loss function, etc. The exact architecture will change, but unless there's a new discovery in that area, we've cracked the text component already. We're hitting the limits of LLMs becau…

This is very interesting to me, and I have new things to read into, thank you.

Re: I were 17, I'd learn how to build LLMs from scratch

#630

Post by blitzar »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

Just play the lottery once, pick the correct numbers and win $100mil.

If you can't do that - not sure anyone can help you.

Post reply on HN