I'd say, if you were 17, you'd probably be better off learning how to socialize or at least gracefully approach and relate to people your own age. That's probably the greatest challenge of the near future, sadly.
I were 17, I'd learn how to build LLMs from scratch
591–600 of 725 posts
Re: I were 17, I'd learn how to build LLMs from scratch
#592A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…
The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.
Yah, "building" it is not sufficient. But a lot of times when I build I want to know the why. "Why does gradient Descent have some clever tricks that easily translate to matrix math"?
Lot's of neat stuff to learn.
Re: I were 17, I'd learn how to build LLMs from scratch
#593Earlier quoted context omitted.
"it doesn't matter what you choose to study because no job is safe from being outsourced overseas, handed over to an indentured servant, or made redundant by a machine. you will spend your whole life surviving while the cannibalistic pedophiles who own everything invent new ways to make you own nothing. be frugal, don't get married, don't have children, do everything you can to stay healthy and independent, and you j…
Most reasonable advice I've seen under this post. Having graduated in 2025 I've experienced the horror of competing with infinitely many third worlders in my small country, who will gladly take 1/3 my pay and be serfs. I really hope we get another hiring boom like in 2020 when I decided to study CS. Otherwise my career will be very rough. I love it and can't imagine doing anything else.
awwww…… so cute……
who would have guessed when first world were enjoying their heyday as they were colonizing the world, and teaching everyone and their mother to learn English.
Kinda sad now you have to compete with serfs from thirls world. chu chu chu. so cute.
shouldn’t have robbed others then. a richer world for you comes at poorer world for someone. we are playing infinite game in finite world.
Re: I were 17, I'd learn how to build LLMs from scratch
#594Earlier quoted context omitted.
I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.
I'd have to push back, though not on the part you'd expect. Your description of human researchers is roughly right: a lot of the field is try-things-and-narrativize-after. But the load-bearing assumption is that intuition has to be human-shaped intuition. Humans can't intuit thousands of orthogonal directions because we project everything down into a 3D metaphor and hope it holds. That's a fact about our hardware, no…
This is no longer funny.
Re: I were 17, I'd learn how to build LLMs from scratch
#595Earlier quoted context omitted.
>understanding modern OS and SDE tools for proper implementation of pipeline etc. Can you provide an example?
How would you filter out garbage from your training data, for example? If you are trying to use someone elses corpus, would it be "from the scratch" then?
I quoted the wrong thing. I meant to quote that Calculus is needed for building LLMs when all the equations have been figured out already of what is needed.
Re: I were 17, I'd learn how to build LLMs from scratch
#596Earlier quoted context omitted.
I disagree. It's absolutely possible to develop an intuition about extremely complex mathematical ideas, including llms or high dimensional systems. Learning to build an llm is a great way to start building that intuition.
What provable conclusions have you intuited around how LLMs work? Give me some examples to prove your point? I'm absolutely happy to change my view with enough data.
I think provable conclusions and building an intuition are different things to be fair. In my opinion it is entirely possible to build intuition about seemingly impossible to understand topics like infinitesimal areas, infinite limits, function mappings, and high dimensional spaces. For example, 3D spaces are easy to intuit if you can visualize vectors in your head. Some people claim it is impossible to understand dimensions higher than 3 but that's obviously not true because you could assign a color to each vector to visualize the next dimension. You could then assign a width to visualize a 5th dimension, and an arrow shape to signify the next, and so on and so on. That's more of a creative act that can start to build at least a visual understanding in you head of these higher dimension spaces. That's a way to start to intuit about those things.
Now with llms we need way more that 6 dimensions so we can start thinking of assigning matrices to each 3d point for example. That allows us to increase the dimension from 3 to 3 + whatever the matrix dimension is.
We can visualize the matrices instead of having numbers as having colors for each entry, so they can be a sort of cube with each vowel being a different color.
Now you can start to visually intuit about how these massively high dimensional spaces can be formed of these colored matrices that can react to some input training data.
That's a start of an idea for intuiting things that might seem impossible to have an intuition about. I think visualization is a great way to start.
Re: I were 17, I'd learn how to build LLMs from scratch
#597I had the same inspiration!
I spent a days reviewing the lecture notes for CS336: Language Modeling from Scratch - and then trained a nanoGPT-esque model in PyTorch.
I'd recommend trying it for those who are curious. Computational bottlenecks become much more intuitive when you've looked at the overall process.
Re: I were 17, I'd learn how to build LLMs from scratch
#598A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…
I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.
You may be right. Nonetheless, it's possible that the act of trying to get your brain to understand a system that works in more than a few orthogonal dimensions will help develop useful intuitions about how LLMs, and the many systems/products/robots that will be built on top of them, work.
Re: I were 17, I'd learn how to build LLMs from scratch
#599Earlier quoted context omitted.
Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs. I saw my first language model in action in 20…
> We're not seeing them differentiate from each other, implying that the design space might not be that large. There is the more likely reason they are not differentiating. They use almost exactly the same class of model. Everything is linear, parallelizable. It's incredible path dependence that's now invisible enough we think it's a natural law. Nature is not linear.
Exactly. They had the capital to gather massive amounts of data and compute and took a path that went well and pushed it to it's max. Are still pushing it. Are there better ways that don't need as much data and compute? Well, that's for the 17-year olds who don't have trillions to spend to figure out. I'm hoping they do so in a way that brings back affordable gaming gpus and memory sticks as a thing again.
Re: I were 17, I'd learn how to build LLMs from scratch
#600I think pg answered the question as “what I’d do as a project” and not “what I’d do as a career.” So the critical comments are kind of missing the point, IMO. I don’t see why learning how LLMs work is a bad project for a 17 year old. Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy thing…
> And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.
And this is a problem with modern society today: expecting 17 year-olds to know what they want to do professionally for the rest of their lives, and to focus heavily on professional development aligned to that.
By that age, I think it's not uncommon for individuals to have interests and perhaps even dreams, but a well-defined career focus that serves as the foundation of an actionable skills development plan? Nah. That just isn't common and I'd argue not desirable. 17 year-olds should be exploring their interests, enjoying early adulthood, learning valuable lessons in the social realm, etc. Not training themselves to become compliant little worker bees.