it is once again time to balance out paulg sycophancy https://idlewords.com/2005/04/dabblers_and_blowhards.htm
I were 17, I'd learn how to build LLMs from scratch
641–650 of 725 posts
Re: I were 17, I'd learn how to build LLMs from scratch
#642A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…
The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.
This is a silly point. Just because the size of these things is too large for our tiny brains, it doesn't mean we have no idea why it does what it does. If you run a physics simulation of a weather system, you have a situation that is unpredictable for a human - but it's not fair to say "we have no idea why the outputs are what they are!!"
Re: I were 17, I'd learn how to build LLMs from scratch
#643A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…
I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.
This is true of most large complex systems though. We run molecular dynamics simulations as an example.
Re: I were 17, I'd learn how to build LLMs from scratch
#644I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
It is indeed puzzling. Back in the day, I built myself a music playing device using TTL gates, eagerly studied what CPU's looked like inside and in general wanted to know how everything worked - even though it had no hope of replicating most of it.
Looking back, it was an ideal start to a 40 year career as a software engineer. And of course currently I'm reading "Why Machines Learn: The Elegant Maths Behind Modern AI" because the urge to understand the machines I want to master and control hasn't gone away.
Re: I were 17, I'd learn how to build LLMs from scratch
#645I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
I like the hacking spirit suggestion. But my disagreement is the specificity of it. The suggestion here seems more inline with not tinkering and hacking but first principles learning typical of science and math curriculum. What made hacking radios and JS cool was the short feedback loops to see the progress you made. Unfortunately for LLMs starting from scratch doesnt give a good feedback loop on improvement, although starting from pre-trained model to quantising, fine-tuning might be more aligned with the hacking culture
Re: I were 17, I'd learn how to build LLMs from scratch
#646Interesting advice I will be interested in how this advice holds up in the future
Re: I were 17, I'd learn how to build LLMs from scratch
#647[dead]
Re: I were 17, I'd learn how to build LLMs from scratch
#648I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
The hacker ethos and "information wants to be free" of the '80s-'90s was drowned out by a flood of coders in the '10s-early '20s whose only creed was FYPM and "what's the minimum career effort for the fattest paycheck?"
Re: I were 17, I'd learn how to build LLMs from scratch
#649LLMs are just one small part of computing. it is fine to learn how they work but they are nothing magical. if you want to build a real working system using LLMs that actually makes money you still have to learn everything else. but i think his point here is that learning LLMs at 17 means hacking together some home brew rig on shitty consumer hardware and if you learn all of that you will be in a much better place than learning how to use chatgpt to automate a spam campaign for your b2b saas startup.
Re: I were 17, I'd learn how to build LLMs from scratch
#650I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
I think it is more about people are a bit sick of filthy rich people giving this kind of advice. I would also not read anything he preaches.
Hacker News is a Y Combinator (which PG co-founded) sponsored website whose original code was created by PG and whose guidelines were originally formed by PG. For those who are "sick"(sic) of rich people, VCs, or capitalism in general, you've come to the wrong place.