Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

581–590 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#581

Post by flockonus »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

Agreed. During comp sci we got to re-implement various algos of networks, OS, database, firmware.. and it all gave complimentary intuitions that were useful when tackling practical implementations and bottlenecks.

Re: I were 17, I'd learn how to build LLMs from scratch

#582

Post by vlyan »

MaxHoppersGhost wrote: post #358
vlyan wrote: post #209

Earlier quoted context omitted.

"it doesn't matter what you choose to study because no job is safe from being outsourced overseas, handed over to an indentured servant, or made redundant by a machine. you will spend your whole life surviving while the cannibalistic pedophiles who own everything invent new ways to make you own nothing. be frugal, don't get married, don't have children, do everything you can to stay healthy and independent, and you j…

> don't get married, don't have children Bad advice. You’re going to need family connections and loyal people you can bet your life on to survive if the rest of what you’re saying is remotely true.

the likelihood of finding a partner you can bet your life on is nonexistent. the likelihood of the opposite outcome is well-known at this point.

Re: I were 17, I'd learn how to build LLMs from scratch

#583

Post by nialv7 »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.

Re: I were 17, I'd learn how to build LLMs from scratch

#584

Post by tech_ken »

janalsncm wrote: post #570
tech_ken wrote: post #560

Earlier quoted context omitted.

But in the case of CPU architecture there are SOME people who understand how things work 100%, and they've built and vetted abstractions/mental models that enable other engineers and scientists to have that kind of mixed shallow/deep understanding in a way that works. On the side of LLMs we're still lacking an expertise which could flawlessly explain how these things operate; the abstractions that we're using are ins…

You are right that the field doesn’t have a theoretically sound explanation for the architectural choices aside from “A works better than B”. However, I would argue this is an ideal opportunity for the “gentleman scientist” or eager 17 year old. Basically every part of the original transformer was replaced with something more efficient or better: LayerNorm -> RMSNorm Sinusoidal position encoding -> RoPE MHA -> GQA Re…

Oh yeah I absolutely agree that there's lots of greenfield for relative newcomers to find some big improvements! I'm just saying that reasoning about 'best practices' in terms of what works for something like a CPU is missing some important and relevant differences.

Re: I were 17, I'd learn how to build LLMs from scratch

#585

Post by quikoa »

sailorganymede wrote: post #507

The core point here is that AI is a massive thing (at the moment) so it's probably a good idea to understand it deeply. Not sure why people are so worked up about it.

> (at the moment)

That's why. That future is uncertain. So why gamble your future on something that's popular at the moment for something that could change completely a year later?

Re: I were 17, I'd learn how to build LLMs from scratch

#587

Post by [deleted] »

tech_ken wrote: post #550
megaloblasto wrote: post #449

Earlier quoted context omitted.

I disagree. It's absolutely possible to develop an intuition about extremely complex mathematical ideas, including llms or high dimensional systems. Learning to build an llm is a great way to start building that intuition.

Developing an intuition about high dimensional systems is pretty different from understanding the character of some specific point on a 1e9+ dimensional manifold of parameters, in my professional opinion (setting aside all the degrees of freedom that come from the structure of the thing). Sure one can understand generic principles like the curse of dimensionality, but truly groking how an LLM works is basically an op…

[deleted]

Re: I were 17, I'd learn how to build LLMs from scratch

#588

Post by BodyCulture »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

The next class of problems to solve for future generations is how to survive on a hot planet and how to recover from the damage done by reckless lunatic profiteering assholes.

Re: I were 17, I'd learn how to build LLMs from scratch

#589

Post by michael0church »

nialv7 wrote: post #583
mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.

This is a great point. The complexity sources are entirely different. An OS is complex because it’s solving a difficult problem and we’ve seen decades of evolution both in knowledge and-how and, at the frontier, in performance standards.

The abilities of LLMs are emergent. You can experiment with LLMs and know as much about their observable behavior as the experts. But there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives; as far as we currently understand, it’s all mashed together.

Re: I were 17, I'd learn how to build LLMs from scratch

#590

Post by potatoking »

BodyCulture wrote: post #588
mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

The next class of problems to solve for future generations is how to survive on a hot planet and how to recover from the damage done by reckless lunatic profiteering assholes.

Maybe people want to take it as cynicism but I think you are factually correct that personal cooling, infectious disease, atypical weather and fire breaks are all more realistic topics for people too young to already have patents, connections or their own oil well.
Post reply on HN