Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

561–570 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#561

Post by janalsncm »

dominotw wrote: post #452
sarmike31 wrote: post #291

Earlier quoted context omitted.

Which resources from these two would you recommend? Or just blanket-recommend all their videos/books?

actually just use chatgpt. There is a new mode of learning thats now avaiable that doesnt require you to read about things that are already discovered leaving you with a shallow knowledge. you can now play the inventor and start with question "i want to build next token prediction software" and go as far as you can with your current knowledge while brainstroming with chatgpt as a rubber duck. dont not start with a co…

I think ChatGPT is a great way to get depth on a topic. You can Q&A for a while until you get it, and it won’t get impatient with you.

That said, I think traditional resources are a better way to get breadth and to frame the topic. Just chatting about something can be a little disorienting imo.

Re: I were 17, I'd learn how to build LLMs from scratch

#563

Post by mixmastamyk »

islandfox100 wrote: post #552
kdkdifnejd wrote: post #535

Earlier quoted context omitted.

> No one is indulging career changers (and most fresh grads) for the time being. Oh hey thank you for that. It really helps. Hope you have your rug pulled from under you today too. — signed, a career changer trying his best.

it's just a fact of the market, it's not an attack on you.

Indeed, I speak from experience. Generalists like myself are unfavored as well, it seems.

Re: I were 17, I'd learn how to build LLMs from scratch

#564

Post by highfrequency »

andy_ppp wrote: post #419
mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

It is sometimes the opposite - a large number of things makes the system easier to predict and reason about (statistics, behavior of gases etc).

Re: I were 17, I'd learn how to build LLMs from scratch

#565

Post by [deleted] »

Creamsicle47 wrote: post #443
andy_ppp wrote: post #419

Earlier quoted context omitted.

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I'd have to push back, though not on the part you'd expect. Your description of human researchers is roughly right: a lot of the field is try-things-and-narrativize-after. But the load-bearing assumption is that intuition has to be human-shaped intuition. Humans can't intuit thousands of orthogonal directions because we project everything down into a 3D metaphor and hope it holds. That's a fact about our hardware, no…

[deleted]

Re: I were 17, I'd learn how to build LLMs from scratch

#566

Post by kazinator »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

[dead]

Re: I were 17, I'd learn how to build LLMs from scratch

#567

Post by kazinator »

janalsncm wrote: post #558
andy_ppp wrote: post #419

Earlier quoted context omitted.

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I think it’s reasonable to have a shallow understanding of most parts and a deep understanding of a small number of parts. That’s how most engineers are. Most software engineers do not have a deep understanding of CPU architectures. In fact they probably don’t even have a shallow understanding and get around just fine. How many of them are looking up the instruction set for the CPUs they deploy their CRUD app to in E…

In fact, one of the jobs of an engineer is to make sure that other engineers who don't work in his or her area do not need to understand that area deeply, yet build something reliable with it. They need just the summary that he or she writes up into the datasheet for the part. Ensure these conditions are met for safe/reliable operation, give it these inputs, expect these outputs, these timings, this energy consumption, this heat generation, frequency response, tensile strength, whatever.

Re: I were 17, I'd learn how to build LLMs from scratch

#568

Post by jcmoscon »

I think learning how to build your own agent harness framework from scratch and really understand each part of it, what are the modern components that a good agent harness are using these days, is more valuable than learn how to build LLMs, but that depends on what you want to do with your career.

Re: I were 17, I'd learn how to build LLMs from scratch

#570

Post by janalsncm »

tech_ken wrote: post #560
janalsncm wrote: post #558

Earlier quoted context omitted.

I think it’s reasonable to have a shallow understanding of most parts and a deep understanding of a small number of parts. That’s how most engineers are. Most software engineers do not have a deep understanding of CPU architectures. In fact they probably don’t even have a shallow understanding and get around just fine. How many of them are looking up the instruction set for the CPUs they deploy their CRUD app to in E…

But in the case of CPU architecture there are SOME people who understand how things work 100%, and they've built and vetted abstractions/mental models that enable other engineers and scientists to have that kind of mixed shallow/deep understanding in a way that works. On the side of LLMs we're still lacking an expertise which could flawlessly explain how these things operate; the abstractions that we're using are ins…

You are right that the field doesn’t have a theoretically sound explanation for the architectural choices aside from “A works better than B”. However, I would argue this is an ideal opportunity for the “gentleman scientist” or eager 17 year old.

Basically every part of the original transformer was replaced with something more efficient or better:

LayerNorm -> RMSNorm

Sinusoidal position encoding -> RoPE

MHA -> GQA

ReLU -> GELU

What this means is that there is ample opportunity to improve on what we’ve done thus far.

Post reply on HN