Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

631–640 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#631

Post by s1artibartfast »

firemelt wrote: post #487
fancyfredbot wrote: post #85

I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka. I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier conc…

so can u explain to me how they are works?

How much time are willing to spend and money are you willing to pay?

Re: I were 17, I'd learn how to build LLMs from scratch

#632

Post by ebbi »

As someone who doesn't work in this field (I'm in boring old Accounting/Finance), but would like to learn this field (both to future-proof my skillset, but also because I have an interest in it), would it be wise for me to learn math?

Is it something that's going to be fundamental for future technologies? I always plan to learn, but end up in a death spiral feeling like I'll invest so much time and energy only for the puck to have moved somewhere completely different.

Would gladly accept any advice :)

Re: I were 17, I'd learn how to build LLMs from scratch

#633

Post by s1artibartfast »

embedding-shape wrote: post #32
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> I am kind of amazed how negative the comments are here, especially on HN. I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hazarding a guess, I think the moment Altman became the CEO and later durin…

If you did a data analysis, I expect you would see an uptick with the 2023 Reddit API controversy. HN got a big influx of redditors with different sociopolitical perspectives and discussion norms.

This might be hard to disentangle from the tech correction layoffs of 2023 and 2024, but my perception is that most of the spite comes from Tech Outsiders opposed to jaded Developers.

Re: I were 17, I'd learn how to build LLMs from scratch

#634

Post by andy_ppp »

janalsncm wrote: post #570
tech_ken wrote: post #560

Earlier quoted context omitted.

But in the case of CPU architecture there are SOME people who understand how things work 100%, and they've built and vetted abstractions/mental models that enable other engineers and scientists to have that kind of mixed shallow/deep understanding in a way that works. On the side of LLMs we're still lacking an expertise which could flawlessly explain how these things operate; the abstractions that we're using are ins…

You are right that the field doesn’t have a theoretically sound explanation for the architectural choices aside from “A works better than B”. However, I would argue this is an ideal opportunity for the “gentleman scientist” or eager 17 year old. Basically every part of the original transformer was replaced with something more efficient or better: LayerNorm -> RMSNorm Sinusoidal position encoding -> RoPE MHA -> GQA Re…

The original RoPE paper is a great example of the authors not actually understanding how their own discovery was functioning fully which is a fun example of what I’m talking about I guess…

Yes you might progress the field, but will you really understand why? You can make up an explanation and anthropomorphise it with a few contrived diagrams and everyone will cheer!

Re: I were 17, I'd learn how to build LLMs from scratch

#635

Post by andy_ppp »

dominotw wrote: post #617
megaloblasto wrote: post #596

Earlier quoted context omitted.

I think provable conclusions and building an intuition are different things to be fair. In my opinion it is entirely possible to build intuition about seemingly impossible to understand topics like infinitesimal areas, infinite limits, function mappings, and high dimensional spaces. For example, 3D spaces are easy to intuit if you can visualize vectors in your head. Some people claim it is impossible to understand di…

i read all that i still dont see what your intuition is about how llm work. visualizign 100 dim matrix will not tell you how llm work. so what you even talking about.

Yes, if it cannot be measured it doesn’t exist no matter how much it would make everyone feel more comfortable if we could intuitively understand these systems.

Re: I were 17, I'd learn how to build LLMs from scratch

#636

Post by paulryanrogers »

dylanowen wrote: post #474
paulryanrogers wrote: post #332

Earlier quoted context omitted.

They'll learn so much more that won't transfer to as many job opportunities. For ex, say more about C++ and less about cutting edge CSS (because modern browser tech is an ocean). I suppose they might luck into other adjacent or unrelated roles with the same skills.

I have a hard time imagining anything where writing a browser wouldn't be an excellent transferable foundation. Even your example: writing css will never teach you as much as writing a css engine.

Sorry if I was unclear. I mean they'll never have enough time to learn a significant amount of modern web tech because they'll be working up to it from lower layers and basics.

Browsers from scratch are multi-year projects for multiple people. Even just skinning and minor tweaks to modern browsers is a deep well for one person.

Re: I were 17, I'd learn how to build LLMs from scratch

#637

Post by lossolo »

megaloblasto wrote: post #596
andy_ppp wrote: post #476

Earlier quoted context omitted.

What provable conclusions have you intuited around how LLMs work? Give me some examples to prove your point? I'm absolutely happy to change my view with enough data.

I think provable conclusions and building an intuition are different things to be fair. In my opinion it is entirely possible to build intuition about seemingly impossible to understand topics like infinitesimal areas, infinite limits, function mappings, and high dimensional spaces. For example, 3D spaces are easy to intuit if you can visualize vectors in your head. Some people claim it is impossible to understand di…

It's like saying that horse can understand physics, because he knows when to jump using his intuition.

Re: I were 17, I'd learn how to build LLMs from scratch

#638

Post by zem »

greenowl wrote: post #486
jgraham wrote: post #252

Earlier quoted context omitted.

This seems like a much more interesting question to me. Telling other people's children what to do is easy and basically doesn't have any downside to being wrong. With your own children things are a bit different. So: what are people here with school age children telling their own kids about the future? If their kids ask, what kind of careers would they encourage them to pursue, assuming they have the skills and inte…

We've wronged an entire generation by pushing "learn to code".

"learn to code because it will get you a good career" perhaps, but "learn to code at at least a high school level, because it will give you a very good mental model for how computers do things" is still very useful.

Re: I were 17, I'd learn how to build LLMs from scratch

#639

Post by dang »

raincole wrote: post #33
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

If he has said "to learn the math and programming skills needed to understand how to build LLMs" it'd have been much more positively received.

He did say that. That's what "from scratch" means, along with actually going ahead and building one.

Re: I were 17, I'd learn how to build LLMs from scratch

#640

Post by danielmarkbruce »

rdedev wrote: post #235
kalms wrote: post #87

Earlier quoted context omitted.

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful. The above is, after all, the wh…

How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasona…

You can build a model with 100m parameters..... the first L in LLM is not meaningful anymore.

So, you build one from scratch.

Post reply on HN