Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

651–660 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#651

Post by zephyreon »

Sorry but Paul is just so out of touch with reality. I’m tired of seeing everyone on HN evangelize this man. He is very intelligent and has done some really great things but times have changed and we need to let the next generation think for themselves. They are increasingly anti-AI. Maybe that pans out, maybe it doesn’t. If I were 17, I’d pursue my dreams — whatever those may be — and learn how to connect meaningfully with my peers. Perhaps that’s naive of me or perhaps that’s what humanity needs most right now. I’d rather let the 17 year old version of me make that choice for themselves.

I’m sure I’ll get torn to pieces for this but it’s frustrating to continually witness people treat a single person’s prose as the Word of God.

Re: I were 17, I'd learn how to build LLMs from scratch

#652

Post by mettamage »

fancyfredbot wrote: post #445
sarmike31 wrote: post #291

Earlier quoted context omitted.

Which resources from these two would you recommend? Or just blanket-recommend all their videos/books?

I kind of want to blanket recommend but that's not very helpful I would suggest starting with Andrej Karpathy's YouTube video: https://youtu.be/kCc8FmEb1nY?is=oiDsrBYJg_MUUmoD This video is excellent. I'm a huge fan. Also the video is zero commitment and instantly available which makes it a good way to check you are interested. The book by Sebastian Raschka is slightly less accessible but very reasonably priced and t…

Thanks! Will definitely look into it. I'm currently grinding Math Academy to fix my math deficiencies so I can understand LLMs.

Re: I were 17, I'd learn how to build LLMs from scratch

#653

Post by LarsDu88 »

lern_too_spel wrote: post #604
LarsDu88 wrote: post #555

I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded. There are other types of models like diffusion models right now that are sho…

Yann has been consistently wrong about the limits of LLMs.

He has, but I think he's right in the sense that there will be better models. It would be premature to think that the current frontier model architectures will last into the future. It would be like making DVD engineering a standard part of high school curricula

Re: I were 17, I'd learn how to build LLMs from scratch

#658

Post by conartist6 »

chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

No, but there's a WHOLE ENTIRE WORLD of tech that isn't LLMs. LLMs aren't the future or anything. Sure they'll keep existing. Right now though all the opportunities exist for people looking literally anywhere else

Re: I were 17, I'd learn how to build LLMs from scratch

#659

Post by lstodd »

lossolo wrote: post #637
megaloblasto wrote: post #596

Earlier quoted context omitted.

I think provable conclusions and building an intuition are different things to be fair. In my opinion it is entirely possible to build intuition about seemingly impossible to understand topics like infinitesimal areas, infinite limits, function mappings, and high dimensional spaces. For example, 3D spaces are easy to intuit if you can visualize vectors in your head. Some people claim it is impossible to understand di…

It's like saying that horse can understand physics, because he knows when to jump using his intuition.

Horse, ha.

Like a cat who knows how to teleport so rapidly around the room that he becames a blur. All the while taking into account stuff falling around he knocked down.

Re: I were 17, I'd learn how to build LLMs from scratch

#660

Post by musebox35 »

rdedev wrote: post #235
kalms wrote: post #87

Earlier quoted context omitted.

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful. The above is, after all, the wh…

How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasona…

This is one question that is not just for kids but for all "gpu poor" adults as well. One concept that I found useful is "toy models": small compute models that replicate a certain, not very well explained phenomenon that the large models have. Of course, you can't have a toy model for everything, some behaviors are only visible at scale. But, there is a bunch of things like in-context learning that you can play with without burning a ton of compute.

The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.

Post reply on HN