Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

671–680 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#671

Post by carbondrone »

wy35 wrote: post #132
loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

What would be better advice for a 17 year old?

Be first to invent and patent the "$15 dollar, battery-less, flying carbon capture drone" and save the world from global warning.

What? You want me to tell kids not to be ambitious?

Re: I were 17, I'd learn how to build LLMs from scratch

#672

Post by carbondrone »

lern_too_spel wrote: post #604
LarsDu88 wrote: post #555

I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded. There are other types of models like diffusion models right now that are sho…

Yann has been consistently wrong about the limits of LLMs.

Well good for them, I think their comment is heavily suggesting they're dissatisfied if said limits, and want to see them broken.

Re: I were 17, I'd learn how to build LLMs from scratch

#673

Post by finaard »

Funnily enough, I recently started on a series of demos and documentation to explain to my 10 year old daughter how LLMs work - in this case in the context of "don't use LLMs for looking stuff up if you figure out how to circumvent the school blocks yet again, or if you do, at least verify its output", but also figured it can be useful in general. Plus, if I get a 10 year old to understand that there's a slim chance I can get customers to understand that as well.

Re: I were 17, I'd learn how to build LLMs from scratch

#674

Post by aaron695 »

LarsDu88 wrote: post #555

I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded. There are other types of models like diffusion models right now that are sho…

[dead]

Re: I were 17, I'd learn how to build LLMs from scratch

#675

Post by southbound »

floro wrote: post #290
vlyan wrote: post #209

Earlier quoted context omitted.

"it doesn't matter what you choose to study because no job is safe from being outsourced overseas, handed over to an indentured servant, or made redundant by a machine. you will spend your whole life surviving while the cannibalistic pedophiles who own everything invent new ways to make you own nothing. be frugal, don't get married, don't have children, do everything you can to stay healthy and independent, and you j…

Most reasonable advice I've seen under this post. Having graduated in 2025 I've experienced the horror of competing with infinitely many third worlders in my small country, who will gladly take 1/3 my pay and be serfs. I really hope we get another hiring boom like in 2020 when I decided to study CS. Otherwise my career will be very rough. I love it and can't imagine doing anything else.

> Having graduated in 2025 I've experienced the horror of competing with infinitely many third worlders in my small country, who will gladly take 1/3 my pay and be serfs

You should really read up on disruptive innovation. Many of those who you call "serfs" who'll "take 1/3rd of my pay" will tomorrow establish themselves, justify their presence, and move up the wage/income scale, while you'll be left complaining.

I am from and live in a third-world country and have spent some time in a first-world one. I've seen the ETH Zurich tag open doors that I didn't even know existed. If you aren't leveraging your passport, your education and your network to reach a place where you don't need to worry about "serfs", then you're doing something seriously wrong. What you have, compared to the competition you denigrate, is invaluable, making best use of it is your job.

Re: I were 17, I'd learn how to build LLMs from scratch

#676

Post by mcmoor »

BeetleB wrote: post #454
mcmoor wrote: post #38

Earlier quoted context omitted.

Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM? It seems like another thing that needs big investment to work properly, unlike those other techs where results can be shown even with little materials.

> Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM? "Barely?" It's insane to lump vacuum tubes together with servers with 1 MB RAM. My first PC, which I used for a decade, had 640KB RAM. And that was an upgrade from 512 KB RAM. My other PC had only 128KB RAM. None of these were considered the equivalent of (by then long dead) vacuum tubes in their day. You can…

Yeah 1 MB is too big for my part. But I mean is there anyone hacking with computers in 50s when it's still vacuum tubes and to get 100KB RAM you already need "supercomputer".

Re: I were 17, I'd learn how to build LLMs from scratch

#678

Post by auggierose »

danielmarkbruce wrote: post #642
nialv7 wrote: post #583

Earlier quoted context omitted.

The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.

This is a silly point. Just because the size of these things is too large for our tiny brains, it doesn't mean we have no idea why it does what it does. If you run a physics simulation of a weather system, you have a situation that is unpredictable for a human - but it's not fair to say "we have no idea why the outputs are what they are!!"

That is not a silly point at all. You confuse understanding the mechanics of it with having a theory of why it works. For the physics simulation, physics provides us with the theories which give us the equations underlying the physics simulation. For neural networks, why is next word prediction giving us AI that can do math? We don't really know!

Re: I were 17, I'd learn how to build LLMs from scratch

#679

Post by zethsg »

felixrieseberg wrote: post #43

I'll use this post as a shameless opportunity to tell more people about a little side project, I made: http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.

Man, this is excellent! thank you so much for sharing this for free

Re: I were 17, I'd learn how to build LLMs from scratch

#680

Post by vlyan »

throwaway1183 wrote: post #593
floro wrote: post #290

Earlier quoted context omitted.

Most reasonable advice I've seen under this post. Having graduated in 2025 I've experienced the horror of competing with infinitely many third worlders in my small country, who will gladly take 1/3 my pay and be serfs. I really hope we get another hiring boom like in 2020 when I decided to study CS. Otherwise my career will be very rough. I love it and can't imagine doing anything else.

awwww…… so cute…… who would have guessed when first world were enjoying their heyday as they were colonizing the world, and teaching everyone and their mother to learn English. Kinda sad now you have to compete with serfs from thirls world. chu chu chu. so cute. shouldn’t have robbed others then. a richer world for you comes at poorer world for someone. we are playing infinite game in finite world.

very compelling. do the nations that never participated in colonization -- such as Switzerland, for example -- get exemption, or is the blood guilt shared among all wh*te people?
Post reply on HN