[deleted]
I were 17, I'd learn how to build LLMs from scratch
451–460 of 725 posts
Re: I were 17, I'd learn how to build LLMs from scratch
#452I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka. I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier conc…
Which resources from these two would you recommend? Or just blanket-recommend all their videos/books?
actually just use chatgpt. There is a new mode of learning thats now avaiable that doesnt require you to read about things that are already discovered leaving you with a shallow knowledge.
you can now play the inventor and start with question "i want to build next token prediction software" and go as far as you can with your current knowledge while brainstroming with chatgpt as a rubber duck.
dont not start with a course on probabality , linear algebra or calculus . do not watch 3 hr videos or 3blue animations .
there is agood video on this way to learn.
Re: I were 17, I'd learn how to build LLMs from scratch
#453Earlier quoted context omitted.
For this example, if you built a browser in 2000 it would put you in a great position to launch android in 2008. I think this is pg’s point. You spend all this time learning the interesting bits and when an inflection comes along that makes a new product or service possible you could be the one to likely launch it
I am talking from the perspective of the individual developer. You don't need your impressive product to succeed to land a good career. If Alice is doing LLM-from-scratch work today, and Bob is doing agent harness work today, Bob's project is far more likely than Alice's to become useful/popular/profitable. But if neither project survives, in 5 years, Alice will be more employable/at a higher market rate than Bob.
Neither one has even gone to college yet. I’d expect Alice’s project to be more likely to interest professors, get some undergrad research experience, maybe that can turn into something.
Re: I were 17, I'd learn how to build LLMs from scratch
#454I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM? It seems like another thing that needs big investment to work properly, unlike those other techs where results can be shown even with little materials.
> Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM?
"Barely?"
It's insane to lump vacuum tubes together with servers with 1 MB RAM. My first PC, which I used for a decade, had 640KB RAM. And that was an upgrade from 512 KB RAM. My other PC had only 128KB RAM. None of these were considered the equivalent of (by then long dead) vacuum tubes in their day.
You can get a lot done in 1 MB.
Re: I were 17, I'd learn how to build LLMs from scratch
#455Earlier quoted context omitted.
They'll learn so much more that won't transfer to as many job opportunities. For ex, say more about C++ and less about cutting edge CSS (because modern browser tech is an ocean). I suppose they might luck into other adjacent or unrelated roles with the same skills.
Are you asserting that C++ is not a marketable skill? If anyone reading this has this un-marketable skill, Carnegie robotics in Pittsburgh is hiring carnegie-robotics.breezy.hr/p/2d85f5321cc7-software-engineer
I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you had general experience writing CRUD apps..."
Re: I were 17, I'd learn how to build LLMs from scratch
#456Earlier quoted context omitted.
In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good l…
I was there at the start of the smartphone boom. I built a demo Android device that was capable of telephony/data, 3D rendering, etc. all before Google open-sourced the OS, for a SoC vendor that wasn't in Google's inner circle. Yet the industry was not interested in my junior profile during the subprime crisis.
Ridiculous frankly.
Re: I were 17, I'd learn how to build LLMs from scratch
#457Earlier quoted context omitted.
Hey, I'm 19 CS undergrad but i don't really know what to do but I really wanna build something that can shape the world. Can you tell me more about why you would advice someone to learn about transistors? what are the possible career paths?
Not the original commenter, but in my view as a crusty old software engineer, the reason to learn how transistors is because they're interesting and the key to how all computation is implemented. Once you learn how transistors work, you can learn how logic gates work. And then how CPUs and memory work. Bit by bit, computers stop being magic boxes and start being something built from technologies you understand. Now,…
Thank you
Re: I were 17, I'd learn how to build LLMs from scratch
#458There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…
[dead]
Re: I were 17, I'd learn how to build LLMs from scratch
#459I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka. I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier conc…
[deleted]
Re: I were 17, I'd learn how to build LLMs from scratch
#460I thought pg was trying to live forever. Has he learned LLMs from scratch?