Paul Graham wants you to work for him, not be him.
I were 17, I'd learn how to build LLMs from scratch
501–510 of 725 posts
Re: I were 17, I'd learn how to build LLMs from scratch
#502Earlier quoted context omitted.
Veritasium has a good piece for this kind of bias: https://www.youtube.com/watch?v=3LopI4YeC4I An advantage that is not "advisable", like being born in january, in a rich country, in an above average family, or just having luck, might have more influence on the outcome than any conscious action. It is almost sure that one-in-a-million level people only edge over the other 999,999 they competed with is just "have more…
It's not the only edge. Wildly successful people are usually lucky and talented/hard working.
Nobody said otherwise, even the video linked talks about talent (which might be also seen as luck!) and hard working. Say you have 100 people, and you pre-select the 10 hardworkers, that "hardwork" between 98 and 100% capacity: even if luck only accounts for 5% of the success, it doubles the 2% you can still extract from hardwork. Of course, someone that only works at 30% capacity has to be immensenly lucky to overcome the 70% they have until the 100%.
Of the 10 most talented and hardworking people above, I bet they have wildy different advise other than "do hardwork": use more LLM, use less LLM, keep away from computers, etc. And we are talking 1 in 100, not 1 in 1,000,000!
It comes to mind someone like Kary Mullis. He was no dumb, but I wont put him either at the peak of talented or even hard working people in the field. Yet, he was "lucky" enough to invent the PCR, which earned him deservedly the Nobel Price. Now that he was made to believe that he is some kind of illuminated and well above the top human, he started giving (at the bare minimum wrong, probably even dangerous) "advise" left, right and center: take LSD, astrology works, extraterrestial glowing racoons exist, etc. Downtune the guy a notch or two, and you can find thousands of Kary Mullis around giving useless advise as they were messiahs.
Re: I were 17, I'd learn how to build LLMs from scratch
#503Earlier quoted context omitted.
There is an entire graveyward of browsers, almost all that didn't die are now forgotten.
I think the point isn't that you'll necessarily build a great browser (or LLM), but the experience will benefit you in other ways.
Yup like you go far enought you will pick lots of transferable skills like data cleaning in the case of LLM, DOM parsing case of web browser or welding case of rocket.
Re: I were 17, I'd learn how to build LLMs from scratch
#504Earlier quoted context omitted.
They'll learn so much more that won't transfer to as many job opportunities. For ex, say more about C++ and less about cutting edge CSS (because modern browser tech is an ocean). I suppose they might luck into other adjacent or unrelated roles with the same skills.
They will also know the fundamental underpinnings of the web, you know - the thing that those html devs are actually using. That’s like saying someone who fabricates cars doesn’t have the skills to drive them. Perhaps not, but they’re very well placed to pick it up quickly. They’ve also shown they can do something far more challenging, which is actually better than hiring for narrow immediate skills. In other words:…
You'd hire an auto designer to be a limo driver? I think it's just two different skill sets. The auto designer is clearly smart but a limo driver also needs to have customer service skills, etc.
Re: I were 17, I'd learn how to build LLMs from scratch
#505While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…
> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…
The better alternative is to always ignore unsolicited advice.
Re: I were 17, I'd learn how to build LLMs from scratch
#506There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…
AI is the subtrate the future runs on. And so I think the idea is more to understand tomorrow ... from first principles. In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me. I haven't written assembly in years. But whatever I did in my career: it helped me and g…
I don't really agree, I think the future should run on humans.
AI is a great solution in niche areas but generally doesn't make much money. All the large companies are in the negative.
The steam engine was less of a bubble and was much more revolutionary and had a greater impact.
Re: I were 17, I'd learn how to build LLMs from scratch
#507The core point here is that AI is a massive thing (at the moment) so it's probably a good idea to understand it deeply. Not sure why people are so worked up about it.
Re: I were 17, I'd learn how to build LLMs from scratch
#508Paul Graham:
>"Someone asked what I'd do if I were 17. I'd learn how to build LLMs from scratch..."
That's funny, Paul Graham, because if I were 17 again,
I'd learn how to program in LISP.
https://www.paulgraham.com/rootsoflisp.html
https://www.paulgraham.com/iflisp.html
https://www.paulgraham.com/hundred.html
(And/or other LISP derived languages... Clojure, Scheme, Racket, TinyScheme, etc.)
I guess "the grass is always greener..." as that old expression, that old "chestnut", goes... :-)
Re: I were 17, I'd learn how to build LLMs from scratch
#509There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…
It is not a skill that you will use in your day to day life, but I think it is part of the fundamentals now. Sure, LLMs are in a bubble, just like the web during the dotcom bubble, but web didn't disappear, and I don't expect LLMs to, even after the bubble bursts.
I didn't write a LLM from scratch but it is on my "wishlist" so to speak. From what it seems, a GPT-1 class LLM can be done from scratch in a few days and tens of dollars of cloud compute or a high-end gaming GPU.
It is an exercise not unlike building a compiler, a school classic. You are unlikely to ever work on a compiler, but at least, now, you know your tools a little better. It is not about becoming an expert, that takes years, it is about knowing what you are doing.
If you intend to make software engineering your career, you will want more than surface knowledge. And that part is entirely on you, or on your school if you are a student. Companies will not pay for you to learn the fundamentals, they want short term returns, because you may leave at any time. But you as a software engineer may have 40+ years left, so it is worth thinking long term. Claude code may become obsolete a few years, but linear algebra is not going anywhere.
Re: I were 17, I'd learn how to build LLMs from scratch
#510There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…
> With all due respect, paulg is in somewhat of a bubble
I feel like the "ALWAYS HAS BEEN" meme is apropos here.