Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

291–300 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#291

Post by sarmike31 »

fancyfredbot wrote: post #85

I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka. I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier conc…

Which resources from these two would you recommend? Or just blanket-recommend all their videos/books?

Re: I were 17, I'd learn how to build LLMs from scratch

#292

Post by captainbland »

embedding-shape wrote: post #32
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> I am kind of amazed how negative the comments are here, especially on HN. I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hazarding a guess, I think the moment Altman became the CEO and later durin…

Imo it's a breakdown of trust of the startup ecosystem as a whole. Repeatedly startups have enshittified and it's become undeniable that the investment apparatus around startups is partly responsible. We have seen a great driver of uncreative destruction, industries undermined, small businesses undermined just to drive masses of money into few pockets - less fairness for the people working in what is now the gig economy and ultimately prices and other costs that end up as high or higher than they were before for consumers. Not to mention the whole AI/OpenAI situation which many perceive as threatening their skillset per se, essentially tearing up the social contract that existed on this site.

The sycophancy on HN is starting to break down because there is a higher proportion of users sceptical towards the outputs of the VC and wider investment world than ones who believe they're potential beneficiaries of it.

Tech industry people are becoming less interested in HN as a warm handshake into the startup world because, frequently, they're disgusted by it. And this reflects on the sentiments people post on PG's articles.

Increasingly if those at YC want the same kind of low-bar praise they got before, they will need to get it from machines.

Re: I were 17, I'd learn how to build LLMs from scratch

#293

Post by enos_feedler »

serbuvlad wrote: post #190
bmacho wrote: post #162

Earlier quoted context omitted.

It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.

If you learnt to build a browser in 2000 you're probably doing well for yourself lol. Not a lot of demand, but also probably not a lot of supply.

For this example, if you built a browser in 2000 it would put you in a great position to launch android in 2008. I think this is pg’s point. You spend all this time learning the interesting bits and when an inflection comes along that makes a new product or service possible you could be the one to likely launch it

Re: I were 17, I'd learn how to build LLMs from scratch

#294

Post by femto »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good living. Most people who got rich off phones didn't do it by knowing how phones work.

Incidentally, the skills for the lowest levels of LLMs aren't that far removed from those needed for mobile telephony, in that both are based on maths, computation and information theory.

Re: I were 17, I'd learn how to build LLMs from scratch

#297

Post by HarHarVeryFunny »

LLMs are the new compilers.

I don't think you can really call yourself a developer unless you at least have an idea how to build a more complex software project like a compiler, and maybe have built a toy one either at uni or for fun.

It's not clear how long this LLM age of AI will last (to be replaced by something better), but nowadays any developer should at least understand the basics of ANNs, and more than just the "hello world" of a cat vs dog CNN. An LLM/Transformer is maybe the equivalent of a compiler in that regard - something that we all use and is complex enough to present a bit of a challenge. You should at least understand the basics of how an LLM is built, and maybe building a toy LLM will/should become the new Comp. Sci. degree toy compiler replacement.

Re: I were 17, I'd learn how to build LLMs from scratch

#298

Post by len_testa »

sarmike31 wrote: post #291
fancyfredbot wrote: post #85

I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka. I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier conc…

Which resources from these two would you recommend? Or just blanket-recommend all their videos/books?

I've read Raschka's "Build Your Own LLM from Scratch" book and really enjoyed it. I haven't tried the code yet, but the code from his previous Python ML book worked great.

Re: I were 17, I'd learn how to build LLMs from scratch

#299

Post by ignoramous »

nextaccountic wrote: post #273
bmacho wrote: post #162

Earlier quoted context omitted.

It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.

If someone writes a browser in their teens, they will probably learn more about the web than if they were just writing web pages

A few did.

  [Blake Ross] worked as an intern at Netscape at the age of 16 ... Ross became disenchanted with the browser he was working on and the direction given to it by America Online, which had recently purchased Netscape. Ross and Hyatt envisioned a smaller, easy-to-use browser that could have mass appeal, and Firefox was born from that idea ... in 2003 all of Mozilla's resources were devoted to the Firefox and Thunderbird projects. Released in November 2004, when Ross was 19, Firefox quickly grabbed market share ... with 100 million downloads in less than a year
https://en.wikipedia.org/wiki/Blake_Ross

Re: I were 17, I'd learn how to build LLMs from scratch

#300

Post by armcat »

femto wrote: post #294
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good l…

This is super interesting because I moved from mobile telephony into ML and data science, and information theory and working with data in statistically correct way was what helped me! This was 10 years ago though.
Post reply on HN