Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

551–560 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#551

Post by __rito__ »

lucasoshiro wrote: post #489
__rito__ wrote: post #90

Earlier quoted context omitted.

1. Build an LLM from Scratch by Sebastian Raschka ( https://sebastianraschka.com/llms-from-scratch/ ) 2. LLM from 0 to Hero, and nanoGPT by Andrej Karpathy

I second 1. I'm a newbie in neural networks and I think it's an excellent book! One of my barriers in ML is the resources, I find them overcomplicated or too simplistic without a mid term. It's not the case of this book, everything is well-explained. Neural Networks aren't fun for me, but this book makes it very interesting.

If you are serious about ML, you can email me to get recommendations on resources.

(Fully free, of course.)

Re: I were 17, I'd learn how to build LLMs from scratch

#552

Post by islandfox100 »

kdkdifnejd wrote: post #535
mixmastamyk wrote: post #500

Earlier quoted context omitted.

That may have worked at the time, but no companies are interested in learning projects today. If you didn’t do the reqs list for the last 5-10 years with the same title, forget about it. Because there are dozens of folks who have, lined up. No one is indulging career changers (and most fresh grads) for the time being.

> No one is indulging career changers (and most fresh grads) for the time being. Oh hey thank you for that. It really helps. Hope you have your rug pulled from under you today too. — signed, a career changer trying his best.

it's just a fact of the market, it's not an attack on you.

Re: I were 17, I'd learn how to build LLMs from scratch

#553

Post by 4lx87 »

If I were 17, I'd be going to parties, music festivals, chasing girls, and enjoying my youth.

But sure, make the kids even more depressed by telling them they need to learn how to build an LLM so they can get a job working themselves to death to make Paul and friends rich.

Re: I were 17, I'd learn how to build LLMs from scratch

#554

Post by miki123211 »

femto wrote: post #294
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good l…

Same with telecom companies.

We don't really have the demand for as many telecom companies as actually exist in the world. There's a reason we just have one Whatsapp and one Instagram, not three or four almost-but-not-quite clones in every single country that mostly differ in branding. The reason for the current situation has mostly to do with regulation and traditional, enterprise, "obviously every country needs a separate local branch, because that's what mcDonalds does" thinking. Technology has very little to do with it.

This is why the telecom world now consist of equipment manufacturers, who do most of the hard tech stuff, and actual telecom companies, who operate the equipment, rig towers in their local country, and maybe write some glue code to integrate a core from vendor A, a billing system from vendor B and a CRM / corporate invoicing system from government-approved local vendor C.

Banking also works similarly, though modern Neobanks / Fintechs and bank consolidation are slowly dissolving the concept of national bank branches.

Re: I were 17, I'd learn how to build LLMs from scratch

#555

Post by LarsDu88 »

I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded.

There are other types of models like diffusion models right now that are showing more efficiency and have a higher ceiling for improvement. Understanding math and fundamentals are more important.

Re: I were 17, I'd learn how to build LLMs from scratch

#556

Post by drblast »

Mr. McGuire: I just want to say one word to you. Just one word.

Benjamin: Yes, sir.

Mr. McGuire: Are you listening?

Benjamin: Yes, I am.

Mr. McGuire: Plastics.

Benjamin: Exactly how do you mean?

Mr. McGuire: There's a great future in plastics. Think about it. Will you think about it?

---

I love this scene because it so perfectly captures what it's like to be young and given advice, however well-meaning, by an older generation living in a world that no longer exists for the young. And it's ambiguous and trite enough to be essentially useless even if the underlying idea isn't terrible.

Re: I were 17, I'd learn how to build LLMs from scratch

#557

Post by dang »

4lx87 wrote: post #553

If I were 17, I'd be going to parties, music festivals, chasing girls, and enjoying my youth. But sure, make the kids even more depressed by telling them they need to learn how to build an LLM so they can get a job working themselves to death to make Paul and friends rich.

This community is filled with people who were obsessed with microcomputers or (depending on which generation) websites during their teenage years. HN was created by and is managed by people of this type. No doubt we're a minority, and the type you describe is more common, but if you're implying there's something wrong with teenagers being intellectually curious about technology, I can't help but think of the way "nerds" used to get put down and shamed in the past.

Re: I were 17, I'd learn how to build LLMs from scratch

#558

Post by janalsncm »

andy_ppp wrote: post #419
mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I think it’s reasonable to have a shallow understanding of most parts and a deep understanding of a small number of parts. That’s how most engineers are.

Most software engineers do not have a deep understanding of CPU architectures. In fact they probably don’t even have a shallow understanding and get around just fine. How many of them are looking up the instruction set for the CPUs they deploy their CRUD app to in EC2?

Re: I were 17, I'd learn how to build LLMs from scratch

#559

Post by Petersipoi »

rfgplk wrote: post #312
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

> The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams. And the job postings are often ridiculous. I recently was an AMD…

It reminds me of that "stone soup" story.

1. I can make turn a stone and water into a delicious soup

"17 year olds, learn to build an LLM from scratch"

2. This soup would be more delicious if we add a few carrots. Does anyone have carrots

"Increase your chance of success by getting a Masters degree"

3. How about potatoes?

"And get a PHD"

4. What about some salt?

"And publish some peer reviewed articles in {journals_list}

5. We should also add beef

"Now work in the industry for 10 years"

6. See, this soup is delicious, and I made it all with a stone

"See, you're rich, and it's all because you learned LLMs as a 17 year old"

Re: I were 17, I'd learn how to build LLMs from scratch

#560

Post by tech_ken »

janalsncm wrote: post #558
andy_ppp wrote: post #419

Earlier quoted context omitted.

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I think it’s reasonable to have a shallow understanding of most parts and a deep understanding of a small number of parts. That’s how most engineers are. Most software engineers do not have a deep understanding of CPU architectures. In fact they probably don’t even have a shallow understanding and get around just fine. How many of them are looking up the instruction set for the CPUs they deploy their CRUD app to in E…

But in the case of CPU architecture there are SOME people who understand how things work 100%, and they've built and vetted abstractions/mental models that enable other engineers and scientists to have that kind of mixed shallow/deep understanding in a way that works. On the side of LLMs we're still lacking an expertise which could flawlessly explain how these things operate; the abstractions that we're using are instead derived inductively and are totally unvetted.
Post reply on HN