Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

261–270 of 724 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#261

Post by brandonb »

g3e0 wrote: post #257
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

"Necessity is the mother of invention" - limited hardware has always forced people to find cleverer ways of doing more with less. Current models are clearly nowhere near the efficiency limit (the brain does vastly more with far less power).

This is roughly how GPUs for neural networks got started: after Andrew Ng left Google Brain, he no longer had access to a 10,000-CPU cluster used to train the original DistBelief system. But his Stanford students could buy a GPU...

Re: I were 17, I'd learn how to build LLMs from scratch

#262

Post by weatherlite »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

A lot of startup companies are not training frontier models but help solve and optimize pain points of LLMs: cyber security, token usage, harnesses etc. These jobs don't require a PHD in machine learning but it does help if you understand LLMs at a deeper level.

Re: I were 17, I'd learn how to build LLMs from scratch

#264

Post by Avicebron »

giancarlostoro wrote: post #255
yobbo wrote: post #140

Earlier quoted context omitted.

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age. It is viable as a toy project, but there are vanishingly few career opportunities.

Building an LLM covers a lot of CS fundamentals and forces you to do lots of research in order to implement one, especially on more modest hardware.

So do a lot of things, that isn't what Paul is implying when he says "I would build a LLM if I was 17".

Re: I were 17, I'd learn how to build LLMs from scratch

#265

Post by zorked »

serbuvlad wrote: post #190
bmacho wrote: post #162

Earlier quoted context omitted.

It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.

If you learnt to build a browser in 2000 you're probably doing well for yourself lol. Not a lot of demand, but also probably not a lot of supply.

There is an entire graveyward of browsers, almost all that didn't die are now forgotten.

Re: I were 17, I'd learn how to build LLMs from scratch

#266

Post by captainbland »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

The other thing to add as well is that the research teams who do the actual research work are relatively small and very specifically qualified which naturally keeps the barrier to entry high.

Re: I were 17, I'd learn how to build LLMs from scratch

#267

Post by mrits »

bmacho wrote: post #162
yobbo wrote: post #140

Earlier quoted context omitted.

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age. It is viable as a toy project, but there are vanishingly few career opportunities.

It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.

Early internet let me create the best personal web page in my city that I knew of with 2 weeks experience as a 12 year old. I imagine that same 12 year old could be more knowledgeable about LLMs than 99% of people in the same period.

Re: I were 17, I'd learn how to build LLMs from scratch

#268

Post by agentultra »

Most 17 year olds I know don’t want anything to do with AI and see the entire industry as an existential threat.

Nothing wrong with learning the theory and understanding the papers. Getting to that point you’ll have to get your fundamentals down. Might be an interesting exercise.

But as a future? I guess we’ll see. I suspect the next financial apocalypse will determine if there is one. Another AI Winter that may outlast all others so far.

Re: I were 17, I'd learn how to build LLMs from scratch

#269

Post by SadErn »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

[dead]

Re: I were 17, I'd learn how to build LLMs from scratch

#270

Post by make_it_sure »

jasonkester wrote: post #98
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> Personally I do both That’s what you get from listening to “successful people”. You get to learn about all the things they tried that failed, then the things that did work on that 24th try, which was successful. The “survivorship bias” people always seem to assume that the “survivor” lucked into his fortune on his first try ever, so he can’t have learned anything, so we don’t have to listen to him. But that’s seldo…

very good read. I get into the same discussion sometimes with people. If you're successful you're automatically labelled as "lucky".

There are successful people that failed many times before being successful. Somehow these days, previous failures and persistence seems to be ignored and they focus just on the luck you got on the 20th try.

Post reply on HN