Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

111–120 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#111

Post by epolanski »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> seek advice from unsuccessful people? That seems worse :)

Not sure why would you think so.

Inverse reasoning is very powerful, and unsuccessful people can give you plenty of "don't do this mistake", which the survivors would not even think about.

Re: I were 17, I'd learn how to build LLMs from scratch

#112

Post by wiseowise »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

Kids don’t even know what it is, even less actually feel what it is. At 17 you think that it won’t hit you, that you will be the one to survive until you don’t.

Re: I were 17, I'd learn how to build LLMs from scratch

#113

Post by DanielHB »

kevmo314 wrote: post #16
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.

But can you _sell_ that? If you can't you can't get a job doing it.

Re: I were 17, I'd learn how to build LLMs from scratch

#114

Post by embedding-shape »

epolanski wrote: post #107
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

+1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. The only jobs that he found he was highly over qualified or paid very little. In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.

> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position.

Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more work, some of them even trying to recruit me to work for them full-time as well. I'm based in continental Europe, fwiw.

Re: I were 17, I'd learn how to build LLMs from scratch

#115

Post by epolanski »

wiseowise wrote: post #110

[flagged]

Is going to university really that good of advice nowadays?

Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker.

edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went into trades, one learned to tattoo fake eyelashes, one became a (successful) farmer and most make significantly more than the average law/chemist/mathematician/physics/architecture/languages graduates. Sure, the lowest salaries are to be found among the non-graduates too, but I don't see any evidence that graduates make that much more, and that graduating is worth it.

Some answers talking about how "formative college is", but my 25 years old friend with her own shop knows more about real life, business and economy than ivy league MBAs.

Re: I were 17, I'd learn how to build LLMs from scratch

#116

Post by aaron695 »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

[dead]

Re: I were 17, I'd learn how to build LLMs from scratch

#118

Post by embedding-shape »

epolanski wrote: post #111
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> seek advice from unsuccessful people? That seems worse :) Not sure why would you think so. Inverse reasoning is very powerful, and unsuccessful people can give you plenty of "don't do this mistake", which the survivors would not even think about.

> unsuccessful people can tell you teach you plenty of "don't do this mistake",

But how can I know for sure that that particular mistake is actually why they were unsuccessful? Has exactly the same issue as listening only to successful people as they hardly know what actually made them successful most of the time, but they still compose large blog posts with their reasoning for why.

Again, I still think my approach of reading both but then regardless go my own way is the preferable approach, at least for me, ymmv.

Re: I were 17, I'd learn how to build LLMs from scratch

#119

Post by stymaar »

oersted wrote: post #19
kevmo314 wrote: post #16

Earlier quoted context omitted.

That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

> . But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large"

The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters.

Due to how much high quality data is readily available, anyone can now train a sub-billion (L?)LM on commodity hardware.

And I'm personally convinced that pretty much any enterprise use-case of an LLM (except coding) is better served by a fine-tuned small (<2B) model that is trained specifically on the task, rather than a generalist frontier model, so learning the engineering around fine-tuning is a key skill that companies will realize they need sooner than later.

Re: I were 17, I'd learn how to build LLMs from scratch

#120

Post by utopiah »

eptcyka wrote: post #18
utopiah wrote: post #9

... and it would be totally pointless. I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon. The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training…

Why would 17 year old do something that only brings them money? I do not think Mr Graham here is advocating for the path that makes most money as a result of learning how to train a model. I assume that tinkering and learning about LLMs is what enterprising 17 year olds will do to discover ways they can get a competitive edge or further the SotA with their insights further down the line.

he does mention that it would later on be about building a startup, which is about making money
Post reply on HN