Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

201–210 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#201

Post by khalic »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

lol unnecessary and inefficient…

I just finished fine tuning Gemma e2b for local code completion on my local machine.

This comment just reinforces what the post actually means. We need people that are LLM natives, computing solves itself with time and with scale adjustments

Re: I were 17, I'd learn how to build LLMs from scratch

#202

Post by djeastm »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

>What's the alternative really, seek advice from unsuccessful people?

Seek advice from the averagely successful people, since that is statistically what you're most likely to be.

Re: I were 17, I'd learn how to build LLMs from scratch

#204

Post by kypro »

I get the sense things have changed a bit since I graduated and there are lot more jobs in AI outside of academia these days, but it's still a very different field from other SWE pursuits, and it's not really accessible to hacker-minded people.

Learning AI isn't like learning HTML in the 90s then expecting to get a job at a tech company building websites. You can't just "learn how to build LLMs" and expect a frontier lab to hire you so I'd argue this is rather bad advise.

Additionally, unlike web development in the 90s you cant really do anything interesting yourself... All of the interesting/useful stuff will require huge amounts of compute and data so there isn't even much point in learning to start your own thing either.

As someone whose built many of NNs from scratch (hand written code, long before the days of LLMs), it's more or less useless knowledge if I wanted to work in a frontier lab or do anything interesting in the field.

I also think anyone thinking about going into a field which is basically a crossover of CompSci and Maths is absolutely insane right now. Even if you think there is a place for CompSci and Maths post LLMs, there's almost no chance anything you learn today will be relevant to the skills required in say 5-10 years.

Re: I were 17, I'd learn how to build LLMs from scratch

#205

Post by frabcus »

Oddly, I just remembered I did the nearest possible thing to this when I was 17... back in 1991.

On an Amiga, I took various public domain text documents from cover disks and counted the probability of the next word given the previous word. Then spat out random sequences of words from it and printed them out. It was called "Splurge". Basically a very very simple single layer statistical language model.

Some of the sentences were randomly not bad sentences, which seemed amazing at the time!

That kind of thing (and Core Wars and Tierra etc) did lead me to getting a job at an artificial life startup at the end of the decade. But that was in turn about 10/15 years too early (no GPUs).

There's some lesson from this about timing, but honestly I've gained the most as a person when I did something that was fun, ethical and gained an audience. A tricky combination.

Re: I were 17, I'd learn how to build LLMs from scratch

#206

Post by Chance-Device »

I think a problem a lot of people are grappling with here is that due to LLMs and AI generally, it’s basically impossible to predict what the future will look like or what jobs will still be around.

I’d probably say something like: do something you enjoy and seems like it might be useful, but accept that the pace of change may mean that whatever you study ends up being irrelevant.

Whatever solution there ends up being to this, it’s not going to be one that an individual 17 year old can implement. We’re past the point where individual good and bad choices matter that much to economic outcomes.

Re: I were 17, I'd learn how to build LLMs from scratch

#207

Post by sph »

repeekad wrote: post #12
sph wrote: post #8

Earlier quoted context omitted.

Depends if you’re 17 with rich parents or not.

This only changes whether you are naive enough to believe “honest” business means anything in today’s age. If anything, I worry being honest is holding back smart people who try to compete in a rigged game.

There's two ways to read my comment. I believe you read it along the lines of "you need to be rich to start a business".

What I meant was that if you're rich you don't need any trade school or to run a business, you can just sit on your ass living off interest from capital. Only "poor" people need to work for a living.

Re: I were 17, I'd learn how to build LLMs from scratch

#208

Post by pjerem »

wy35 wrote: post #132
loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

What would be better advice for a 17 year old?

Throw the computer and the smartphone out of the window.

Or more reasonably, the same old thing : use Linux, hack a little, why not learn programming basics. But learn to own your technology, fight against centralization of technology. The same old RMS story.

IDK where the tech industry is going, if there will be jobs anymore or not, but what I'm sure (and what have been the case for the last 10-15 years anyway) is that for most tech jobs, having good technical knowledge beyond the basics is pretty useless and will probably not be recognized.

If you can, stay a computer geek if that's your thing, but don't make it your career choice, the Eldorado is behind us.

Re: I were 17, I'd learn how to build LLMs from scratch

#209

Post by vlyan »

wy35 wrote: post #132
loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

What would be better advice for a 17 year old?

"it doesn't matter what you choose to study because no job is safe from being outsourced overseas, handed over to an indentured servant, or made redundant by a machine. you will spend your whole life surviving while the cannibalistic pedophiles who own everything invent new ways to make you own nothing. be frugal, don't get married, don't have children, do everything you can to stay healthy and independent, and you just might live a reasonably comfortable life."

Re: I were 17, I'd learn how to build LLMs from scratch

#210

Post by DonsDiscountGas »

embedding-shape wrote: post #35
koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> What's the alternative really, seek advice from unsuccessful people?

Do both. Get advice from successful and unsuccessful people and take the diff

Post reply on HN