Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

311–320 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#311

Post by mike_hearn »

sho_hn wrote: post #220
mike_hearn wrote: post #184

Earlier quoted context omitted.

Really? The latter was immediately useful to lots of people which is motivating, and it had a nice smooth learning curve (html -> js -> php -> databases -> apps -> backend). Learning HTML is the first step to learning how to make full blown apps. Making a browser at 17 is like trying to climb Everest as your first hike. The expected outcome is burnout and demotivating failure. At best you'll learn some C++ or Rust. 1…

Anecdotally, I think it's great advice. I contributed to a browser engine around that age (KHTML, which later became WebKit and Blink), and while I don't work in browsers right now, much of that knowledge, mindset and of course the professional network have done much to shape my life. And a fairly successful career, for that matter.

Contributing to an open source project is fine, but the original analogy was "it's like telling teenagers to build browsers".

If teenagers could make small contributions to LLMs via open source then sure, go for it. Optimizing llama.cpp or similar would be a good learning project that might later get you good work via social networks. Contributing to open source is how I got started too.

Unfortunately, training LLMs isn't something that fits well to open source open collaboration. Inferencing codebases are better.

Re: I were 17, I'd learn how to build LLMs from scratch

#312

Post by rfgplk »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

> The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.

And the job postings are often ridiculous. I recently was an AMD job advert in Germany for an ML Kernel Engineer, not Senior mind you. The requirements went something like

> Masters Degree required with strong preference for a PhD with peer reviewed articles in {journals_list} > 10+ years of experience in C/C++ > GPU programming experience required > 10 more ridiculous lines

No idea how a teenager self teaching himself LLMs is supposed to even get a shot...

Re: I were 17, I'd learn how to build LLMs from scratch

#313

Post by Grombobulous »

khalic wrote: post #307
dosisking wrote: post #281

Earlier quoted context omitted.

Why is it disheartening? LLMs are a dead-end technology with respect to AI.

Have you been living under a rock? We might not get "super-intelligences" (if such a thing even exists) from LLMs but they've proven incredibly useful in basically anything relating to text, including code.

The point is that they’re going to be surpassed by a much better technology that doesn’t have the inherent and unavoidable downsides of LLMs.

Like, right now especially in the US you’d probably sound crazy saying that one day nobody’s going to buy a gasoline car, and that EVs are going to completely replace them. But that’s basically an inevitability based on the direction we know technology is going to go, it’s only a matter of when.

Re: I were 17, I'd learn how to build LLMs from scratch

#314

Post by bigDinosaur »

InsideOutSanta wrote: post #40
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> seek advice from unsuccessful people Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.

If there's a single unifying fault to the unsuccessful people I know (for any definition of unsuccesful, including my own failures) is that they're precisely bad at working out why they failed - if they even think to ask the question. The successful people are generally much clearer that it was studying hard or networking or natural talent that contributed to their success. The latter group may have people who don't realise luck played a part, but they massively increased their chances of good luck by the aforementioned behaviours.

Re: I were 17, I'd learn how to build LLMs from scratch

#315

Post by khalic »

Grombobulous wrote: post #313
khalic wrote: post #307

Earlier quoted context omitted.

Have you been living under a rock? We might not get "super-intelligences" (if such a thing even exists) from LLMs but they've proven incredibly useful in basically anything relating to text, including code.

The point is that they’re going to be surpassed by a much better technology that doesn’t have the inherent and unavoidable downsides of LLMs. Like, right now especially in the US you’d probably sound crazy saying that one day nobody’s going to buy a gasoline car, and that EVs are going to completely replace them. But that’s basically an inevitability based on the direction we know technology is going to go, it’s only…

World models will definitely incorporate LLMs or at least text decoders in their final architecture, LLMs are here to stay.

Re: I were 17, I'd learn how to build LLMs from scratch

#316

Post by doginasuit »

chasd00 wrote: post #296

It’s probably wise to learn how to build one to understand what you’re dealing with. However, if I were 17 I would lean how to apply an LLM to a problem instead of strictly building one.

Why not both? The best thing about LLMs is they make virtually any domain approachable.

Re: I were 17, I'd learn how to build LLMs from scratch

#317

Post by Grombobulous »

DarmokTanagra wrote: post #263

telling a 17 year old to get into tech right now is horrible advice, literally telling them to get at the back of a line with a better part of a million more experienced people in it.

Telling them not to get into tech is also terribly reactionary advice. The truth of the matter is that we don’t yet know whether tech roles will be eliminated or if they’re just going to follow previous innovation breakthroughs where “one person producing way more work” makes software even more of a desirable industry to be involved in.

There really isn’t a very strong correlation between tech industry hiring strength and AI as of yet. Various studies that are out there haven’t even witnessed AI workflows contributing more than modest gains in software engineering efficiency. I.e., being able to write code 20-40% faster isn’t a seismic shift in the industry where everyone is getting laid off tomorrow and we’re all replaced by software.

Even with the questions surrounding the current job market, it’s still an incredibly good ROI career compared to so many other jobs out there.

For example, in my local area you can get a job as a registered nurse working nights in the emergency room and only make ~$115k.

I make almost double that telling an LLM what to do from my house in my pajamas during the day with less time spent in university.

Even if tech roles lose half their salary to automation pressure it’s still a really good gig.

Re: I were 17, I'd learn how to build LLMs from scratch

#318

Post by IshKebab »

otherme123 wrote: post #95
IshKebab wrote: post #41

Earlier quoted context omitted.

Seek advice from people who have had a normal level of success. Not a one-in-a-million level.

Veritasium has a good piece for this kind of bias: https://www.youtube.com/watch?v=3LopI4YeC4I An advantage that is not "advisable", like being born in january, in a rich country, in an above average family, or just having luck, might have more influence on the outcome than any conscious action. It is almost sure that one-in-a-million level people only edge over the other 999,999 they competed with is just "have more…

It's not the only edge. Wildly successful people are usually lucky and talented/hard working.

Re: I were 17, I'd learn how to build LLMs from scratch

#319

Post by stymaar »

mike_hearn wrote: post #151
stymaar wrote: post #119

Earlier quoted context omitted.

> . But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters. Due to how much high quality data is readily available, anyone can now trai…

Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs. I saw my first language model in action in 20…

> They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also.

Good luck relying on in-context learning for a 600M LLM.

> The actual adapter training is automated and put behind simple APIs.

That's like saying it's worthless to learn infra because you can use serverless instead…

> All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large

The design space for a generalist model isn't large, by definition. But the design space for specialized smaller models is much larger. If you can train a 200M model that, for your use-case, is competitive with a frontier one, then you'll make your company save a lot of money in tokens.

> 2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies.

We are in a strange place where a few companies are collectively burning a hundreds of billions a year to sell things a few pennies for the dollar. Of course it's going to be cheap and concentrated. How is it supposed to end though?

> 3. It's unclear how much demand for better models there actually is. Do we actually need smarter models?

That's the thing actually: I don't think we need better models this much, and if we don't need better models we need the cheapest possible model for a given use-case.

Re: I were 17, I'd learn how to build LLMs from scratch

#320

Post by dofm »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

> A lot of people here are responding to the message but not to the meaning.

Well it is framed as quite specific advice.

(I'm done with mining PG tweets for meaning)

Post reply on HN