Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

151–160 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#151

Post by mike_hearn »

stymaar wrote: post #119
oersted wrote: post #19

Earlier quoted context omitted.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

> . But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters. Due to how much high quality data is readily available, anyone can now trai…

Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs.

I saw my first language model in action in 2014, I was writing blog posts about them back in 2016. In recent years, like many of us, I spent some time learning ML frameworks to see if it'd be a fun career pivot.

But:

1. It doesn't seem especially creative. All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large. In which case the opinions and unique approaches of specific engineers aren't that important, they are interchangeable at the right level of skill, and what to do next is usually obvious to everyone.

2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies. It's a small community which greatly increases career risk, especially as so many are unprofitable and/or have strong ideological requirements.

3. It's unclear how much demand for better models there actually is. Do we actually need smarter models? In robotics clearly yes and robotics is interesting and high potential, but for pure LLMs/image models, most users are already incapable of setting tasks that stress the best models and are happy with the cheaper smaller ones.

Using the models on the other hand is a very large design space, and has a lot of scope for creativity. I see use cases for AI everywhere, but most companies seem to stop at putting a chatbot on their website or asking Copilot to rewrite an email before they send it. A lot of companies have hollowed out their IT departments over the past twenty years. It feels like a new golden age of consulting work could be upon us.

I really don't think I'd tell a 17 year old to learn how to train LLMs. Learn how they work and how to use them, sure, absolutely.

Re: I were 17, I'd learn how to build LLMs from scratch

#152

Post by InsideOutSanta »

embedding-shape wrote: post #44
InsideOutSanta wrote: post #40

Earlier quoted context omitted.

> seek advice from unsuccessful people Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.

Intuitively, that'd make sense if that unsuccessful person eventually found success, otherwise who knows if they actually picked up what made them unsuccessful in the first place?

I think it's reasonably common for people to understand what makes them unsuccessful but be unable or unwilling to resolve the problem.

For me, I know that one major reason for putting an upper limit on my success is my inability to form effective professional relationships. I know I should go to events and talk to people and use these relationships to my advantage, but that's just not something I've ever been good at, and I find it so incredibly unpleasant that I also just don't want to do it.

Re: I were 17, I'd learn how to build LLMs from scratch

#153

Post by kenjackson »

joshuakcockrell wrote: post #123
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

This is like saying, “teens shouldn’t learn how to make their own game engine because no one is hiring for that.” You’re missing the point. Understanding how Unity works fundamentally makes you a better Unity dev.

Except knowing how LLMs work don’t actually provide much understanding for using them. People don’t use LLMs the way we’ve built on most other tons or platforms. It’s more learning Unity hoping to be a better gamer.

Re: I were 17, I'd learn how to build LLMs from scratch

#154

Post by borski »

epolanski wrote: post #115
wiseowise wrote: post #110

[flagged]

Is going to university really that good of advice nowadays? Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker. edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went in…

If you’re talking purely financial benefit: yes, it is still good advice, as most white collar jobs still require a degree.

If you’re talking about a place to mature, around others who are at a similar phase of life, also yes.

It is where most people meet their cofounders, for example, even if they don’t found anything until much later.

Re: I were 17, I'd learn how to build LLMs from scratch

#155

Post by num42 »

If you’re 17, you might learn hands on knowledge, like tacit knowledge in areas such as lathes, precision engineering, metrology, and other very niche fields. You can also try climbing and explore arts like music, painting, and drawing. And, of course, spend some time in nature.

Re: I were 17, I'd learn how to build LLMs from scratch

#156

Post by mdp2021 »

hugodan wrote: post #129

hackers and painters and kids and ROI and startups and capitalism and the destruction of nature and old guys with money talking like they know better in fascist social networks

> fascist social networks

Sometimes if you want to be heard you must go to the public square, whatever the flags there.

The rest of the post is unintelligible: add some verbs at least.

Re: I were 17, I'd learn how to build LLMs from scratch

#157

Post by bodash »

koe123 wrote: post #31

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias). Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as…

IMO these kind of advices never matters. Any individual still needs to make tens or hundreds of little decisions (every day) themselves, and that's what really makes all the difference.

Re: I were 17, I'd learn how to build LLMs from scratch

#158

Post by keiferski »

I think pg answered the question as “what I’d do as a project” and not “what I’d do as a career.” So the critical comments are kind of missing the point, IMO.

I don’t see why learning how LLMs work is a bad project for a 17 year old.

Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy things.

And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.

When you’re 17-25, the timescale of a decade seems infinite. But it’s really not, and a decade spent “exploring and keeping your options open” sometimes just ends up with you being pretty decent but not amazing at a lot of random things.

Sometimes I wish I had just become a carpenter, chef, electrician, etc. – a specific skill set that leads to mastery over time, rather than the endless exciting-new-thing hamster wheel of working in tech.

Re: I were 17, I'd learn how to build LLMs from scratch

#159

Post by tripledry »

wy35 wrote: post #150
epolanski wrote: post #115

Earlier quoted context omitted.

Is going to university really that good of advice nowadays? Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker. edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went in…

It depends on the university and your goals, I suppose. Many people are surprised to learn how affordable elite colleges are if you genuinely need financial aid. I had no idea -- was pleasantly surprised when my alma mater took over 80% off of my tuition.

Indeed, would the advice be good if university was free?

Where I live it's basically free and I still sometimes regret not going into the trades. But I suspect this feeling might mostly be a "grass is greener" thing.

Re: I were 17, I'd learn how to build LLMs from scratch

#160

Post by epolanski »

borski wrote: post #154
epolanski wrote: post #115

Earlier quoted context omitted.

Is going to university really that good of advice nowadays? Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker. edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went in…

If you’re talking purely financial benefit: yes, it is still good advice, as most white collar jobs still require a degree. If you’re talking about a place to mature, around others who are at a similar phase of life, also yes. It is where most people meet their cofounders, for example, even if they don’t found anything until much later.

But it is not true financially, it really depends.

Plenty of blue collar workers make more than white collar ones, and have a huge debt free head start in life.

A plumber or electrician will make significantly more than the average bank employee or translator or nurse or teacher.

And they will also have an easier time starting a business as many trade workers are self employed and make much more than hired ones.

Post reply on HN