Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

341–350 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#341

Post by Der_Einzige »

maciejzj wrote: post #226
Chance-Device wrote: post #206

I think a problem a lot of people are grappling with here is that due to LLMs and AI generally, it’s basically impossible to predict what the future will look like or what jobs will still be around. I’d probably say something like: do something you enjoy and seems like it might be useful, but accept that the pace of change may mean that whatever you study ends up being irrelevant. Whatever solution there ends up bein…

I would advise any somewhat ambitious 17 year old to avoid tech and get into healthcare if they can stomach human interactions and bodily fluids. Sure, it is not all sunshine and rainbows, but there will still be plenty of work helping people who are ill or elderly. Even in the worst-case economic scenario, medicine will be a more socially rewarding and stable life path.

The real reason to recommend this is that they have an excellent chance of avoiding the "nerd-to-incel" pipeline that tech all-but-guarantees for its best nerds. Post GenAI boom, the social cost of working in tech combined with the coming collapse in high paying jobs, means that unironically coders should be learning a bit about coal mines.

In this regard, Healthcare is a polar opposite. It's pretty hard as a male nurse to not "accidentally" become a home wrecker.

Re: I were 17, I'd learn how to build LLMs from scratch

#342

Post by geremiiah »

I'm usually a fan of pg, but this post is ignorant of modern AI technologies. Building an LLM from scratch is both a trivial and a useless exercise. There's probably in the range of 5000 github repos doing exactly that. What makes LLMs work is scale, and what makes engineering and training LLMs hard is also scale. And scale is not something you can achieve in your garage.

If the goal is to understand LLMs deeply, one would be better served by either joining one of the big AI companies or doing a PhD. And to be honest, I think this journey should have been started 5 years ago, because right now there's too much competition.

Re: I were 17, I'd learn how to build LLMs from scratch

#343

Post by criddell »

shell0x wrote: post #222

What would you do if you’re 30 years old now? As a platform engineer being based mainly out of Australia/Hong Kong, opportunities seem to be getting less unless targeting high frequency trading or banking. It seems like building a startup with the help of some AI tools might be the best bet.

> It seems like building a startup with the help of some AI tools might be the best bet.

I'd only recommend taking that path if you already have your first paying customers standing by or are extraordinarily good at marketing.

Re: I were 17, I'd learn how to build LLMs from scratch

#344

Post by paulryanrogers »

virtualritz wrote: post #246
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

AI is the subtrate the future runs on. And so I think the idea is more to understand tomorrow ... from first principles. In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me. I haven't written assembly in years. But whatever I did in my career: it helped me and g…

How did that edge manifest?

Re: I were 17, I'd learn how to build LLMs from scratch

#345

Post by jimnotgym »

If I was 17 again I would sack off work and study and focus on chasing the opposite sex, without the angst I had at the time. I don't know a single person who regrets having had too much sex when they were young. I would not build an llm, too hard, too expensive to run. Learn how easy life is if your morals allow you to grift money of vcs into your own funds and retire.

Re: I were 17, I'd learn how to build LLMs from scratch

#346

Post by fifilura »

paulryanrogers wrote: post #333
enos_feedler wrote: post #293

Earlier quoted context omitted.

For this example, if you built a browser in 2000 it would put you in a great position to launch android in 2008. I think this is pg’s point. You spend all this time learning the interesting bits and when an inflection comes along that makes a new product or service possible you could be the one to likely launch it

Each new bet will be expensive and depend less on technical merit than luck and having a backup so your family doesn't starve.

Agree.

FWIW I built browsers from 1999 for a long time. (But I was never a wunderkind, just somewhat tenaciously curious).

And I guess I am doing fine, but not amazingly rich or so.

Browsers were always a project closer to research/charity. I think Marc Andreessen said something similar - that he would never do that again. B2B is where you can make money.

Re: I were 17, I'd learn how to build LLMs from scratch

#347

Post by khalic »

Grombobulous wrote: post #325
khalic wrote: post #315

Earlier quoted context omitted.

World models will definitely incorporate LLMs or at least text decoders in their final architecture, LLMs are here to stay.

Maybe this is a dumb analogy but is that kind of like saying “well EVs will still use lubricant! Cars that use oil are here to stay!”? The point being that the way we heavily rely on LLMs specifically right now is likely to cease to be the case, and that in the future they’ll be almost like an irrelevant component.

There's far more shared concepts between them than you think.

You should check the code of both world models (JEPA class for example) and compare to GPT. Many of the tricks stay the same, representation is still embeddings, there is a loss function, etc.

The exact architecture will change, but unless there's a new discovery in that area, we've cracked the text component already. We're hitting the limits of LLMs because of the intrinsic limits of text as a medium. But the way we work with text is pretty much settled, fundamentally.

Re: I were 17, I'd learn how to build LLMs from scratch

#348

Post by inciampati »

mike_hearn wrote: post #151
stymaar wrote: post #119

Earlier quoted context omitted.

> . But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters. Due to how much high quality data is readily available, anyone can now trai…

Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs. I saw my first language model in action in 20…

> We're not seeing them differentiate from each other, implying that the design space might not be that large.

There is the more likely reason they are not differentiating. They use almost exactly the same class of model. Everything is linear, parallelizable. It's incredible path dependence that's now invisible enough we think it's a natural law. Nature is not linear.

Re: I were 17, I'd learn how to build LLMs from scratch

#349

Post by sho_hn »

mike_hearn wrote: post #311
sho_hn wrote: post #220

Earlier quoted context omitted.

Anecdotally, I think it's great advice. I contributed to a browser engine around that age (KHTML, which later became WebKit and Blink), and while I don't work in browsers right now, much of that knowledge, mindset and of course the professional network have done much to shape my life. And a fairly successful career, for that matter.

Contributing to an open source project is fine, but the original analogy was "it's like telling teenagers to build browsers". If teenagers could make small contributions to LLMs via open source then sure, go for it. Optimizing llama.cpp or similar would be a good learning project that might later get you good work via social networks. Contributing to open source is how I got started too. Unfortunately, training LLMs…

Excellent point, and I agree. Much of the benefit I saw was from working within a like-minded, smart team, not going it alone. And also specifically working on software with a real user audience to learn what providing value to them actually constitutes.

In that sense it's more a "seek out the open source community and real projects when young" rather than "do web browsers", with a bit of "look for ambitious types of projects few get to work on".

Re: I were 17, I'd learn how to build LLMs from scratch

#350

Post by criddell »

DarmokTanagra wrote: post #263

telling a 17 year old to get into tech right now is horrible advice, literally telling them to get at the back of a line with a better part of a million more experienced people in it.

What else can he say? His business depends on that line continuing to exist.
Post reply on HN