Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

191–200 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#191

Post by shep101 »

kenjackson wrote: post #153
joshuakcockrell wrote: post #123

Earlier quoted context omitted.

This is like saying, “teens shouldn’t learn how to make their own game engine because no one is hiring for that.” You’re missing the point. Understanding how Unity works fundamentally makes you a better Unity dev.

Except knowing how LLMs work don’t actually provide much understanding for using them. People don’t use LLMs the way we’ve built on most other tons or platforms. It’s more learning Unity hoping to be a better gamer.

this is just wrong idk why u came to that like obviously it does make u better at using them…

Re: I were 17, I'd learn how to build LLMs from scratch

#192

Post by hashmap »

i am a small fan of pg, nevertheless i find this to be an exceptionally good take and it is strange to me to see so much piling on to this one in particular here.

learning about llms is not useful so that you can make llms later, you want to learn about it so that you can work on next generation architectures. llms before long i imagine will be left in the dust by ebm / physics oriented models especially that can have an embodied understanding of the world. but a lot of things you learn about them are transferable by doing something like this

Re: I were 17, I'd learn how to build LLMs from scratch

#193

Post by teitoklien »

felixrieseberg wrote: post #43

I'll use this post as a shameless opportunity to tell more people about a little side project, I made: http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.

This is the coolest link I saw on HN the last 2 months. You rock man !

Thanks a ton for building this.

Re: I were 17, I'd learn how to build LLMs from scratch

#194

Post by Panoramix »

reacharavindh wrote: post #103
oersted wrote: post #19

Earlier quoted context omitted.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes. On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources envir…

To me the bottleneck is not even the compute, which is an issue for sure, but the data. All these large companies got their hands into petabytes of data, a lot of which of illegally acquired, but now they are large enough to pay the fines.

Re: I were 17, I'd learn how to build LLMs from scratch

#195

Post by zipy124 »

epolanski wrote: post #107
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

+1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. The only jobs that he found he was highly over qualified or paid very little. In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.

I mean it makes sense right? Anthropic for instance has like, a couple hundred staff in London with plans to expand to somewhere just shy of a thousand. There are far far more ML/Maths/CS/Stats PhD's than there are openings. Especially in London there is no shortage of suitable candidates given Cambridge/Oxford/Imperial/UCL are surrounding it. 2% of the UK population has a PhD alone...

Re: I were 17, I'd learn how to build LLMs from scratch

#198

Post by kevmo314 »

mike_hearn wrote: post #163
kevmo314 wrote: post #136

Earlier quoted context omitted.

I suspect most who worked at Google did not work on Google infrastructure before getting there.

No, but the skills Google needed (back then) were just normal programming and sysadmin skills scaled up. I had eight interviews that covered Linux sysadmin, programming, debugging, networking, maths and more. If Google hadn't wanted me there'd have been plenty of other companies who needed those skills. If you look at model training jobs a lot of the work at this point is creating RL gyms (normal programming work), b…

It does sound quite hard if that’s the mentality you’re approaching it with.

Re: I were 17, I'd learn how to build LLMs from scratch

#199

Post by mike_hearn »

kevmo314 wrote: post #198
mike_hearn wrote: post #163

Earlier quoted context omitted.

No, but the skills Google needed (back then) were just normal programming and sysadmin skills scaled up. I had eight interviews that covered Linux sysadmin, programming, debugging, networking, maths and more. If Google hadn't wanted me there'd have been plenty of other companies who needed those skills. If you look at model training jobs a lot of the work at this point is creating RL gyms (normal programming work), b…

It does sound quite hard if that’s the mentality you’re approaching it with.

You can do it at home, for sure. I've done some NanoGPT training runs and modified the architecture, it wasn't that hard. Came up with some potential research ideas too. It does take money for GPU rental so for a 17 year old, it's not so easy unless their parents give them a budget. For an adult with income you can do it.

The question is more one of opportunity cost. At 17 you need to start finding your way in the world. It's best to learn skills lots of people need.

Re: I were 17, I'd learn how to build LLMs from scratch

#200

Post by hnlmorg »

mike_hearn wrote: post #184
amelius wrote: post #174

Earlier quoted context omitted.

Well, there's a lot more to learn from the former than the latter ...

Really? The latter was immediately useful to lots of people which is motivating, and it had a nice smooth learning curve (html -> js -> php -> databases -> apps -> backend). Learning HTML is the first step to learning how to make full blown apps. Making a browser at 17 is like trying to climb Everest as your first hike. The expected outcome is burnout and demotivating failure. At best you'll learn some C++ or Rust. 1…

Early web didn’t have JS. Nor PHP. In fact a lot of early web pages were written using static HTML with C++ invoked via CGI/bin for processing form data. So writing a browser would teach you the HTML plus C++ too.

The early Internet (which the GP mentioned) didn’t even have the web. But that’s nitpicking.

Post reply on HN