Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

101–110 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#101

Post by pjmlp »

willtemperley wrote: post #49
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues. 1. Both training and optimisation will get significantly cheaper and easier quickly. 2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029. 3. The big AI firms will become part of the surveillance capitalism network, i…

Most companies that build physical goods don't care for one second about their IT department other than how much money they can save per month, starting by outsourcing whole of it, thus they have little use for internal LLMs.

Re: I were 17, I'd learn how to build LLMs from scratch

#103

Post by reacharavindh »

oersted wrote: post #19
kevmo314 wrote: post #16

Earlier quoted context omitted.

That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.

You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market. Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to…

> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex.

I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes.

On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….

The world needs pragmatic folks who work at a higher abstraction and make LLMs useful, AND also folks who think why not “this other way”? And build newer ways to do fundamental things.

Given the usefulness of current LLMs, I would certainly encourage anybody to try and build their own LLMs, and see what they come up with…

Heck if they build a rack full of old laptops and run something with it that could be done “better” with modern servers, I’d still appreciate the learning running things on those little machines bring.

Re: I were 17, I'd learn how to build LLMs from scratch

#105

Post by Foobar8568 »

pjmlp wrote: post #101
willtemperley wrote: post #49

Earlier quoted context omitted.

I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues. 1. Both training and optimisation will get significantly cheaper and easier quickly. 2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029. 3. The big AI firms will become part of the surveillance capitalism network, i…

Most companies that build physical goods don't care for one second about their IT department other than how much money they can save per month, starting by outsourcing whole of it, thus they have little use for internal LLMs.

And it's across the industry, thinking banks, private banks, insurance, pharamcy etc don't outsource their IT, including development... I believe US outsource even more than Europe on this matter.

Re: I were 17, I'd learn how to build LLMs from scratch

#106

Post by embedding-shape »

BoredomIsFun wrote: post #93
embedding-shape wrote: post #76

Earlier quoted context omitted.

> I just voiced my opinion. Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X. > youngsters who'd be able to extract tremensous value from such an excercise But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding…

> Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO). This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights ( https://en.wikipedia.org/wiki/Discounted_utility ). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and…

> Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.

To you it isn't, is my entire point here. But why extrapolate what you think is fun, to others? Sure, I don't find that fun either (although useful), but who am I to say it isn't fun for others?

Re: I were 17, I'd learn how to build LLMs from scratch

#107

Post by epolanski »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

+1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position.

The only jobs that he found he was highly over qualified or paid very little.

In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.

Re: I were 17, I'd learn how to build LLMs from scratch

#108

Post by Foobar8568 »

rhdunn wrote: post #88
willtemperley wrote: post #49

Earlier quoted context omitted.

I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues. 1. Both training and optimisation will get significantly cheaper and easier quickly. 2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029. 3. The big AI firms will become part of the surveillance capitalism network, i…

The cost of training a model from scratch is going to be cost prohibitive for the vast majority of companies (even if renting the hardware needed for the 1-2 month training time). It's an interesting learning exercise, and some of the things learned can be applied to other parts of the process. There's also the issue of needing a huge amount of data needed to get decent weights. Fine-tuning a model or LoRA based on t…

Even for most organizations, testing AI systems is too cost prohibitive, so they YOLO in production, including public facing systems.

Re: I were 17, I'd learn how to build LLMs from scratch

#109

Post by molf »

BoredomIsFun wrote: post #54
molf wrote: post #34

Earlier quoted context omitted.

Why would you tell people that the correct order is to build foundational knowledge before exploring a subject? For some (many?) people, a 'proper' understanding develops _after_ the exploration.

> Why would you tell people that the correct order Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17? > For some (many?) people, a 'proper' understanding develops _after_ the exploration. I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be una…

> unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring"

That's what I would call exploring.

I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.

Post reply on HN