Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

61–70 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#61

Post by Tarq0n »

willtemperley wrote: post #49
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues. 1. Both training and optimisation will get significantly cheaper and easier quickly. 2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029. 3. The big AI firms will become part of the surveillance capitalism network, i…

Right, just like companies don't use SAAS.

In reality, enterprises are happy to offload even risky tasks to others as long as they get some contractual guarantees about their data. Would they like more choice in who to buy from? Yes, but not enough to in-house such a specific discipline.

Re: I were 17, I'd learn how to build LLMs from scratch

#63

Post by physicsguy »

spwa4 wrote: post #20
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

That's not true, because everyone, everyone, everyone seems to want to do training. Which results in a 50 person company training, say, a voice model that then fails, because it's just not good enough. In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% bette…

The big question is whether companies hold enough proprietary data to do useful things that for e.g. Anthropic, etc. can't easily replicate.

For some very niche cases I think this is probably the case but for the vast majority, the company's data isn't as useful as they think it is or anywhere near the size needed.

Re: I were 17, I'd learn how to build LLMs from scratch

#64

Post by BoredomIsFun »

embedding-shape wrote: post #22
BoredomIsFun wrote: post #17

I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc. I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into matur…

> I do not think it is a proper thing to do for 17 y.o If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this. Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is o…

I just voiced my opinion. I just think buiding an LLM from the scratch for 17 y.o. is pointless exercise, advising a teenager to do so is borderline irresponsible, and frankly PG is simply virtue signalling here, as LLMs are still trendy, esp. in his circles.

There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.

Re: I were 17, I'd learn how to build LLMs from scratch

#65

Post by globular-toast »

chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> With LLMs and online tutorials, this isn't even a difficult suggestion.

Don't many of the commercial ones prevent you from using them to build LLMs?

I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.

Re: I were 17, I'd learn how to build LLMs from scratch

#66

Post by muragekibicho »

livinglist wrote: post #55

When I was 17 I was building Windows Phone apps, bad decision on my part.

Incredible counterexample, but oddly relatable. I'd probably have achieved techbro 'post-economic' status earlier if I focused on Android dev instead of the shiny (and new at that time) Xamarin for Windows phones.

Re: I were 17, I'd learn how to build LLMs from scratch

#67

Post by retsibsi »

InsideOutSanta wrote: post #40
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

> seek advice from unsuccessful people Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.

I don't know; I think people in general are just not great at this. Successful people tend to underrate luck and overrate the brilliance of their own decisions, but the rest of us are prone to either reversing that and blaming everyone but ourselves, or being so determined to take accountability (or just depressed) that we become overly self-critical, or simply not understanding why things happened the way they did and reaching for any explanation that resolves the chaos into something narratively satisfying.

Re: I were 17, I'd learn how to build LLMs from scratch

#68

Post by BoredomIsFun »

hresvelgr wrote: post #51
BoredomIsFun wrote: post #17

I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc. I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into matur…

> I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.

That'd would be a terrible advice if there weren't a plenty of other things "you are not equipped for", but far less daunting both theoretically and practically. Such as, say, convolutional neural networks, or some older ML tech. Or even something totally unrelated to ML.

Transformers are difficult to understand even to people with strong ML background, let alone a teenager.

Re: I were 17, I'd learn how to build LLMs from scratch

#69

Post by teaearlgraycold »

kevmo314 wrote: post #16
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.

Interesting/capable diffusion models are much smaller than similarly interesting language models. But yes you could always scale things down to learn the fundamentals.

Re: I were 17, I'd learn how to build LLMs from scratch

#70

Post by BoredomIsFun »

charcircuit wrote: post #28
BoredomIsFun wrote: post #17

I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc. I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into matur…

>understanding modern OS and SDE tools for proper implementation of pipeline etc. Can you provide an example?

How would you filter out garbage from your training data, for example? If you are trying to use someone elses corpus, would it be "from the scratch" then?
Post reply on HN