Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

121–130 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#121

Post by michaelscott »

embedding-shape wrote: post #114
epolanski wrote: post #107

Earlier quoted context omitted.

+1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. The only jobs that he found he was highly over qualified or paid very little. In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.

> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more…

How've you gone about getting into this btw? I have extensive experience in infra and pipeline rollout but have struggled to find freelance clients for this kind of thing. Would be great to tie it into ML as a learning opportunity there

Re: I were 17, I'd learn how to build LLMs from scratch

#122

Post by ballooney »

embedding-shape wrote: post #114
epolanski wrote: post #107

Earlier quoted context omitted.

+1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. The only jobs that he found he was highly over qualified or paid very little. In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.

> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more…

This only proves the original point which is that there is not much demand for actual machine learning expertise because that is only carried out in a small number of places and what demands there is is for the more basic software carpentry like infrastructure and operations rather than the actual technology and Engineering side of things

Re: I were 17, I'd learn how to build LLMs from scratch

#123

Post by joshuakcockrell »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

This is like saying, “teens shouldn’t learn how to make their own game engine because no one is hiring for that.”

You’re missing the point. Understanding how Unity works fundamentally makes you a better Unity dev.

Re: I were 17, I'd learn how to build LLMs from scratch

#124

Post by embedding-shape »

michaelscott wrote: post #121
embedding-shape wrote: post #114

Earlier quoted context omitted.

> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more…

How've you gone about getting into this btw? I have extensive experience in infra and pipeline rollout but have struggled to find freelance clients for this kind of thing. Would be great to tie it into ML as a learning opportunity there

Spent a year of freetime catching up on everything and learning as much as possible, started sharing what I've found works or not, write a bunch of comments on HN and elsewhere, and have a email in your profile, eventually people will find you if you put out good stuff :)

Also bunch of past workplaces who've adopted AI in various ways who reach out once they find out what my current focus lies, but that's harder for others to replicate unless you've already had a career as a developer.

Re: I were 17, I'd learn how to build LLMs from scratch

#125

Post by aero-glide2 »

loveparade wrote: post #77

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

you don't reach cutting edge immediately. you start with the basics

Re: I were 17, I'd learn how to build LLMs from scratch

#126

Post by embedding-shape »

ballooney wrote: post #122
embedding-shape wrote: post #114

Earlier quoted context omitted.

> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position. Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more…

This only proves the original point which is that there is not much demand for actual machine learning expertise because that is only carried out in a small number of places and what demands there is is for the more basic software carpentry like infrastructure and operations rather than the actual technology and Engineering side of things

What parent says about "there are very few available positions" for "engineers that can do real LLM machine-learning" is fair, yeah, I'd agree with this.

I don't think the "incredibly small minority of companies in the world do any real training or optimisation" part is necessarily as true, as some parts of the work I do get is about helping them optimize training and infrastructure around training. Mind you, none of this is for building LLMs from scratch, it's 99% fine-tuning existing checkpoints.

I'd also agree with "paulg is in somewhat of a bubble" regardless of this, which is worth remembering whenever you read his content. Same goes for any person living in SF, and dare I say the US. But also, YMMV, I live and work in Europe, probably why I have this perspective.

Re: I were 17, I'd learn how to build LLMs from scratch

#127

Post by latexr »

embedding-shape wrote: post #32
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

> I am kind of amazed how negative the comments are here, especially on HN. I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hazarding a guess, I think the moment Altman became the CEO and later durin…

> I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.

Hard disagree. This submission is still being highly upvoted, while another recent post[1] on the harms caused by Graham’s fellows[2], with a fairly tame comment section, has been flagged. That is a constant on HN. It’s not a fluke, it’s as predictable as the sunrise and getting more pronounced.

I’m sure we’re both biased in our perceptions. Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote, together with others like Musk, until things started to really go to shit and many eyes have been opened to the effects of the unfettered greed of rich tech guys out of touch with reality.[4]

[1]: https://news.ycombinator.com/item?id=49411762

[2]: A better English word is escaping me.

[3]: That word I choose hyperbolically but deliberately. It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.

[4]: That is not “knee-jerk” but a slow realisation still ongoing.

Re: I were 17, I'd learn how to build LLMs from scratch

#128

Post by BoredomIsFun »

embedding-shape wrote: post #106
BoredomIsFun wrote: post #93

Earlier quoted context omitted.

> Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO). This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights ( https://en.wikipedia.org/wiki/Discounted_utility ). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and…

> Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler. To you it isn't, is my entire point here. But why extrapolate what you think is fun, to others? Sure, I don't find that fun either (although useful), but who am I to say it isn't fun for others ?

We can continue this pointless conversation, in the tone "who you are to tell what is fun to others and whst is not". You'd be impervious to any argument stating that dealing with far beyound someone understanding and requiring countless hours of digging into difficult math is not fun even for those who thinks it should be fun, as they presumably, loves everything STEM.

Re: I were 17, I'd learn how to build LLMs from scratch

#130

Post by mdp2021 »

BoredomIsFun wrote: post #68
hresvelgr wrote: post #51

Earlier quoted context omitted.

> I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.

That'd would be a terrible advice if there weren't a plenty of other things "you are not equipped for", but far less daunting both theoretically and practically. Such as, say, convolutional neural networks, or some older ML tech. Or even something totally unrelated to ML. Transformers are difficult to understand even to people with strong ML background, let alone a teenager.

You are assuming the 17yo in question as an untrained underdeveloped savage. If I were 17 in 2026, I would certainly have exploited all the availabilities from 2010 on - including YouTube, OpenCourseware, the Web simply (Sebastian Raschka etc.) and LLMs.

That 17yo would have already built many uncommon bases, and would build further.

That is, a 17yo with proper mentality.

Post reply on HN