Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

301–310 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#301
post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

> With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.

Paul, I think, is talking about achieving outsized outcomes in relatively shorter timeframes (as the timing is just right to be investing in learning this tech) for high agency folks who can also afford the ordeal in wanting to maximize for impact & ambition. Of course, there's real risk one may get no where, but even in failure, given you were building the LLM yourself, you might end up with other adjacent, high reward opportunities.

Re: I were 17, I'd learn how to build LLMs from scratch

#302

Earlier quoted context omitted.

> I am kind of amazed how negative the comments are here, especially on HN. I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content. Hazarding a guess, I think the moment Altman became the CEO and later durin…

Imo it's a breakdown of trust of the startup ecosystem as a whole. Repeatedly startups have enshittified and it's become undeniable that the investment apparatus around startups is partly responsible. We have seen a great driver of uncreative destruction, industries undermined, small businesses undermined just to drive masses of money into few pockets - less fairness for the people working in what is now the gig econ…

[deleted]

Re: I were 17, I'd learn how to build LLMs from scratch

#303

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions. But of course, 10 years ago this wasn't obvious.

you don't reach cutting edge immediately. you start with the basics

I think the answer then is to get a Phd? I mean, one should learn it to satisfy their curiosity and to build things but I don't think it's going to help you career in any significant way.

Re: I were 17, I'd learn how to build LLMs from scratch

#304
post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I think a lot of this is based on preconceptions. A lot of apps were made with Electron, because it was common wisdom that native is 'too hard'. Now with LLMs, people write native apps in Rust, and I'd like to think some of them found that there isn't such a huge jump in difficulty they assumed there would be.

Oh yes I agree, LLMs are not that complex in principle, most engineers could build a toy version completely from scratch without too much difficulty.

But that’s the tip of the iceberg. If you have any ambitions of doing this professionally, it quickly becomes clear that all it’s all about knowing how to deal with problems that are only present at massive scale, when an LLM is actually L and becomes AI.

The mundane details about how to build a tiny autocomplete model and the maths behind it you can learn in a couple weeks easily. It’s not black magic, there are much harder areas of computers science.

Re: I were 17, I'd learn how to build LLMs from scratch

#306

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

> understanding the bare-metal firmware for a computer IMO this is still relevant, everything surrounding the LLMs needs such a vast infrastructure that I don't know if I would find it more useful to learn the maths behind ML than CS

As a 17 year old, I agree with this. Ofc I'm against all the hate directed at PG, I believe that all knowledge has value regardless of its economic utility, but I understand where the hate is coming from. Personally, I find LLMs boring for now, and I'm more focused on CS and electrical engineering.

Re: I were 17, I'd learn how to build LLMs from scratch

#307
post #203

It’s really disheartening to see how many people don’t know shit about LLMs, by reading the comments… ironic given what OP is trying to say

Why is it disheartening? LLMs are a dead-end technology with respect to AI.

Have you been living under a rock? We might not get "super-intelligences" (if such a thing even exists) from LLMs but they've proven incredibly useful in basically anything relating to text, including code.

Re: I were 17, I'd learn how to build LLMs from scratch

#309
I genuinely don't think telling young people to do anything tech-related is good career advice. We don't even know if entry level roles will ever come back. The situation couldn't be worse for these roles. PG thinks that some random teenager will build a startup and get rich from it, or some shit. Like get real, man. Completely out of touch, tech bro who hasn't worked a real tech job for the last like 20 years... Remind me how many startups succeed again, Paul? What about the market dynamics for LLMs and how one might "secure compute"?

Re: I were 17, I'd learn how to build LLMs from scratch

#310

Earlier quoted context omitted.

I agree with mostly all of this, but personally I wrote a toy LLM almost 5 years ago and while it never saw much use outside of boring my wife with a shitty command line demo with glee it did help me understand how they worked and how to apply them, played a lot with JAX and pytorch, ended up building a ghetto version of MCP and an LLM-Pool to proxy requests to my baby local models and so I didn't struggle to see the…

Game cheating and reverse engineering MMO backends taught me a lot: databases, networking, securing a backend (and frontend), limitations of simpler languages when comparing them to more native options for building backends.

This was my introduction, too, but with Counter-Strike cheats.
Post reply on HN