Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

541–550 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#541

Post by bluecheese452 »

ikety wrote: post #422
beemboy wrote: post #373

Earlier quoted context omitted.

Yes and no. I believe the point he is making is simply that there is no substitute for fundamentals and first-principles thinking. We had scores of students study how microprocessors work and compilers work over decades, yet we have 3 or 4 major processor companies and a handful of programming languages. Yet, what they learned was probably crucial in their development as engineers. We are also so early right now that…

This is kind of different though isn't it? Doing an assembly or compiler class has pretty clear benefits in this regard. But LLMs are tools. Does a great engineer need to know how vscode works? Might be helpful to understand how extensions work, LSPs, and project configurations. Usually when working with any tools, you need to understand how to get the most out of your tool for your needs and that's about it. Core fu…

Aren’t llms tools in the same way compilers are tools?

Re: I were 17, I'd learn how to build LLMs from scratch

#542

Post by kashnote »

Don't really understand why people are saying this is terrible advice. If you're 17, you should be learning everything. At that age, your brain is a sponge and your energy levels are the highest they will ever be.

None of us know what the future of work, education, or AI is going to look like. But your best bet is to become a life-long learner. Be it LLMs, musical instruments, physics, or business.

Re: I were 17, I'd learn how to build LLMs from scratch

#545

Post by bluecheese452 »

armcat wrote: post #46
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you, and then take advice from people that are amazing in that specific field. In mid 2000s in Australia all the "top people" were telling me not to get into a software engineering career because it was dead. It's certainly challenged right now, but it took off during those 15+ years.

A lot of the advice that is relevant for top performers in a field has no relevance to an average person interested in that field.

Re: I were 17, I'd learn how to build LLMs from scratch

#546

Post by andy_ppp »

hdz wrote: post #534
andy_ppp wrote: post #476

Earlier quoted context omitted.

What provable conclusions have you intuited around how LLMs work? Give me some examples to prove your point? I'm absolutely happy to change my view with enough data.

I'm not looking to change your view. But for other readers who are curious, here is a link to an interesting task to gain intuition. Ahmad is a good data point for someone who tinkered, built intuition, then started his own ai company. https://twitter.com/TheAhmadOsman/status/2087742080793620593...

I think we are talking about different things to be honest. Understanding LLMs is a fine thing to do but to say you understand what the network is actually doing, even in toy examples like MNIST is extremely tricky. Figuring out what a ~3 trillion parameter network is doing in thousands of dimensions seems intractable to me.

Linking to a Twitter thread about training a small LLM to be good at Wordle is not an example of what I'm talking about. It might well be a useful task but it doesn't allow us to understand deeply what's happening.

Re: I were 17, I'd learn how to build LLMs from scratch

#547

Post by InsideOutSanta »

sejje wrote: post #522
InsideOutSanta wrote: post #152

Earlier quoted context omitted.

I think it's reasonably common for people to understand what makes them unsuccessful but be unable or unwilling to resolve the problem. For me, I know that one major reason for putting an upper limit on my success is my inability to form effective professional relationships. I know I should go to events and talk to people and use these relationships to my advantage, but that's just not something I've ever been good a…

How do you know that's limiting your success? Do you think the most successful people got that way by attending events? I thought they were hacking in their garages, building great products.

> How do you know that's limiting your success?

I know there were opportunities I could have benefited from if I'd had better relationships with specific people.

> Do you think the most successful people got that way by attending events?

That's not what I said.

Re: I were 17, I'd learn how to build LLMs from scratch

#549

Post by mips_avatar »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

A single 3090 will train qwen 0.8B just fine. While it’s not a very capable model any training technique you would want to master can be used to make real progress. And all the skills you need to learn how to do this can be learned watching Andrej Karpathy’s zero to hero series (shame he quit educational content and went to anthropic)

Re: I were 17, I'd learn how to build LLMs from scratch

#550

Post by tech_ken »

megaloblasto wrote: post #449
andy_ppp wrote: post #419

Earlier quoted context omitted.

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I disagree. It's absolutely possible to develop an intuition about extremely complex mathematical ideas, including llms or high dimensional systems. Learning to build an llm is a great way to start building that intuition.

Developing an intuition about high dimensional systems is pretty different from understanding the character of some specific point on a 1e9+ dimensional manifold of parameters, in my professional opinion (setting aside all the degrees of freedom that come from the structure of the thing). Sure one can understand generic principles like the curse of dimensionality, but truly groking how an LLM works is basically an open problem as far as I'm aware. I'm not saying there's no benefit for amateurs to study how LLMs work, but let's be realistic about how far mere intuition can truly take anyone in this space.
Post reply on HN