Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

511–520 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#511

Post by glitchc »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

Of course, because it's not the LLM that's special but the training data. Nowadays, your favourite AI service to generate code for an LLM whenever you ask for it.

Re: I were 17, I'd learn how to build LLMs from scratch

#512

Post by qarl2 »

qarl2 wrote: post #365

I remember when I caught PG on reddit arguing with some guy who'd said something mean about him. He didn't reveal who he was. But you could tell from his history - his first post was from before reddit opened to the public. Good times. I wonder if you can dig that out of the historical reddit database. I'd like to see that again. I love how everything is recorded now.

> I love how everything is recorded now.

I mean, except when it's censored. That part is a shame. HEY! Is there a tool that tracks the censored comments? I bet there is. And if there's not... that will be a delicious new project for my agents.

Re: I were 17, I'd learn how to build LLMs from scratch

#513

Post by psadri »

Paul G is not writing this for a general audience of your run of the mill “engineer” hoping to be employed by someone. He is writing it for future founders. What knowledge / skills you need to develop today to be well positioned to have a startup worthy insight when you are 24.

Re: I were 17, I'd learn how to build LLMs from scratch

#514

Post by nixonaddiction »

i was trying to build llms from scratch at 17. failed miserably because i did not know linear algebra. ended up in a different but adjacent field. when chatgtp got big suddenly there were so many people doing llms and i didnt want to compete like that. so much of what was happening was hype, and that really turned me off. am back to building llms from scratch, but like, its a journey teaching myself all the theory on top of my job. i have decent fundamentals, but i need a better grasp of all the advancements in the field in the past 5 years before i would feel comfortable designing anything. baby steps, essentially. am working on better understanding all the layers of an ai while implementing a rag on my local model as an experiment. 17 year olds should learn whatever theyre interested in but need fundamentals in order to do anything advanced.

Re: I were 17, I'd learn how to build LLMs from scratch

#515

Post by paulryanrogers »

endemic wrote: post #413
paulryanrogers wrote: post #392

Earlier quoted context omitted.

Ken Silverman

I think he's doing fine? Sure, he got out of the video game industry, but that's for young people to burn themselves out.

I don't mean he's destitute. Just that he's no longer exceptional, which is fine.

Re: I were 17, I'd learn how to build LLMs from scratch

#516

Post by alexgartrell »

rdedev wrote: post #235
kalms wrote: post #87

Earlier quoted context omitted.

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful. The above is, after all, the wh…

How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasona…

https://thinkingmachines.ai/tinker/ https://github.com/thinking-machines-lab/tinker-cookbook

(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").

TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.

One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.

There are a lot of other high level abstractions here to look at. Prime Intellect has one.

The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.

Re: I were 17, I'd learn how to build LLMs from scratch

#517

Post by Izkata »

storywatch wrote: post #484
Aissen wrote: post #431

Earlier quoted context omitted.

I was there at the start of the smartphone boom. I built a demo Android device that was capable of telephony/data, 3D rendering, etc. all before Google open-sourced the OS, for a SoC vendor that wasn't in Google's inner circle. Yet the industry was not interested in my junior profile during the subprime crisis.

I wonder if the modern incarnation of Android flagships would have taken off at all if not for Apple. When Android first showed up, the dominant "smartphones" were running Symbian/Windows Mobile and they had a very different view of the world when it comes to UX and hardware.

Phones were already going in that direction before the iPhone was released: https://phys.org/news/2007-02-samsung-unveils-revolutionary-...

But it was generally seen as a gimmick instead of desired before Apple made it look good. Even when the iPhone came out, one of the jokes was how the grid of icons looks like how a Windows user's desktop would look like when they didn't understand the filesystem.

Re: I were 17, I'd learn how to build LLMs from scratch

#518

Post by oersted »

glitchc wrote: post #511
oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

Of course, because it's not the LLM that's special but the training data. Nowadays, your favourite AI service to generate code for an LLM whenever you ask for it.

I don’t think that’s correct, the data is not that special either, and getting a similar dataset is significantly easier than getting the compute capacity to use it, even if they are both relatively hard.

Probably this also is too cynical and simplistic, but: really what’s special is the ability to get this kind of capital, with the freedom to burn it on mad moonshots, with long enough leeway to actually get to see a few of the moonshots come true.

No wonder that the head of YC made this happen, this is exactly what they are world-leading at.

Re: I were 17, I'd learn how to build LLMs from scratch

#519

Post by fuzzfactor »

rrr_oh_man wrote: post #135
reacharavindh wrote: post #103

Earlier quoted context omitted.

> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes. On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources envir…

> On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have…. 10000%.

One of the things to think about when it comes to many kinds of "expensive" technology, is that from so many well-funded ventures, from the capitalists on down almost every decision-maker involved has not spent the majority of their life making every dollar count in some way or another.

Even more so when things are not just expensive by nature, but truly overpriced beyond that point.

>10000%

Once in a while you do get somebody who only spends a dollar and gets more out of it than a seasoned high-roller spending $10000. Most of the time the waste is borne by those who can afford to throw away $10000 more easily than an economizer can afford to lose one dollar, so nobody is crying about it.

With how ridiculously large the language models have gotten though, a 10000x improvement in actual intelligence does seem like it could be lurking unrecognized at a different point on the compass.

Re: I were 17, I'd learn how to build LLMs from scratch

#520

Post by hnlmorg »

ryandrake wrote: post #455
bigfishrunning wrote: post #420

Earlier quoted context omitted.

Are you asserting that C++ is not a marketable skill? If anyone reading this has this un-marketable skill, Carnegie robotics in Pittsburgh is hiring carnegie-robotics.breezy.hr/p/2d85f5321cc7-software-engineer

I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you…

CRUD didn’t exist in the early days of the web.

People keep talking about a browser in the modern context but the GP specifically said “early days of the internet” (which, in fairness, would mean pre-web. But I think it’s safe to assume they meant “web” not Internet).

In those days, it was actually a much simpler exercise to write a browser than it is today. I even wrote one! And writing a browser absolutely teaches you how HTTP and HTML worked. Plus a lot of backed development was forms data sent to CGI and thus written in languages we wouldn’t even dream of using for web development nowadays, including C++.

So in the early web, writing a browser absolutely was a transferable skill. It might not be now, but in the context defined by the GP, it was.

Post reply on HN