Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

231–240 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#231

Post by torginus »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I think a lot of this is based on preconceptions. A lot of apps were made with Electron, because it was common wisdom that native is 'too hard'.

Now with LLMs, people write native apps in Rust, and I'd like to think some of them found that there isn't such a huge jump in difficulty they assumed there would be.

Re: I were 17, I'd learn how to build LLMs from scratch

#232

Post by dofm »

I think this is weird advice.

Learn how to make language models from scratch, yes. But learn how to use them, in the context of other machine learning tools, on very small hardware.

When the bubble bursts (and I still tend towards thinking it could burst rather than be deflated in a manageable way), the focus will be on uses of AI that are not like the hyperscalars' products.

People will still be interested in useful AI being added to small things — assistive technologies, home security, garden monitoring, their phones and smartwatches, robotics.

Instead of reductive, reactive make-an-anthropic-competitor advice like this, what about advising 17 year olds to focus on broad, integrated, helpful AI — or on going back through eighty years of history to look at AI projects that failed and reassess them?

Re: I were 17, I'd learn how to build LLMs from scratch

#233

Post by graemep »

epolanski wrote: post #115
wiseowise wrote: post #110

[flagged]

Is going to university really that good of advice nowadays? Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker. edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went in…

Depends on where you live (lots of countries have no tuition fees or far lower than the US), what funding you have, how good a university you go it, what you want to do (some careers require a degree), whether you will enjoy it, and whether you are there just for financial benefits or more than that.

Its not good advice for everybody, but it is good advice for a lot of people. What if you want to be a doctor? What if you want to work in R & D? Not everyone enjoys working in a shop or a farm. Also, how old is your friend group? If they are mid twenties you are ignoring the greater scope for advancement in a lot of white collar careers.

> my 25 years old friend with her own shop knows more about real life, business and economy than ivy league MBAs.

Within the narrow limits relevant to her business. How much does she know about macro-economics or financial economics, or scaling up a business? I also suspect you are comparing her to people who went straight on from bachelors to MBA (which is a bad path - study business after having some experience IMO) and lack experience. How will she compare in 10 years time when those people also have real world experience?

Re: I were 17, I'd learn how to build LLMs from scratch

#234

Post by TrackerFF »

If you're 17, and seriously curious about how modern day AI works, you might as well just sit down and look at a couple of courses on linear algebra + calculus, machine learning, deep learning, and more LLM specific deep learning. Those courses will teach you how to go from writing your first perceptron to a MVP language model. But also so much more.

Re: I were 17, I'd learn how to build LLMs from scratch

#235

Post by rdedev »

kalms wrote: post #87
chris_va wrote: post #23

I am kind of amazed how negative the comments are here, especially on HN. Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful. The above is, after all, the wh…

How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute

I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.

Re: I were 17, I'd learn how to build LLMs from scratch

#236

Post by brandonb »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

FWIW, at least 20 Y Combinator startups have published ML research recently at ICLR, NeurIPS, ICML, and so on.

I think a lot of people assume that only the big AI labs can do cutting edge research, but there's a strong argument you can do it as part of little tech as well.

Re: I were 17, I'd learn how to build LLMs from scratch

#237

Post by bananaflag »

qsera wrote: post #167

LLMs are incredibly boring to me as a technology..Not in terms of what it can do, but how it works.

Same for me. Neural networks in general.

I first studied them in 2011, and I was like what? Just a bunch of partial derivatives?

I keep looking at AI to check if now it's something else but it keeps being gradient descent.

Okay, it's great that you can perform miracles using gradient descent but that doesn't make it captivating in any way.

Re: I were 17, I'd learn how to build LLMs from scratch

#238

Post by boredumb »

oersted wrote: post #13

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities. The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM comp…

I agree with mostly all of this, but personally I wrote a toy LLM almost 5 years ago and while it never saw much use outside of boring my wife with a shitty command line demo with glee it did help me understand how they worked and how to apply them, played a lot with JAX and pytorch, ended up building a ghetto version of MCP and an LLM-Pool to proxy requests to my baby local models and so I didn't struggle to see the evolution of openrouter and MCP agentic workflows. The same way i'm really glad when I was younger I built a bad webserver by myself, a really painful SQLx type database, etc etc etc - none of these things led me to developing for Nginx or Oracle nor will knowing JAX get me a job at an AI research lab, but I do have a lot of depth in understanding how the technology works so that the flavors on top of them are easy to digest and make more use of immediately, and I think the same can be said for engineers coming into the field - if it's a spooky LLM box you aren't going to be squeezing the same amount of juice as the guy that knows how they work inside and out so having at least the understanding of a _babys first LLM_ is going to get you miles ahead of people who don't.

For anyone who wants to dork around there is https://github.com/rasbt/LLMs-from-scratch which is something amazing that I think anyone who wants to engineer things around LLMs should at least blast through and read.

Re: I were 17, I'd learn how to build LLMs from scratch

#239

Post by fancyfredbot »

mickeyp wrote: post #56

Writing, supervising and training LLMs are now the purview of... even larger LLMs. Optimising CUDA kernels; hand-writing SIMD assembly to speed up data loading; tinkering with your particular brand of DRAM to see if there's anything to gain from optimising for its memory topology and NUMA --- these are now the job of AI. There is very little reason for humans to get all too engrossed in this type of work now, today,…

If this is the job of AI then why does AMD struggle to get the best performance on their hardware? Not enough LLMs?
Post reply on HN