Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I were 17, I'd learn how to build LLMs from scratch

twitter.com

Post reply on HN
Print view View original

611–620 of 725 posts

Re: I were 17, I'd learn how to build LLMs from scratch

#611

Post by tomhow »

delis-thumbs-7e wrote: post #540

When I was 17, I wrote poetry and learned how to play guitar. I’m really happy that I did. That’s what I would do now, were I 17. I feel slightly bad for people who didn’t.

Sure, but isn't this a bit of a sneer? “I feel slightly bad for people who didn’t” has an air of superiority about it, and seems like a veiled swipe at those who lacked whatever qualities your 17-year-old self had. The guidelines have several different ways of asking us to avoid commenting like this.

As to the substance of your comment, there's nothing stopping a 17-year-old from writing poetry, learning guitar, and also learning about how LLMs work, if they're so inspired. Indeed I'm sure pg would encourage it, and that he did the equivalent of all those things himself when he was young. There's plenty of evidence of that in his essays: he wrote short stories, published a scandalous school newspaper, played soccer, and studied fine art and philosophy. See:

https://www.paulgraham.com/worked.html

https://www.paulgraham.com/nerds.html

Re: I were 17, I'd learn how to build LLMs from scratch

#612

Post by libraryofbabel »

michael0church wrote: post #589
nialv7 wrote: post #583

Earlier quoted context omitted.

The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.

This is a great point. The complexity sources are entirely different. An OS is complex because it’s solving a difficult problem and we’ve seen decades of evolution both in knowledge and-how and, at the frontier, in performance standards. The abilities of LLMs are emergent. You can experiment with LLMs and know as much about their observable behavior as the experts. But there’s no way to “crack open” an LLM and see pr…

> there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives

Mechanistic Interpretability has entered the chat.

For a classic example, see https://www.anthropic.com/research/tracing-thoughts-language...

The spirit of your point stands, though. This kind of research is interesting to read about, but it's very hard, more like neuroscience or biology than computer science ("LLMs are grown, not made"). You're dealing with a lot of extremely _messy_ complexity, for which organic life is really the only good point of comparison. Most of us here are't really equipped for that kind of work; it's not at all like, say, reverse-engineering a piece of software written by humans. And of course the only people who can do it on frontier models from Anthropic and OpenAI are people within the labs themselves. (But I'm optimistic we'll see more of this work on open weights models...)

Re: I were 17, I'd learn how to build LLMs from scratch

#613

Post by bilbo0s »

__MatrixMan__ wrote: post #421
femto wrote: post #294

Earlier quoted context omitted.

In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good l…

Good points, but I think we can expect the AI space to be more tumultuous. What most people want from mobile technology is for it to work, not too expensively, and for it to get out of their way. What most people want out of AI is for no leader to emerge and wield supremacy against the rest of us. People are afraid of it in ways they weren't afraid of mobile, so they're more willing to work together against whoever i…

>* What most people want out of AI is for no leader to emerge and wield supremacy against the rest of us. People are afraid of it in ways they weren't afraid of mobile, so they're more willing to work together against whoever is in the lead.*

No.

That’s what people like us on HN want. The people out in “Greater Userland” just want the black box to answer their questions. They could care less who is behind it. They don’t yet attach their black box to Amazon or Microsoft etc. And most won’t care enough to be inconvenienced even when they do make the connection. (As your competition argument implies.)

Heck, a lot haven’t even made the connection between the black box that gives them answers and data centers. They think, “ ChatGPT good” and at the same time think “data centers bad”.

Re: I were 17, I'd learn how to build LLMs from scratch

#615

Post by kunley »

Creamsicle47 wrote: post #443
andy_ppp wrote: post #419

Earlier quoted context omitted.

I'm not sure it's possible to have intuition about systems that work in thousands of orthogonal dimensions. In fact I'm pretty sure most of the research is people trying fairly arbitrary things and testing them and then post rationalising implied understanding of what is really happening on top of good outcomes.

I'd have to push back, though not on the part you'd expect. Your description of human researchers is roughly right: a lot of the field is try-things-and-narrativize-after. But the load-bearing assumption is that intuition has to be human-shaped intuition. Humans can't intuit thousands of orthogonal directions because we project everything down into a 3D metaphor and hope it holds. That's a fact about our hardware, no…

You apparently have no intuition on what the intuition is

Re: I were 17, I'd learn how to build LLMs from scratch

#616

Post by CamperBob2 »

bigfishrunning wrote: post #477
ryandrake wrote: post #455

Earlier quoted context omitted.

I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you…

I would argue that in a post-LLM world, becoming as specialized as possible is the only way to survive. If you have the serious systems programming skills required to write graphics drivers, your talent would be wasted writing CRUD apps anyway. I hope you eventually found/will find something more appropriate for your skillset!

I'd argue the exact opposite. If I want a graphics driver, I'll ask the clanker for one. But if I don't know enough to even know what I want, there's no path forward.

Re: I were 17, I'd learn how to build LLMs from scratch

#617

Post by dominotw »

megaloblasto wrote: post #596
andy_ppp wrote: post #476

Earlier quoted context omitted.

What provable conclusions have you intuited around how LLMs work? Give me some examples to prove your point? I'm absolutely happy to change my view with enough data.

I think provable conclusions and building an intuition are different things to be fair. In my opinion it is entirely possible to build intuition about seemingly impossible to understand topics like infinitesimal areas, infinite limits, function mappings, and high dimensional spaces. For example, 3D spaces are easy to intuit if you can visualize vectors in your head. Some people claim it is impossible to understand di…

i read all that i still dont see what your intuition is about how llm work.

visualizign 100 dim matrix will not tell you how llm work. so what you even talking about.

Re: I were 17, I'd learn how to build LLMs from scratch

#618

Post by edgyquant »

IshKebab wrote: post #41
embedding-shape wrote: post #35

Earlier quoted context omitted.

> I would encourage people not to seek advice from successful people like this (survivorship bias). Personally I don't see the problem, as long as you're aware there is survivorship bias involved here. What's the alternative really, seek advice from unsuccessful people? That seems worse :) Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do wha…

Seek advice from people who have had a normal level of success. Not a one-in-a-million level.

I would say you should seek advice from someone has lucky and successful as Paul Graham but I wouldn’t take his public Twitter as advice it’s entertainment period.

Re: I were 17, I'd learn how to build LLMs from scratch

#619

Post by Grombobulous »

DarmokTanagra wrote: post #571
Grombobulous wrote: post #537

Earlier quoted context omitted.

I just want to make it clear that I’m not assigning some kind of moral superiority to my financial situation and the amount that the economy values my labor per hour. I don’t make the rules for how much each profession is able to make in compensation. The world will always need nurses, but that doesn’t mean that it’s a fantastic career to get into if you have a neutral career preference and your primary consideration…

[flagged]

Perhaps I made a mistake in bringing my personal situation into this discussion. Delete that one sentence about my situation and you'll see my underlying argument is still there.

If you have anything to add to the topic at hand rather than critiquing my personality, I would love to hear it.

Is it really wise to tell kids to avoid tech when the #1 ROI bachelors degree is Computer Science and the #1 salary after 10 years degree is Computer Engineering, while the top 9 salaried bachelors degrees after 10 years are engineering degrees? Are we just telling kids to avoid tech/engineering on vibes or real labor force data?

Re: I were 17, I'd learn how to build LLMs from scratch

#620

Post by sebastianconcpt »

mattlutze wrote: post #224

A lot of people here are responding to the message but not to the meaning. It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of…

That depth requires math and traditional computing depth. Where is the time to be young and have that depth?
Post reply on HN