Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

youtube.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#71

Post by bamboozled »

lvl102 wrote: post #69

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

How can we really stop this though?

I feel like Microsoft is doing what Microsoft does yet again. They piggybacked on the whole "open source" angle but this time, rather than be "software vendor lock-in closed source assholes", they're now being "IP theft open source" assholes.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#72

Post by burntalmonds »

dd36 wrote: post #27
coldtea wrote: post #21

Earlier quoted context omitted.

As if history will care?

The history AI will have an opinion that mirrors the politics of the user that asks it what it thinks of Karpathy’s time at Tesla.

Only in the default bubble mode.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#73

Post by fithisux »

lvl102 wrote: post #69

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

Totally agree. I also doing some Gcloud training. This should also be public good. When I move my machines to Google/AWS/ .. my business changes ownership and I become slave chained by the "competition" and "pricing".

To fix this we need to change mentality and governments.

But people are satisfied with their:

"Mirror, Mirror on the Wall, Who's the Coolest of Them All? ... You, because you are indistinguishable from the others!!!!"

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#74

Post by osigurdson »

karpathy wrote: post #6
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

>> de-duplicate, clean, wrangle. this is a lot of work regardless of $.

This sounds like a great job for a specialized GPT!

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#75

Post by bestbuyer__ »

notjonheyman wrote: post #55

I had the please of working under Andrej a few years ago. It was very interesting the problems we were solving until this other guy that is a billionaire would throw a wrench (and fits) into the whole thing.

You mean he got hired by Elon Musk to work at Tesla? :P

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#76

Post by naasking »

somenameforme wrote: post #53
anigbrowl wrote: post #13

Earlier quoted context omitted.

Extremely interested in your take on where language/reasoning competency ends and knowledge retrieval begins. OpenAI stuff has succeeded in part because it can synthesize good bullshit* on a huge variety of topics. For many purposes this makes it as good as asking someone in the same room to look something up for you on Wikipedia. But while vast general and somewhat special knowledge is very impressive, comprehension…

This is why it has always seemed to me that the 'chat bot' -> AI pathway has felt quite analogous to the 'chess bot' -> AI pathway. We're constantly trying to replicate things that look like demonstrations of intelligence, but never really bothering with what intelligence is. What I mean is that a man lifting 400kg is a demonstration of exceptional athleticism. A 400kg man sitting on a balance and having 400kg go up…

We don't know how to define the essence of intelligence, so all we can do is knock all the strawmen we traditionally attribute to intelligence, until we're either left with the essence of true intelligence, or we knock off everything and we find intelligence was just a bunch of tricks after all.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#77

Post by pjmorris »

lvl102 wrote: post #69

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

> I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

I don't know if these ideas are worse technically or politically, or both, but what comes to mind are these alternatives:

1. Have someone start a non-profit that curates public data goods, maybe gaining access to data through voluntary donations and through buying all the data provider feeds, and funding through subscriptions by people and organizations who want to understand the data provider infrastructure.

2. Get legislation passed that identifies public data goods and requires that they be made available to all.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#78

Post by belltaco »

lvl102 wrote: post #69

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

I am not getting the angle here. Anyone, including you, can write GPT-like code, train the model with public data and release it for free. It may cost a few million to train in GPU costs, but if what you say is that important, surely there are folks here(I am assuming a good chunk of HN folks have a decent amount of disposable income) who will donate it for the public good? If there are not, then they either don't consider it important or are just virtue signaling. Or the effort is actually hard to implement.

I am totally okay with OpenAI being worth $30 billion or whatever when compared to crypto scams being worth billions.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#80

Post by somenameforme »

naasking wrote: post #76
somenameforme wrote: post #53

Earlier quoted context omitted.

This is why it has always seemed to me that the 'chat bot' -> AI pathway has felt quite analogous to the 'chess bot' -> AI pathway. We're constantly trying to replicate things that look like demonstrations of intelligence, but never really bothering with what intelligence is. What I mean is that a man lifting 400kg is a demonstration of exceptional athleticism. A 400kg man sitting on a balance and having 400kg go up…

We don't know how to define the essence of intelligence, so all we can do is knock all the strawmen we traditionally attribute to intelligence, until we're either left with the essence of true intelligence, or we knock off everything and we find intelligence was just a bunch of tricks after all.

I don't think the essence is especially elusive. It's the ability to make novel, meaningful, and useful discoveries from precepts that don't immediately "obviously" lead to those discoveries. Observing the sky and nature leading to a mathematical formulation of gravity is an absolutely amazing leap. In part because of what was done, but perhaps even more so for even beginning to imagine it was something that could be done.

Imagine stargazing in a time prior to Newton, observing objects falling on Earth, and somehow managing to derive an accurate mathematical formulation of something you had no reason to imagine even existed. Man hadn't been to space, let alone the moon, and so for all we knew if you dropped an apple anywhere it would just as well fall. Perhaps how Newton may have began his discovery was by asking himself where it would fall, but that's a tangent.

---

Humanity's entire existence has been but a blink of time on any sort of timescale, besides our own lives. And in that time we went from the bleeding edge of technology, perhaps pun intended, being 'poke them with the pointy end' to having men travel into space, voyage to other 'planets', land on them, and imminently live on them.

A truly intelligent machine, given its capacity for practically infinite storage, infinitely more accurate recall, and arbitrarily small 'generations' (in terms of self recursive evolutionary improvement) ought be able to not only match this, from a similar starting base of knowledge, but move rapidly beyond it at an extremely swift rate.

So I don't think it would be particularly ambiguous, or debatable. An intelligent machine would quickly advance nearly all fields of humanity by an unimaginable amount. Given the ability for a machine to also scale its own processing capacity to arbitrarily higher degrees (while we're, more or less stuck with fixed 'hardware') this should be able to continue for an exceptionally long period of time as well.

You would end up completely revolutionizing humanity to a degree we can't even really imagine today - repeatedly, and it would all happen within a matter of years. I'm even being somewhat generous here by allowing for a greater intelligence to remain mindless and servile, which seems quite antithetical to intelligence. But if we can "just" get to here, as described, I think that'd be pretty compelling, servility notwithstanding.

Post reply on HN