Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

11–20 of 334 posts

Re: NanoGPT

#11

Post by abricq »

noidiocyallowed wrote: post #5
awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

Work together and fuck up companies together. That's the way to go.

Or how to apply communism to software engineering. I like that.

More seriously, the risk that a few companies become even more powerful thanks to their restricted access to such NN is very frightening. The worth is, without legal restrictions, there is nothing that we can do against it. And I doubt that legal restrictions come in the next months / years.

Re: NanoGPT

#12

Post by isthisthingon99 »

awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

How long does it take to train a human? It's useless for two years then maybe it can tell you it needs to poop.

The breakthrough will be developing this equivalent in an accessible manner and us taking care to train the thing for a couple of decades but then it becomes our friend.

Re: NanoGPT

#13

Post by breck »

awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

> Are there any possible technologal or scientific leaps on the horizon

Yes. From 2017: "Prediction 4: The simplest 2D text encodings for neural networks will be TLs. High level TLs will be found to translate machine written programs into understandable trees."

We have something coming out that is an OOM better than anything else out there right now.

Re: NanoGPT

#14

Post by hankman86 »

make3 wrote: post #3
awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money

Take a leaf from Seti@Home‘s book and try to come up with a distributed, volunteer-based approach to training an open source LLM. There is already an enormous amount of suitable ML hardware on end user devices.

Re: NanoGPT

#16

Post by spi »

awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

What do you mean by "small players have no chance"? OpenAI was founded in 2015, it used to be a "small player" which just got things right and grew with it - we're not talking of Google or Facebook investing a chunk of their billions cash. In Germany, AlephAlpha has built their own supercomputer and are training similar sized models. It's expensive for sure, but well in the possibilities of startups. In France researchers trained the similarly sized BLOOM model https://huggingface.co/bigscience/bloom. They claim it cost between $2 and $4 millions.

Sure, a single researcher can't replicate this at their university, but even though OpenAI likes to publish it this way, we're not really talking about research here. Research was inventing the transformer architecture, this is just making it bigger by (very smart) engineering choices. It's something companies should do (and are doing), not researchers.

Re: NanoGPT

#17

Post by rjtavares »

varispeed wrote: post #10
rjtavares wrote: post #7

Earlier quoted context omitted.

Small players should focus on applications of this tech. We now know that whatever AI Models succeed in the future, they'll be trained by a huge company and finetuned to a specific use case. Small companies should be working on use cases, and then just upgrade to the latest SOTA model.

> Small players should focus on applications of this tech. That sounds a bit condescending. We are probably at a point from which the government should intervene and help establish level playing field. Otherwise we are going to see a deeper divide between multibillion businesses conquering multiple markets and sort of neofiefdom situation. This is not good.

I'm not being condescending at all, we've learned that the value in AI is in the applications. If you think government should regulate the field, it should be to make AI Models a commodity, like electricity.

Re: NanoGPT

#18

Post by tiborsaas »

varispeed wrote: post #10
rjtavares wrote: post #7

Earlier quoted context omitted.

Small players should focus on applications of this tech. We now know that whatever AI Models succeed in the future, they'll be trained by a huge company and finetuned to a specific use case. Small companies should be working on use cases, and then just upgrade to the latest SOTA model.

> Small players should focus on applications of this tech. That sounds a bit condescending. We are probably at a point from which the government should intervene and help establish level playing field. Otherwise we are going to see a deeper divide between multibillion businesses conquering multiple markets and sort of neofiefdom situation. This is not good.

It's not that condescending, that's todays reality. Should I feel entitled for $600k training time that may or may not work? Do you think the government is a good actor to judge if my qualifications are good enough to grant me resources worth a house?

It's quite reasonable to make use of models already trained for small players.

Re: NanoGPT

#19

Post by licebmi__at__ »

isthisthingon99 wrote: post #12
awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

How long does it take to train a human? It's useless for two years then maybe it can tell you it needs to poop. The breakthrough will be developing this equivalent in an accessible manner and us taking care to train the thing for a couple of decades but then it becomes our friend.

Yes, but to be fair, the system that does the training really sucks and doesn’t scale.

Re: NanoGPT

#20

Post by googlryas »

varispeed wrote: post #10
rjtavares wrote: post #7

Earlier quoted context omitted.

Small players should focus on applications of this tech. We now know that whatever AI Models succeed in the future, they'll be trained by a huge company and finetuned to a specific use case. Small companies should be working on use cases, and then just upgrade to the latest SOTA model.

> Small players should focus on applications of this tech. That sounds a bit condescending. We are probably at a point from which the government should intervene and help establish level playing field. Otherwise we are going to see a deeper divide between multibillion businesses conquering multiple markets and sort of neofiefdom situation. This is not good.

Do you think you'll get a global agreement on this? Or would china just eat America's lunch then?
Post reply on HN