Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3 is no longer the only game in town

lastweekin.ai

Post reply on HN
Print view View original

211–217 of 217 posts

Re: GPT-3 is no longer the only game in town

#211

Post by fragmede »

6gvONxR4sf7o wrote: post #132
Nextgrid wrote: post #102

Earlier quoted context omitted.

Are there any TPUs that can be purchased off-the-shelf and then owned, like you can do with a CPU or GPU? Or are you just limited to paying rent to cloud providers and ultimately being at their mercy when it comes to pricing, ToS, etc?

No, but you probably aren't going to buy an A100 either, so it's a moot point.

An A100 looks to be about $12k or so. A bit out of reach for individuals, but not so bad as a business expense, but maybe you can use it to mine Bitcoin when you're not using it to train models to help pay for itself or something.

Re: GPT-3 is no longer the only game in town

#212

Post by fragmede »

iamcurious wrote: post #57
ComodoHacker wrote: post #51

Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?

We are already there. Machine learning is the flavor of A.I. that keeps business barriers of entry high. If we had invested in symbolic A.I., things would be different. A similar thing happens with programming language flavors. PHP lowers barriers of entry so it is discredited by the incumbents.

And Facebook's success would like a word with them. Even if the company magically disappeared tomorrow, the millionaires who've already cashed out built atop some shitty PHP app that a college kid wrote don't care one whit over that discreditation.

Re: GPT-3 is no longer the only game in town

#213

Post by fragmede »

Dylan16807 wrote: post #156
supermatt wrote: post #56

Earlier quoted context omitted.

GPT-3 is a text generator, so i doubt you would get anything of use. You cant even supply such a large input to GPT-3.

It's pretty good at extending lists. That might require some sorting first.

Gosh, anyone else remember Google Sets? You typed in a few things, eg colors, and Google would add some more things of the same type.

Re: GPT-3 is no longer the only game in town

#214

Post by rg111 »

totoglazer wrote: post #206
rg111 wrote: post #93

The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters. It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it. Anyway, it is surprisingly good. We fine-tuned the…

What library did you use for JAX on TPU? Also curious how much data you had?

We used Flax, of course.

We thought Haiku was cool, but none of us knew it, and learning resources were scarce. But we are very happy with Flax.

We used the Bengali subset of the mC4 dataset [0] for training the GPT-2 model.

[0]: https://huggingface.co/datasets/mc4

Re: GPT-3 is no longer the only game in town

#215

Post by rg111 »

zebraflask wrote: post #153
rg111 wrote: post #93

The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters. It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it. Anyway, it is surprisingly good. We fine-tuned the…

That is a fantastic result - nagging question - these work best on predictable things. How much of Bengali poetry is predictable?

I must tell you that your intuitions are not wrong.

Many language models do predict.

In this case, they either try to predict what the next word (or character, or sub-character in case of Chinese, Japanese, etc.- this is totally the decision of the DS) is , or what some "masked" word are.

    w_i becomes w_(i-1) in the sequence
    where w_i is the last word generated
The ones that are trained to be able to predict the next word are the ones that are good generators.

Re: GPT-3 is no longer the only game in town

#216

Post by TedShiller »

MPSimmons wrote: post #171
TedShiller wrote: post #164

Earlier quoted context omitted.

That's the first time I've heard of a company being worried that their product is too good.

This is legitimately dangerous technology. It probably(?) wouldn't pass the Turing Test, but it would fool a lot of people.

I wish Microsoft would have worried that Windows was too good, and never have released it for this reason. Then we wouldn't have had to go through the whole Windows thing.
Post reply on HN