Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3 is no longer the only game in town

lastweekin.ai

Post reply on HN
Print view View original

91–100 of 217 posts

Re: GPT-3 is no longer the only game in town

#91

Post by lostdog »

iamcurious wrote: post #57
ComodoHacker wrote: post #51

Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?

We are already there. Machine learning is the flavor of A.I. that keeps business barriers of entry high. If we had invested in symbolic A.I., things would be different. A similar thing happens with programming language flavors. PHP lowers barriers of entry so it is discredited by the incumbents.

The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. At my job, dropping the computational load of our ML models is heavily invested in, and every success is celebrated. Everybody wants it to be easier and cheaper to train high quality models, but some things are still intrinsically hard.

Re: GPT-3 is no longer the only game in town

#92

Post by minimaxir »

ComodoHacker wrote: post #51

Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?

Fortunately, costs for training superlarge models are coming down rapidly thanks to TPUs (which was the approach used to train GPT-J 6B) and DeepSpeed improvements.

Re: GPT-3 is no longer the only game in town

#93

Post by rg111 »

The future is not as dark as it seems because of the rat race of megacorps.

You can use reduced versions of language models with extremely good results.

I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters.

It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it.

Anyway, it is surprisingly good. We fine-tuned the model for several downstream tasks and we were shocked when we saw the quality of generated text.

I fine-tuned this model to write Bengali poems with a dataset of just about 2k poems and ran the training for 20 minutes in GPU instance of Colab Pro.

I was really blown away by the quality.

The main training was done in JAX, and it is much faster and seamless than PyTorch XLA, much better than TensorFlow in every way.

So, my point is, although everyone is talking hundreds of billions of parameters and millions in training cost, you can still derive practical value from language models, and that too, at a low cost.

Re: GPT-3 is no longer the only game in town

#95

Post by TulliusCicero »

Tenoke wrote: post #82
evergrande wrote: post #39

Earlier quoted context omitted.

OpenAI takes the Orwellian cake.

I hear a lot of low effort takes about OpenAI but how exactly is providing your service via a paid API the "Orwellian cake"? Is this really the most (or even at all) Orwellian practice for you?

I think it's more the contrast where they claim, via their name, to be open, but actually aren't.

If their name was ProfitableAI, there'd probably be fewer complaints.

Re: GPT-3 is no longer the only game in town

#96

Post by moffkalast »

hubraumhugo wrote: post #71
staticautomatic wrote: post #64

Earlier quoted context omitted.

Did you test it against extractive summarizers?

We experimented with BERT summarization, but the results weren't too good. Do you have any resources or experiences in this area?

That sounds like BERT alright.

Re: GPT-3 is no longer the only game in town

#97

Post by moffkalast »

mark_l_watson wrote: post #16
mrbukkake wrote: post #2

Can anyone tell me what the value of GPT-3 actually is other than generating meaningless prose? What would a business use it for

You can try it yourself - apply for a free API license from OpenAI. If you like to use Common Lisp or Clojure then I have examples in two of my books (you can download for free by setting the price to zero): https://leanpub.com/u/markwatson

I put in a request months ago, I think they're not approving people anymore.

Re: GPT-3 is no longer the only game in town

#98

Post by andreyk »

GhettoComputers wrote: post #38

>However, the ability of people to build upon GPT-3 was hampered by one major factor: it was not publicly released. Instead, OpenAI opted to commercialize it and only provide access to it via a paid API. This made sense given OpenAI’s for profit nature, but went against the common practice of AI researchers releasing AI models for others to build upon. So, since last year multiple organizations have worked towards cr…

I believe you are correct, at least for GPT-3 scaled things. Hopefully that'll change with time, though.

Re: GPT-3 is no longer the only game in town

#99

Post by nradov »

alephaleph wrote: post #73
nradov wrote: post #67

Earlier quoted context omitted.

The fact that GPT3 works at all for coding indicates that our programming languages are too low level and force a lot of redundancy (low entropy). From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement. Of course there might be trade offs. Some of that redundancy might be helping maintenance programmers to understand the code.

That would only follow if we were trying to optimize code for brevity, and I have no clue why that would be your top priority.

Not at all. Brevity (or verbosity) is largely orthogonal to level of entropy or redundancy. In principle it ought to be possible to code at a higher level of abstraction while still using understandable names and control flow constructs.
Post reply on HN