Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?
GPT-3 is no longer the only game in town
51–60 of 217 posts
Re: GPT-3 is no longer the only game in town
#52I have supported my beliefs on this topic in these threads to the point of exhausting myself. The tools that we use to find these agents are the underpinning of AGI, it’s coming way faster than even most people here appreciate, this development is intrinsically against the interest of human beings. Please stop and think, please.
Re: GPT-3 is no longer the only game in town
#53Earlier quoted context omitted.
What do you use it for?
Coding, I actually had to forbid it today in a course I teach because it solves all the exercises :) (given unit tests with titles students needed to fill those tests in)
Isn't that just because others have stored solutions to these problems in GitHub?
Re: GPT-3 is no longer the only game in town
#54Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?
That’s not even distant - most of the self-supervised vision and language models at the bleeding edge of the field require huge compute budgets to train.
Re: GPT-3 is no longer the only game in town
#55Re: GPT-3 is no longer the only game in town
#56If I were to run GPT-3 on my 70000 browser bookmarks, what kind of insights could I get from that? Only by analyzing the page title (from the bookmark, not by re-fetching the url) and eventually also the domain name.
GPT-3 is a text generator, so i doubt you would get anything of use. You cant even supply such a large input to GPT-3.
Re: GPT-3 is no longer the only game in town
#57Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?
We are already there. Machine learning is the flavor of A.I. that keeps business barriers of entry high. If we had invested in symbolic A.I., things would be different. A similar thing happens with programming language flavors. PHP lowers barriers of entry so it is discredited by the incumbents.
Re: GPT-3 is no longer the only game in town
#58Earlier quoted context omitted.
Have you heard of GitHub copilot ? It’s based on GPT3 and I can tell you one thing: it does not generate meaningless prose (90% of the time)
How many % of the time does it produce code that compiles?
In my experience 95% of the time. And 80% of the time it output codes which is better than I would have done myself in a first approach (thinks of corner cases, adds meaningful comments etc.). It’s impressive.
Re: GPT-3 is no longer the only game in town
#59Earlier quoted context omitted.
From my anecdotal experience, the vast majority of the time (90+%).
Interesting. Is there any constraint built into the model that makes this possible? E.g. grammar, or semantics of the language? Or is it all based on deep learning only?
Deep learning only I believe. But real good one
Re: GPT-3 is no longer the only game in town
#60Neat to see more models getting closer, thought it appears only one so far has exceeded GPT-3's 175B parameters. That said, what I'm really curious is how those other models stack up against GPT-3 in terms of performance -- does anyone know of any comparisons?
There's Wu Dao 2.0 and Google has 2 models with 1T+ params.
For clarity, i believe these are all mixture of expert models, where each input only sparsely activates some subset subset of the full model. This is why they were able to make such a big jump over the “dense” GPT3. Not really an apples-to-apples comparison.