So is there any one of them that I could play around with?
GPT-3 is no longer the only game in town
121–130 of 217 posts
Re: GPT-3 is no longer the only game in town
#122Earlier quoted context omitted.
You can try it yourself - apply for a free API license from OpenAI. If you like to use Common Lisp or Clojure then I have examples in two of my books (you can download for free by setting the price to zero): https://leanpub.com/u/markwatson
I put in a request months ago, I think they're not approving people anymore.
You might try signing up directly for a paid non/free account, if that is possible to do. I was using a free account, then switched to paying them. Individual API calls are very inexpensive.
Re: GPT-3 is no longer the only game in town
#123The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters. It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it. Anyway, it is surprisingly good. We fine-tuned the…
> The future is not as dark as it seems because of the rat race of megacorps. Just wait until NVidia comes with a "Neural AppStore" and corresponding restrictions. Then wait until the other GPU manufacturers follow suit.
Much of the work done is fully open source and are liberally licensed.
DeepMind and OpenAI have a bad rep in this regard.
But a lot is available for free (as in beer and speech).
And most of the research papers are released in arXiv. It's very refreshing.
The bottleneck is not the knowledge or code, but the compute. People are fighting this in innovative ways.
I have been an inactive part of Neuropark that first demoed collaborative training. A bunch of folks (some of them close to laypeople) ran their free Colab instances and trained a huge model. You can even utilize a swarm of GT1030s or something like that.
Also, if you have shown signs of success, you are very likely to have people willing to sponsor your compute needs, case in point- Eluether AI.
The situation is far from ideal with this megacorps rat race [0], and NLP research being more and more inaccessible, but it is not completely dark.
[0]: I, along with many respected figures tend to think that this scaling up stuff approach is not even useful. We can write good prose with GPT-3 nowadays, that are, for all intents and purposes, indistinguishable from text written by humans. But we are far, far away from true understanding. These models don't really understand anything and are not even "AI", so to speak.
The Transformer architecture, the backbone of all these approaches- is too brute-force-y for my taste to be considered something that can mimic or, further- be intelligent.
Re: GPT-3 is no longer the only game in town
#124Earlier quoted context omitted.
> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?
they didn't. that was just alpha beta search with some custom hardware to speed it up. also at this point, both of the strongest chess ai (stockfish and lc0) are using neutral networks and are roughly 1000 elo above where deep blue was (and most of that is from software, not hardware)
> just alpha beta search
I will cling to these goal posts every time. Search was and still is AI, unless you think Russell and Norvig should have named the field's foundational textbook something other than "Artificial Intelligence: A Modern Approach"
Re: GPT-3 is no longer the only game in town
#125Earlier quoted context omitted.
OpenAI takes the Orwellian cake.
I hear a lot of low effort takes about OpenAI but how exactly is providing your service via a paid API the "Orwellian cake"? Is this really the most (or even at all) Orwellian practice for you?
Re: GPT-3 is no longer the only game in town
#126The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters. It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it. Anyway, it is surprisingly good. We fine-tuned the…
Good to know. We're trying to attempt something similar[1] but for Tamil. I'm also surprised how well the OSS language model & library AI4Bharat [2] performs for NLP tasks against SoTA systems. Is there a way to contact you? [1] https://vpt.ai/posts/about-us/ [2] https://ai4bharat.org/projects/
Among a master's degree, a consultancy gig, personal research and study, and finding unis abroad- I am living a hectic life.
I don't see how I can be of help.
But I can talk. Leave me something through which I can reach you. And I will reach you within a week.
Re: GPT-3 is no longer the only game in town
#127Earlier quoted context omitted.
The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. At my job, dropping the computational load of our ML models is heavily invested in, and every success is celebrated. Everybody wants it to be easier and cheaper to train high quality models, but some things are still intrinsically hard.
> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?
1. There's a world of problems (such as "perception-related" e.g. vision and NLP) which we tried to solve for decades with symbolic AI and got worse results than what nowadays first-year students can do as a homework with ML;
2. For your example of chess, for some time now ML engines are pretty much untouchable by engines based on pre-ML methods.
Re: GPT-3 is no longer the only game in town
#128Neat to see more models getting closer, thought it appears only one so far has exceeded GPT-3's 175B parameters. That said, what I'm really curious is how those other models stack up against GPT-3 in terms of performance -- does anyone know of any comparisons?
I’m surprised that no one has answered for three hours!
The answer is at https://github.com/kingoflolz/mesh-transformer-jax
It has detailed comparisons and a full breakdown of the performance, courtesy of Eleuther.
Re: GPT-3 is no longer the only game in town
#129Earlier quoted context omitted.
Isn't that just because others have stored solutions to these problems in GitHub?
That is my question too. Is it a fancier autocomplete? Or does it reason about code?
In some sense you could think of as a fancy autocomplete which uses not only code but also comments as input, looks up previous solutions for the same problem but (mostly) appropriately replaces the variable names to those that you are using.
Re: GPT-3 is no longer the only game in town
#130Neat to see more models getting closer, thought it appears only one so far has exceeded GPT-3's 175B parameters. That said, what I'm really curious is how those other models stack up against GPT-3 in terms of performance -- does anyone know of any comparisons?
Whenever I've seen language modeling metrics, GPT-3's largest model has been at the top. If you see a writeup that doesn't include accuracy-type metrics, you're reading a sales pitch, not an honest comparison.