Number of parameters aside, I am really surprised that we havent yet reached hundreds of TB of training data. Especially Chinese model only used less than 10 TB of data.
GPT-3 is no longer the only game in town
141–150 of 217 posts
Re: GPT-3 is no longer the only game in town
#142Earlier quoted context omitted.
1. There's a world of problems (such as "perception-related" e.g. vision and NLP) which we tried to solve for decades with symbolic AI and got worse results than what nowadays first-year students can do as a homework with ML; 2. For your example of chess, for some time now ML engines are pretty much untouchable by engines based on pre-ML methods.
Yes I agree with all your points - I was however responding to the point being made that symbolic AI "wasn't useful"...which in the past it was. Perhaps in the future some new method or breakthrough will mean it becomes useful once again?
this is a great point.
much like deep learning was invented decades ago but didn't become feasible until technology caught up, could the same be true for symbolic AI?
i.e., is the ceiling for symbolic AI technical and transient or fundamental and permanent?
Re: GPT-3 is no longer the only game in town
#143Earlier quoted context omitted.
We are already there. Machine learning is the flavor of A.I. that keeps business barriers of entry high. If we had invested in symbolic A.I., things would be different. A similar thing happens with programming language flavors. PHP lowers barriers of entry so it is discredited by the incumbents.
PHP wasn't discredited by the incumbents. It was discredited by its creator. "I'm not a real programmer. I throw together things until it works then I move on. The real programmers will say Yeah it works but you're leaking memory everywhere. Perhaps we should fix that. I'll just restart Apache every 10 requests." -Rasmus Lerdorf "I was really, really bad at writing parsers. I still am really bad at writing parsers."…
To most programmers that doesn't discredit PHP at all. He cares about a working product, much like 90% of programmers, who don't have the privilige to worry about theory. They just need an ecommerce, or blog or whatever, running asap. To use a pg's analogy, they are there to paint not to worry about painting chemistry.
The incumbents do discredit PHP though. For instance, facebook was built on PHP, and still runs on it. They used the language of personal home pages to give every person on the planet a personal home page. Nevertheless, once they suceeded they forked PHP with a new name and isolated devs culturally.
Re: GPT-3 is no longer the only game in town
#144Earlier quoted context omitted.
> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?
> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work? Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach. Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish no…
“I would challenge you to name any (non-simple) problem where traditional AI methods are still state of the art.”
Lossless file compression. As far as I know none of the algorithms in widespread use are neural-based, despite the fact that compression is clearly a rich statistical modeling problem, at least on par with GPT-3-style language understanding in difficulty. There are published attempts to solve the problem with neural networks, but they simply don’t work well enough to date. Modern solutions also still use old-fashioned AI ingredients like compiled dictionaries of common natural-language words — any other domain where nat-lang dictionaries are useful has been conquered by neural solutions, e.g. spelling and grammar checkers.
Re: GPT-3 is no longer the only game in town
#145People were blaming cryptocurrencies miners for the prices of GPUs, when in fact it was the AI researchers who bought all the GPUs. :D I wonder what if somebody designs an electronic currency rewarded as payment for general GPU computations instead of just computing hashes? You pay some $, to train your model and the miner gets some coins. Every one is happy, electricity is not wasted and the GPUs gets used for a rea…
If everyone offer GPUs, is the same game. If I will buy more GPU I will get more money, so the average payment for a person with a single or a small bunch of GPU will be low.
And second, the principles of electronic currency are different from gold/money. That's why crypto uses GPU ;)
Re: GPT-3 is no longer the only game in town
#146Earlier quoted context omitted.
We are already there. Machine learning is the flavor of A.I. that keeps business barriers of entry high. If we had invested in symbolic A.I., things would be different. A similar thing happens with programming language flavors. PHP lowers barriers of entry so it is discredited by the incumbents.
The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. At my job, dropping the computational load of our ML models is heavily invested in, and every success is celebrated. Everybody wants it to be easier and cheaper to train high quality models, but some things are still intrinsically hard.
>The difference between ML and symbolic AI is that ML works and symbolic AI doesn't.
There was a point when it was the other way around, this is not static but the result of resources being poured. The data heavy, computational heavy, black box style of ML gives power to large business over small business. So it's seen as a safer bet than symbolic A.I. This in turn makes it work better, which makes it an even safer bet. Notice that startups dream of being big business so they still pick ML.
Also notice that in some domains ML is still behind symbolic A.I., for instance a lot of robotics and autonomous vehicles.
Re: GPT-3 is no longer the only game in town
#147Earlier quoted context omitted.
Yes I agree with all your points - I was however responding to the point being made that symbolic AI "wasn't useful"...which in the past it was. Perhaps in the future some new method or breakthrough will mean it becomes useful once again?
this is a great point. much like deep learning was invented decades ago but didn't become feasible until technology caught up, could the same be true for symbolic AI? i.e., is the ceiling for symbolic AI technical and transient or fundamental and permanent?
My feeling is that even in our own thinking symbols are used mostly to communicate our (inherently non-symbolic) thoughts to others or record them; i.e. they are a solution to a bandwidth-limited transfer of information while the actual thinking process happens with concepts that have more similarity to collections of vague parameters and associations which can be compressed to symbols only imperfectly with losses.
From that perspective, I don't see how symbolic AI would be competitive but there would be a role for symbolic AI in designing systems that can be comprehensible for humans, but perhaps just as a distillation/compression output from a non-symbolic system. I.e. have a strong "black box" ML system that learns to solve a task, and then have it construct a symbolic system that solves that task worse, but in an explainable way.
Re: GPT-3 is no longer the only game in town
#148Earlier quoted context omitted.
Somehow I don't think that this is quite how Jerome Powell would respond in an interview: > Interviewer: How do you think a global pandemic would impact the financial system? > Mr. Powell: A pandemic will have impact on the economy. It will have an impact on the financial system. > Interviewer: What would the impact be? > Mr. Powell: If a major pandemic occurred, the economy would be affected, and the financial syste…
This is a bad example. Here's an output from gpt-j-6b by EleutherAI: Prompt( Jerome Powell was interviewed about the impact of a pandemic on the economy. > > Interviewer: What would the impact be? > Mr. Powell: If a major pandemic occurred) Output( it could be very disruptive. > > Interviewer: How disruptive? > Mr. Powell: Well, it could be disruptive in a number of ways. > > Interviewer: How so? > Mr. Powell: Well,…
I think the "bad example" is actually the good one, because it's a reminder that actually you're not getting business advice from someone with Warren Buffet or Jerome Powell's understanding of the economy, you're getting text generated by analysing patterns in other not-necessarily-applicable text. If you start forcing it in very specific directions you start getting text that summarises the commentary in the corpus, but most of that commentary doesn't come from Warren Buffet or Jerome Powell and isn't applicable to the future you're asking it about...
Re: GPT-3 is no longer the only game in town
#149Re: GPT-3 is no longer the only game in town
#150Earlier quoted context omitted.
It works very well with VSCode. It has an integration. It shows differently than normal autocomplete, it shows just like gmail autocomplete (grayed out text sugggestion, and press tab to actually autocomplete). Sometimes the suggestion is just a couple tokens long, sometimes it’s an entire page of correct code. Nice trick: write a comment describing quickly what your code will do (“// order an item on click”) and enj…
Interesting second use case; I use comments like this already as typical practice and I agree Copilot fills in the gaps quite well - never thought to do it in reverse... will give that a shot today.
I also like to do synthesis from example code (@example doccomment) and synthesis from tests.