All warfare is based on deception -- Sun Zu
OpenAI’s CEO says the age of giant AI models is already over
231–240 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#232I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…
I know what it looks like in my head but I can’t quite figure the algorithm out. The spiking is basically reinforcement learning at the neuron level. Get it right and it’s basically all you need. You don’t even need training data because it will just automagically learn from the data it sees.
Re: OpenAI’s CEO says the age of giant AI models is already over
#233"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them."
An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4?
Re: OpenAI’s CEO says the age of giant AI models is already over
#234Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
First of all, if Altman continually makes misleading statements about AI he will quickly lose credibility, and that short term gain in whatever 'financial incentive' that birthed the lie would be eroded in short order by a lack of trust of the head of one of the most visible AI companies in the world. Secondly, all the competitors of OpenAI can plainly assess the truth or validity of Altman's statements. There are ma…
It is possible that GP meant that Altman’s word is basically worthless to them, in which case that’s not something that can be argued about. It’s a factually true statement that that is their opinion of that man.
I personally can see why someone could arrive at that position. As you’ve pointed out, taking Sam Altman at face value can involve suppositions about how much he values his credibility, how much stock OpenAI competitors put in his public statements, and the mindsets people in general have when reading what he writes.
Re: OpenAI’s CEO says the age of giant AI models is already over
#235The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.
Re: OpenAI’s CEO says the age of giant AI models is already over
#236Earlier quoted context omitted.
Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…
Many years ago, there was an image that floated around with Craigslist and all the websites that replaced small parts of it—personals, for sale ads, etc. It turned out the way to beat Craigslist wasn’t to build Yet Another Monolithic Craigslist, but to chunk it off in pieces and be the best at that piece. This is analogous to what’s happening with AI models. Sam Altman is saying we have reached the point where spendi…
https://www.today.com/money/speculation-craigslist-slowly-dy... from 2011 - is that what you were thinking of? Strange how few of those logos have survived, and how many new logos would now be on it. It would be interesting to see a modernised version.
Re: OpenAI’s CEO says the age of giant AI models is already over
#237Earlier quoted context omitted.
The reason we put everyone through school is we believe that it’s in society’s best interest to educate everyone to the peak of their abilities. It’s good for many different reasons. It would be much easier to identify gifted kids and only educate them, but I happen to agree that universal education is better.
It would be much easier to identify gifted kids and only educate them Is it so easy?
There’s downsides and tradeoffs but yes, if we wanted to we could stop trying to teach kids with below average IQs calculus, unless they specifically wanted to.
Re: OpenAI’s CEO says the age of giant AI models is already over
#238Earlier quoted context omitted.
I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models…
Why are you discussing OpenAI with ChatGPT? I’m honestly interested. I would imagine that any answer of ChatGPT on that topic is either (a) „hallucinated“ and not based on any verifiable fact or (b) scripted in by OpenAI. The same question pops up for me whenever someone asks ChatGPT about the internals and workings of ChatGPT. Am I missing something?
Simple curiosity. I wanted to see if it could explain the shift in OpenAIs operating in a way that might give some interesting or perhaps novel insight (even if hallucinated) other than what their corpo-speak public facing reasoning is.
For the most part it just regurgitated the corpo-speak with an odd sense of confidence. I know that’s the point of the model, but it can also be surprisingly honest when it incorporates what it knows about human motivation and business.
Re: OpenAI’s CEO says the age of giant AI models is already over
#239Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.
Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.
Re: OpenAI’s CEO says the age of giant AI models is already over
#240Earlier quoted context omitted.
On the Lex Fridman podcast, he pretty much admitted he's not an AI (per se) and isn't the most excited about the tech (as he could be).
> he pretty much admitted he's not an AI Yeah, I also had a hunch he wasn't an AI. (I assume you meant "AI researcher" there :)) All joking aside, I wonder how that's affecting company morale or their ability to attract top researchers. I know if I was a top AI researcher, I'd probably rather work at a company where the CEO was an expert in the field (all else being equal).
Honestly I'm not sure it matters that much. CEOs who are not experts or researches in a domain can still build great companies and empower their employees to do incredible work. Lots of tech people absolutely love to point out that Steve Jobs was not an engineer, but under his leadership the company invented three products that totally revolutionized different industries. Now, I'm not going to sit here and say Altman is Jobs, but running a company, knowing how to raise money, knowing how to productize technologies, etc are all very important skills that industry researchers aren't always good at.