These kind of comments have me update my priors so I downgrade them in relevance. Two days ago we were talking about AGI and Altman was letting himself be cuddled, knowing full well it was mindless hype. Now he's trolling the other way. Ok. My weights have been updated.
OpenAI’s CEO says the age of giant AI models is already over
271–280 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#272I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…
Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?
> Why wouldn't we have already seen spiking networks contribute something by now?
This presupposes we've explored this space thoroughly, and we haven't. When everything you do with NNs improves results (how it mostly is now), that means not enough people are trying out ideas and new things.
I don't think you can invoke EMH-like reasoning quite yet. Give us a nice long winter like physics has had, and then we can use this heuristic.
Re: OpenAI’s CEO says the age of giant AI models is already over
#273[deleted]
Re: OpenAI’s CEO says the age of giant AI models is already over
#274dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.
[flagged]
So your embrace of nihilism is total. Here are a couple of books worth reading:
- On Tyranny: https://bookshop.org/p/books/on-tyranny-twenty-lessons-from-...
- The Road to Unfreedom: https://bookshop.org/p/books/the-road-to-unfreedom-russia-eu...
Re: OpenAI’s CEO says the age of giant AI models is already over
#27542
train the entire model for a single question.
Re: OpenAI’s CEO says the age of giant AI models is already over
#276Earlier quoted context omitted.
First of all, if Altman continually makes misleading statements about AI he will quickly lose credibility, and that short term gain in whatever 'financial incentive' that birthed the lie would be eroded in short order by a lack of trust of the head of one of the most visible AI companies in the world. Secondly, all the competitors of OpenAI can plainly assess the truth or validity of Altman's statements. There are ma…
Elon Musk has been constantly doing this and thriving.
Hacker News posters too
Re: OpenAI’s CEO says the age of giant AI models is already over
#277I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…
I'm constantly telling this to people who don't know what I'm talking about and don't know why I brought it up at a wedding, but I think it's inevitable.
Re: OpenAI’s CEO says the age of giant AI models is already over
#278Earlier quoted context omitted.
Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?
I think people said the same thing about NNs in general before we hit a scale where they started performing magic. There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.
> There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.
The reason I like the CPU approach is the memory scaling is bonkers compared to GPU. You can buy a server that has 12TB of DRAM (in stock right now) for the cost of 1 of those H100 GPU systems. This is enough memory to hold over 3 trillion parameters with full 32-bit FP resolution. Employ some downsampling and you could get even more ridiculous.
If 12TB isn't enough, you can always reach for things like RDMA and high speed interconnects. You could probably get 100 trillion parameters into 1 rack. At some point you'll need to add hierarchy to the SNN so that multiple racks & datacenters can work together.
Imagine the power savings... It's not exactly a walk in the park, but those DIMMs are very eco friendly compared to GPUs. You don't need a whole lot of CPU cores in my proposal either. 8-16 very fast cores per box would probably be more than enough, looking at how fintech does things. 1 thread is actually running the entire show in my current prototype. The other threads are for spike timers & managing other external signals.
Re: OpenAI’s CEO says the age of giant AI models is already over
#279Earlier quoted context omitted.
[flagged]
So your embrace of nihilism is total. Here are a couple of books worth reading: - On Tyranny: https://bookshop.org/p/books/on-tyranny-twenty-lessons-from-... - The Road to Unfreedom: https://bookshop.org/p/books/the-road-to-unfreedom-russia-eu...
[flagged]
Re: OpenAI’s CEO says the age of giant AI models is already over
#280The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
OpenAI may have those internally though.