GPT4 already obsolete? Geez, we got the wrong bot, silly heads: https://www.youtube.com/watch?v=EdRMVhlNP5I
OpenAI’s CEO says the age of giant AI models is already over
251–260 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#252Earlier quoted context omitted.
All the LC grinding may come in handy after all! /s What algorithms specifically show the most results upon improvement? Going into this I thought the jump of improvements were really related more advanced automated tuning and result correction, in which it could be done at scale as it were allowing a small team of data scientists to tweak the models until desired results were being achieved. Are you saying instead,…
I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabi…
How did we jump to AGI?
"Sammy A thinks we've made the best engine with the tools at hand" -> "this will never get us out of the solar system"
Sorry to unload on you. It is frustrating to constantly see AGI get brought up needlessly on HN
Re: OpenAI’s CEO says the age of giant AI models is already over
#253I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…
Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?
I think people said the same thing about NNs in general before we hit a scale where they started performing magic.
There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.
Re: OpenAI’s CEO says the age of giant AI models is already over
#254Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
These specific questions are very hard for an AI to answer. Just like humans suck at calculating numbers, AIs aren't good at sparse self-questioning.
They're extremely good at other tasks, like taking very difficult tests that require a lot of knowledge storage.
It's pretty obvious they're on the right path for what they're trying to achieve.
Re: OpenAI’s CEO says the age of giant AI models is already over
#255Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
"The third word of this sentence is "the," and its third letter is "e." - GPT-4
Re: OpenAI’s CEO says the age of giant AI models is already over
#256Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
Videos - all of youtube, all the movies, everything that's ever been captured on film. Transcribe the audio, automatically describe the images and try to predict the next one.
Re: OpenAI’s CEO says the age of giant AI models is already over
#257Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…
I believe this is the paper which you are referring to: https://aclanthology.org/P01-1005.pdf
("Scaling to Very Very Large Corpora for Natural Language Disambiguation" by Michele Banko and Eric Brill, Microsoft Research, 2001)
Re: OpenAI’s CEO says the age of giant AI models is already over
#258Maybe they're now training teams of AIs to work together? This way the hardware limitations are less of a barrier in terms of maximum memory per AI. They can run in parallel in separate hardware and communicate to reach higher goals.
Re: OpenAI’s CEO says the age of giant AI models is already over
#259"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them." An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are…
>Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4?
Yes, that's the whole thing. As others have pointed out, GPT-4 seems like an optimum point that balances cost, ROI, etc... it won't improve much just by throwing more data at it.
Re: OpenAI’s CEO says the age of giant AI models is already over
#260Earlier quoted context omitted.
> Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore. Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models. Or augment the dataset using RLHF, which provides an "infinite" dataset to train LLMs on. Limited by the capabil…
> Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models. Deepmind and others would disagree with you! No-one really knows in actual fact. [1] https://www.deepmind.com/publications/an-empirical-analysis-...
I don't recall the Chinchilla paper disputing my point. They establish "training-compute optimal" scaling laws, but none of their findings suggest that loss hits any kind of asymptote.