121–130 of 525 posts
It really shouldn't matter that it can give the exact birthdate of Steve Wozniac, as long as it can properly make a query to fetch it and deal with the result.
>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…
Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…
Even if we don't expend any cost on education the number of people with IQ 135 stays the same.
Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
Secondly, all the competitors of OpenAI can plainly assess the truth or validity of Altman's statements. There are many companies working in tandem on things at the OpenAI scale of models, and they can independently assess the usefulness of continually growing models. They aren't going to take this statement at face value and change their strategy based on a single statement by OpenAI's CEO.
Thirdly, I think people aren't really reading what Altman actually said very closely. He doesn't say that larger models aren't useful at all, but that the next sea change in AI won't be models which are orders of magnitude bigger, but rather a different approach to existing problem sets. Which is an entirely reasonable prediction to make, even if it doesn't turn out to be true.
All in all, "his word is basically worthless" seems much to harsh an assessment here.
We will need a combination of technologies we have in order to really achieve emergent intelligence. Humans are comprised of various "subnets" modelling aspects which, in unison, produce self-conciousness and real intelligence. What is missing in the current line of approaches is that we only rely on auto-alignment of subnetworks by machine learning, which scales only up to a point. If we would produce a model which…
The other elements that may be required could be some version of the continuous sensory input that to us creates the sensation of "living" and, this one is a bit more philosophical, the sensation of suffering and a baseline establishment that the goal of the entity is to take actions that help it avoid suffering.
>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…
I've been training large 65b models on "rent for N hours" systems for less than 1k per customized model. Then fine tuning those to be whatever I want for even cheaper. 2 months since gpt 4. This ride has only just started, fasten your whatevers.
Trying to replicate the quality of GPT-3 from scratch, using all the tricks and training optimizations in the books that are available now but weren't used during GPT-3 actual training, will still cost you north of $500K, and that's being extremly optimistic.
GPT-4 level model would be at least 10x this using the same optimism (meaning you are managing to train it for much cheaper than OpenAI). And That's just pure hardware cost, the team you need to actually makes this happen is going to be very expensive as well.
edit: To quantify how "extremely optimistic" that is, the very model you are finetuning, which I assume is Llama 65B, would cost around ~$18M to train on google cloud assuming you get a 50% discount on their listed GPU prices (2048 A100 GPUs for 5 months). And that's not even GPT-4 level.
Earlier quoted context omitted.
Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685 There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.
And why will google allow them to do that at scale?
Earlier quoted context omitted.
5 cents. Per resume. $500 per 10k. 1-3 hours of a fully loaded engineers salary per year. You are being criminally cheap.
Also you could likely use GPT3.5 for this and still get near perfect results.
I have tried GPT3.5 and GPT4 for this type of task - the "near perfect results" is really problematic because you need to verify that it's likely correct, notify you if there's issues, and even then you aren't 100% sure that it selected the correct first/last name.
This is compared to a standard html form. Which is.... very reliable and (for us) automatically has error handling built in, including alerts to us if there's a 504.
Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
If you were devious enough, you could be listening in on billions of phone conversations and messages and adding that to your data set. This also makes me doubt that NSA hasn't already cracked this problem. Or that China won't eventually beat current western models since it will likely have way more data collected from its citizenry.
Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.