281–290 of 525 posts
Earlier quoted context omitted.
So your embrace of nihilism is total. Here are a couple of books worth reading: - On Tyranny: https://bookshop.org/p/books/on-tyranny-twenty-lessons-from-... - The Road to Unfreedom: https://bookshop.org/p/books/the-road-to-unfreedom-russia-eu...
[flagged]
Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
First of all, if Altman continually makes misleading statements about AI he will quickly lose credibility, and that short term gain in whatever 'financial incentive' that birthed the lie would be eroded in short order by a lack of trust of the head of one of the most visible AI companies in the world. Secondly, all the competitors of OpenAI can plainly assess the truth or validity of Altman's statements. There are ma…
I don't find this implausible, but it folds slightly to Ockham's razor when you consider that this is the exact type of statement that would be employed to obfuscate a major breakthrough.
It just makes me crook my eyebrow and look to more credible sources.
Sure yeah the cost numbers are getting very large and we can't keep scaling forever. But Google could easily 10x the training cost of GPT-4 if they thought it would protect their search business. I'm still skeptical that scaling is enough to reach the thresholds we want, but I'm surprised that it's being claimed right now when there's a huge rush of new money into the space. I wonder if this is some sort of misdirection by Sam
Related reading: https://dynomight.net/scaling/ In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third. Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise…
I don't think there is such a clear separation between algorithms and data as your comment suggests.
Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
Videos - all of youtube, all the movies, everything that's ever been captured on film. Transcribe the audio, automatically describe the images and try to predict the next one.
Maybe if you trained it on movies before CGI existed ?
Earlier quoted context omitted.
GPT4: In the sentence "What is the third letter in the third word in this sentence?", the third word is "third." The third letter in "third" is 'i'. Given that we have only provided a text prompt and no context or body language hints, GPT seems to interpret this literally. The answer is correct. The third letter of the word "third" is "i".
>GPT seems to interpret this literally. No it doesn't interpret anything. (which is why I've gotten three completely different answers here on HN within 10 minutes). It cannot count at all or even understand the question which becomes obvious when you ask it for word counts or letter counts that don't match anything in the sentence, at which point it makes up nonsense. what is the tenth word in the following sentence…
That's what I just got. No clue what you're using/getting.
I don't quite understand this. The returns to scale have always been sublinear (ie "diminishing"), but the scale-maximalists didn't worry about that before. Also I heard Sam tout on the Lex Friedman podcast how amazing it is that the scaling laws worked so well for GPT-4. So I wonder what changed? Sure yeah the cost numbers are getting very large and we can't keep scaling forever. But Google could easily 10x the trai…
Maybe cost & latency for both training and inference it getting too high. If costs doubled for every 5% better performance, would it be worth it? NVIDIA is making a small fortune from this.
> But Google could easily 10x the training cost of GPT-4 if they thought it would protect their search business
Google makes /\$0\.[0+]\d/ per search query. If the inference cost of the model exceeds that, they go from making money to losing money. It is not clear if the Bing integration is a money maker or a lost leader.
Earlier quoted context omitted.
I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabi…
How did we jump to AGI? "Sammy A thinks we've made the best engine with the tools at hand" -> "this will never get us out of the solar system" Sorry to unload on you. It is frustrating to constantly see AGI get brought up needlessly on HN
> Are you saying instead, that concrete predictive algorithms need improvement or are we lumping the tuning into this?
in the context of what's needed to get to AGI - just as if NASA built an engine we'd talk about its effectiveness in the context of space flight.
dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.
He’s not presenting false evidence here, he’s presenting a hunch. It’s a guess. No one is going to gain anything from this one way or another.
Then why did Altman even bother to make this very public statement?
GPT4 already obsolete? Geez, we got the wrong bot, silly heads: https://www.youtube.com/watch?v=EdRMVhlNP5I