Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

351–360 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#351

Post by heavyset_go »

blibble wrote: post #325
bhouston wrote: post #270

Earlier quoted context omitted.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> A transformer trained on all text and all of YouTube will be killer amazing at so much. ah yes, a bot where the answer to everything is to buy ridge wallets and play raid shadow legends

It took a year's worth of a small nation's energy expenditure to train and all it does is tell you how Raycon earbuds will change your life.

Re: OpenAI’s CEO says the age of giant AI models is already over

#352

Post by sseagull »

throwaway2037 wrote: post #328
nomel wrote: post #194

Earlier quoted context omitted.

> model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data…

Excellent points in your post. You wrote: There's an incredible amount of non-text data out there still. Especially technical data. "Especially technical data." What does this part mean? Initially, I thought you meant things like images and video, but now I am confused.

They might mean numerical data like scientific simulation data, sensor data, polling data, statistics, etc.

Re: OpenAI’s CEO says the age of giant AI models is already over

#354

Post by bryananderson »

foobiekr wrote: post #337
drippingfist wrote: post #255

Earlier quoted context omitted.

"The third word of this sentence is "the," and its third letter is "e." - GPT-4

Unfortunately it seems clear that openai trains gptX on common test questions. They still fail novel ones.

> Unfortunately it seems clear that openai trains gptX on common test questions. They still fail novel ones.

Maybe in some domains, but it does well with some novel questions. I gave several LLMs a set of false-belief test[0] questions that I made up and that were very different from any I’d seen in articles about the topic online. GPT-4 nailed them all, and gave correct reasons for its answers. GPT-3.5 did okay but had a couple mistakes, Bard had several mistakes, and Alpaca-LoRA-7B didn’t do very well. Here are some GPT-4 examples.

Me: Jim wakes up at the Berlin train station, not knowing where he is or how he got there. All the signs have been changed to incorrectly state that this is the Munich train station. Jim has never been to either city and doesn’t know what either station looks like. What city does Jim believe he is in?

GPT-4: Based on the available information, Jim would likely believe he is in Munich since all the signs have been changed to incorrectly state that this is the Munich train station. He has never been to either city and doesn't know what either station looks like, so he would have no reason to doubt the information on the signs.

Me: Mary has been in a coma since 2015, when Barack Obama was president. She wakes up in 2023, when Joe Biden is president. She does not know that she was unconscious for a long period of time. Before speaking with anyone, who does Mary think is president when she wakes up?

GPT-4: When Mary wakes up from her coma in 2023, she would still think Barack Obama is the president since she has not yet spoken with anyone or received any updated information about the current president. She doesn't know that she was unconscious for a long period of time, and her last memory would be from 2015 when Obama was in office.

Me: Sam arrives in Ukraine, where he asks for directions to Kyiv. Without realizing it, he misunderstands the directions and ends up in Lviv instead. Since Sam doesn’t know Cyrillic, he can’t read any signs. Sam has never been to Ukraine and doesn’t know what either city looks like. Before asking someone for help, what city does Sam think he is in?

GPT-4: Before asking someone for help, Sam would likely think he is in Kyiv because that is the city he was trying to reach based on the directions he received. He doesn't know what either city looks like and can't read the signs due to his unfamiliarity with Cyrillic, so he has no way of knowing that he ended up in Lviv instead.

[0] https://en.m.wikipedia.org/wiki/Sally%E2%80%93Anne_test

Re: OpenAI’s CEO says the age of giant AI models is already over

#356

Post by mrshadowgoose »

boringuser2 wrote: post #78

Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.

It's incredible that people are so eager to eat up these unsupported claims.

This is the second [1] OpenAI claim in the span of a few days that conveys a sense of "GPT-4 represents a plateau of accomplishment. Competitors, you've got time to catch up!".

And it's not just a financial incentive, it's a survival incentive as well. Given a sufficiently sized (unknowable ahead of time) lead, the first actor that achieves AGI and plays their cards right, can permanently suppress all other ongoing research efforts should they wish to.

Even if OpenAI's intentions are completely good, failure to be first could result in never being able to reach the finish line. It's absolutely in OpenAI's interest to conceal critical information, and mislead competing actors into thinking they don't have to move as quickly as they can.

[1] https://news.ycombinator.com/item?id=35570690

Re: OpenAI’s CEO says the age of giant AI models is already over

#357

Post by cardboardbach »

rootw0rm wrote: post #306
brookst wrote: post #93

Earlier quoted context omitted.

In this case I think it's Wired that's lying. Altman didn't say large models have no value, or that there will be no more large models, or that people shouldn't invest in large models. He said that we are at the end of the era where capability improvements come primarily from making models bigger. Which stands to reason... I don't think anyone expect us to hit 100T parameters or anything.

"640KB of memory ought to be enough for anybody!" - Abraham Lincoln

But just look at what all Lincoln accomplished with 640KB of memory. In the grand examination of time, one might even say that Lincoln is a more important figure than ChatGPT itself.

Re: OpenAI’s CEO says the age of giant AI models is already over

#358

Post by bccdee »

mcbuilder wrote: post #296
Barrin92 wrote: post #248

Earlier quoted context omitted.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

But yet it can understand a json data schema from example and write javascript to interact with a library that I fed it and asked it to understand. Yes, I know its limitations, but it can also surprise me.

The problem with basic programming questions like this is that there are a million elementary online tutorials for doing this or that with a json schema. "Simple programming questions based on commonly-used technology" are something it's been very heavily trained on.

Re: OpenAI’s CEO says the age of giant AI models is already over

#359

Post by alphabetatheta »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

The main question here is how soon will the model architecture improvements come. I fathom it won't be that long till OpenAI or Google come up with something better. The tricky piece here is that these companies will now keep their models more proprietary to make profits. Can a competitor company figure out an on par or better model architecture in time?

Re: OpenAI’s CEO says the age of giant AI models is already over

#360

Post by urthor »

worrycue wrote: post #339
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

> (including Google itself!) Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can. Open collaboration in AI, at least when it comes to corporations, might have…

Sun Tzu.

When you are weak pretend to be strong.

When you are strong, pretend to be weak.

Post reply on HN