Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

71–80 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#71

Post by neel8986 »

machdiamonds wrote: post #41
cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685 There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.

And why will google allow them to do that at scale?

Re: OpenAI’s CEO says the age of giant AI models is already over

#72

Post by beachy »

ericmcer wrote: post #66
thewataccount wrote: post #48

Earlier quoted context omitted.

Nah - GPT-4 is crazy expensive, paying 20$/mo only get's you 25messages/3hours and it's crazy slow. The api is rather expensive too. I'm pretty sure that GPT-4 is ~1T-2T parameters, and they're struggling to run it(at reasonable performance and profit). So far their strategy has been to 10x the parameter count every GPT generation, and the problem is that there's diminishing returns everytime they do that. AFAIK they…

Yeah I am noticing this as well. GPT enables you to do difficult things really easily, but then it is so expensive you would need to replace it with custom code for any long term solution. For example: you could use GPT to parse a resume file, pull out work experience and return it as JSON. That would take minutes to setup using the GPT API and it would take weeks to build your own system, but GPT is so expensive tha…

> For example: you could use GPT to parse a resume file, pull out work experience and return it as JSON. That would take minutes to setup using the GPT API and it would take weeks to build your own system, but GPT is so expensive that building your own system is totally worth it.

True, but an HR SaaS vendor could use that to put on a compelling demo to a potential customer, stopping them from going to a competitor or otherwise benefiting.

And anyway, without churning the numbers, for volumes of say 1M resumes (at which point you've achieved a lot of success) I can't quite believe it would be cheaper to build something when there is such a powerful solution available. Maybe once you are at 1G resumes... My bet is still no though.

Re: OpenAI’s CEO says the age of giant AI models is already over

#73

Post by thewataccount »

beachy wrote: post #72
ericmcer wrote: post #66

Earlier quoted context omitted.

Yeah I am noticing this as well. GPT enables you to do difficult things really easily, but then it is so expensive you would need to replace it with custom code for any long term solution. For example: you could use GPT to parse a resume file, pull out work experience and return it as JSON. That would take minutes to setup using the GPT API and it would take weeks to build your own system, but GPT is so expensive tha…

> For example: you could use GPT to parse a resume file, pull out work experience and return it as JSON. That would take minutes to setup using the GPT API and it would take weeks to build your own system, but GPT is so expensive that building your own system is totally worth it. True, but an HR SaaS vendor could use that to put on a compelling demo to a potential customer, stopping them from going to a competitor or…

I work for a company with the web development team. We have ~6 software developers.

I'd love to be able to just have people submit their resume's and extract the data from there, but instead I'm going to build a form and make applicants fill it out because chatGPT is going to be at least $0.05USD depending on the length of the resume.

I'd also love to have mini summeries of order returns summerized in human form, but that also would cost 0.05USD per form.

the tl;dr here is that there's a TON of usecases for a LLM outside of your core product (we sell clothes) - but we can't currently justify that cost. Compare that to the rapidly improving self-hosted solutions which don't cost 0.05USD for literally any query (and likely more for anything useful).

Re: OpenAI’s CEO says the age of giant AI models is already over

#74

Post by sottol »

JasonZ2 wrote: post #23
cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Video. > YouTubers upload about 720,000 hours of fresh video content per day. Over 500 hours of video were uploaded to YouTube per minute in 2020, which equals 30,000 new video uploads per hour. Between 2014 and 2020, the number of video hours uploaded grew by about 40%.

But what are you mostly "teaching" the LLM then? Mundane everyday stuff? I guess that would make them better at "being average human" but is that what we want? It already seems that prompting the LLM to be above-average ("pretend to be an expert") improves performance.

Re: OpenAI’s CEO says the age of giant AI models is already over

#75

Post by generalizations »

I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run.

Pretty sure Microsoft swapped out Bing for something a lot smaller in the last couple of weeks; Google hasn't even tried to implement a publicly available large model. And OpenAI still has usage caps on their GPT-4.

I'd bet that they can still see improvement in performance with GPT-5, but that when they look at the usage ratio of GPT3.5 turbo, gpt3.5 legacy, and GPT4, they realized that there is a decreasing rate of return for increasingly smart models - most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant.

Obviously some practitioners of some niche disciplines (like ours here) would like a hyperintelligent AI to do all our work for us. But even a lot of us are on the free tier of ChatGPT 3.5; I'm one of the few paying $20/mo for GPT4; and idk if even I'd pay e.g. $200/mo for GPT5.

Re: OpenAI’s CEO says the age of giant AI models is already over

#76

Post by MuffinFlavored »

Uehreka wrote: post #36
MuffinFlavored wrote: post #13

Earlier quoted context omitted.

eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)

I’d reconsider the “might as well just be as time consuming” thing. I see this argument about Copilot a lot, and it’s really wrong there, so it might be wrong here too. Like, for most of the time I’m using it, Copilot saves me 30 seconds here and there and it takes me about a second to look at the line or two of code and go “yeah, that’s right”. It adds up, especially when I’m working with an unfamiliar language and…

> Like, for most of the time I’m using it, Copilot saves me 30 seconds here and there and it takes me about a second to look at the line or two of code and go “yeah, that’s right”.

I've never used Copilot but I've tried to replace StackOverflow with ChatGPT. The difference is, the StackOverflow responses compile/are right. The ChatGPT responses will make up an API that doesn't exist. Major setback.

Re: OpenAI’s CEO says the age of giant AI models is already over

#77

Post by MuffinFlavored »

MattPalmer1086 wrote: post #45
MuffinFlavored wrote: post #13

Earlier quoted context omitted.

eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)

Thing is, you can't trust what you find on stack overflow or other sources either. And searching, reading documentation and so on takes a lot of time too. I've personally been using it to explore using different libraries to produce charts. I managed to try out about 5 different libraries in a day with fairly advanced options for each using chatGPT. I might have spent a day in the past just trying one and not to the…

> Thing is, you can't trust what you find on stack overflow or other sources either.

Eh. An outdated answer will be called out in the comments/downvoted/updated/edited more often than not, no?

Re: OpenAI’s CEO says the age of giant AI models is already over

#79

Post by strangattractor »

tfehring wrote: post #20

Related reading: https://dynomight.net/scaling/ In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third. Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise…

Good thing he got a bunch of companies to pony up the dough for LLM before he announced they where already over.

Re: OpenAI’s CEO says the age of giant AI models is already over

#80

Post by mejutoco »

ericmcer wrote: post #66
thewataccount wrote: post #48

Earlier quoted context omitted.

Nah - GPT-4 is crazy expensive, paying 20$/mo only get's you 25messages/3hours and it's crazy slow. The api is rather expensive too. I'm pretty sure that GPT-4 is ~1T-2T parameters, and they're struggling to run it(at reasonable performance and profit). So far their strategy has been to 10x the parameter count every GPT generation, and the problem is that there's diminishing returns everytime they do that. AFAIK they…

Yeah I am noticing this as well. GPT enables you to do difficult things really easily, but then it is so expensive you would need to replace it with custom code for any long term solution. For example: you could use GPT to parse a resume file, pull out work experience and return it as JSON. That would take minutes to setup using the GPT API and it would take weeks to build your own system, but GPT is so expensive tha…

You could use those examples to finetune a model only for resume-data extraction.
Post reply on HN