Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's plans according to sama

humanloop.com

Post reply on HN
Print view View original

251–260 of 269 posts

Re: OpenAI's plans according to sama

#251

Post by dbmnt »

londons_explore wrote: post #5

> is limited by GPU availability. Which is all the more curious, considering OpenAI said this only in January: > Azure will remain the exclusive cloud provider for all OpenAI workloads across our research, API and products [1] So... OpenAI is severely GPU constrained, it is hampering their ability to execute, onboard customers to existing products and launch products. Yet they signed an agreement not to just go rent…

I don't think Amazon offers what Azure does (yet) in terms of HPC or multi-GPU capacity. The blog post doesn't say how long the agreement is for, but the relationship probably makes sense at the moment.

All the cloud providers are building out this type of capacity right now. It's already having a big impact in terms of quarterly spend, which we just saw in the NVDA Q1 results. AWS, Azure, and GCP for sure, but also smaller players like Dell and HPE and even NVidia themselves are trying to get into this market. (Disclaimer: I work at one of these places but don't feel like saying which). I suspect the GPU constraints won't be around too long, at which point we'll find out if OpenAI made a contractual mistake.

Re: OpenAI's plans according to sama

#252

Post by dbmnt »

Would someone please explain like I'm five which components of LLM's like ChatGPT are still closed source? What are the specific technologies that OpenAI is holding on to?

I know a lot of LLM stuff has either been released or leaked out, but don't have enough expertise in this area to understand the competitive advantages or breakthroughs OpenAI has obtained.

Re: OpenAI's plans according to sama

#253

Post by MarcoZavala »

wintogreen74 wrote: post #66
ftxbro wrote: post #43

Earlier quoted context omitted.

yes i also can't wrap my head around how a ceo of a billion dollar company isn't sincere in his public statements

Really? Even after saying this? "While Sam is calling for regulation of future models, he didn’t think existing models were dangerous and thought it would be a big mistake to regulate or ban them."

[dead]

Re: OpenAI's plans according to sama

#254

Post by pnt12 »

dbmnt wrote: post #252

Would someone please explain like I'm five which components of LLM's like ChatGPT are still closed source? What are the specific technologies that OpenAI is holding on to? I know a lot of LLM stuff has either been released or leaked out, but don't have enough expertise in this area to understand the competitive advantages or breakthroughs OpenAI has obtained.

As far as I understand, it's mostly the weights. If you only have the models, you're still gonna need to get a massive amount of training material, huge costs for training it and fine tuning the hyper params until it works.

Re: OpenAI's plans according to sama

#255

Post by SanderNL »

jasmer wrote: post #243
SanderNL wrote: post #192

Earlier quoted context omitted.

> incapable of any kind of reasoning If this were true the debate would be a hell of lot easier. Unfortunately, it is not.

There is no reasoning, which is why it will be impossible to move the LLM's past certain kinds of tasks. They are 'next word prediction models' which elicit some kinds of reasoning embedded in our language, but it's a crude approximation at best. The AGI metaphors are Ayahuasca Koolaid, like a magician duped by his own magic trick. There will be no AGI, especially because there will be not 'automaton' aka distinct en…

> There is no reasoning

> elicit some kinds of reasoning

I know it's hard, but you have to choose here. Are they reasoning or are they not reasoning?

> next word prediction models

238478903 + 348934803809 = ?

Predict the next word. What process do you propose we use here? "Approximately" reason? That's one hell of a concept you conjured up there. Very interesting one. How does one "approximates" reason and what makes it so that the approximation will forever fail to arrive at its desired destination?

> Whereas each automaton in the human world had it's own distinct 'context' - the AI world will not have that at all. Context will be as fleeting as memory in RAM, and it will be across various systems that we use daily.

Human context is fleeting as well. Time, dementia and ultimately death can attest to that. Even in life identity is complicated and multifaceted without singular I. For all intents and purpose we too are composed of massive amounts of loosely linked subsystems vaguely resembling some sort of unity. I agree with you on that one. General intelligence IMO probably requires some form of cooperation between disparate systems.

But you see some sort of fundamental difference here between "biology" and "tech" that I just cannot. If RAM was implemented biologically, would it cease to be RAM? I fail to see what's so special about the biological substrate.

To be clear, I'm not saying LLMs are AGI, but I have a hard time dismissing the notion that some combination of systems - of which LLMs might be one - will result in something we just have to call generally intelligent. Biology just beat us to it, like it did with so many things.

> It's just tech, that's it.

The human version is: it's just biology, that's it. What's the purpose of stating that?

Re: OpenAI's plans according to sama

#256

Post by Aerbil313 »

PoignardAzur wrote: post #234
outside1234 wrote: post #177

Earlier quoted context omitted.

There is no way this guy hasn't figured out some way to get paid out of this. We just haven't figured out how yet.

I mean, Occam's Razor says if he wanted a way to get paid, he would just have equity or a very large salary, the normal things CEOs do when they want to make lots of money from their very profitable company.

He gained a huge amount of reputation by saying "I don't have equity in the company. I'm doing it because I love it" in front of the Congress. While he was demanding licensing for AI because it's too dangerous.

I love it. It's dangerous. Hypocrisy? No. It's the tendency of man to pursue what is harmful to himself. Even though optimistic technologists of HN won't agree, much of technology has not benefitted humanity.

Re: OpenAI's plans according to sama

#257

Post by renonce »

paxys wrote: post #67
sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

More like – it won't be useful to small-time developers (since they won't have the capability to host and run it themselves) and so all the benefits will be reaped by AWS and other large players.

A big chunk of developments based on Facebook’s LLaMA model are by small-time developers and individuals, not large players. Facebook has already shown a viable way to release models in the way you described.

Re: OpenAI's plans according to sama

#259

Post by renonce »

legendofbrando wrote: post #154
brucethemoose2 wrote: post #118

Earlier quoted context omitted.

Anthropic's Claude is said to be very good. Instruction tuned LLaMA 65B/Falcon 40B are good, especially with an embeddings database. ...But OpenAI has all the name recognition and ease of use now, so it might not even matter if others ambiguously surpass OpenAI models.

The problem with Claude is that it is quite literally impossible to get off the waiting list to use it. To OpenAI’s credit they actually ship the product in an accessible way to developers.

Apart from poe.com there is also nat.dev. It even supports Claude-100K. Just pay $5 and it will bill by API pricing, proportional to number of tokens.

Re: OpenAI's plans according to sama

#260

Post by renonce »

jiggawatts wrote: post #103
londons_explore wrote: post #5

> is limited by GPU availability. Which is all the more curious, considering OpenAI said this only in January: > Azure will remain the exclusive cloud provider for all OpenAI workloads across our research, API and products [1] So... OpenAI is severely GPU constrained, it is hampering their ability to execute, onboard customers to existing products and launch products. Yet they signed an agreement not to just go rent…

One of Azure's unique offerings is very large HPC clusters with GPUs. You can deploy ~1,000 node scale sets with very high speed networking. AWS has many single-server GPU offerings, but nothing quite like what Azure has. Don't assume Microsoft is bad at everything and that AWS is automatically superior at all product categories...

Large scale sets are only needed for training. For inference, 8x NVIDIA A100 80G will allow inference for 300b models (GPT-3 is 175b) or 1200b models with 4-bit quantization (quantization impact is negligible for large models), so a single machine is sufficient.
Post reply on HN