Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's plans according to sama

humanloop.com

Post reply on HN
Print view View original

161–170 of 269 posts

Re: OpenAI's plans according to sama

#161

Post by Imnimo »

>The fact that scaling continues to work has significant implications for the timelines of AGI development. The scaling hypothesis is the idea that we may have most of the pieces in place needed to build AGI and that most of the remaining work will be taking existing methods and scaling them up to larger models and bigger datasets. If the era of scaling was over then we should probably expect AGI to be much further away. The fact the scaling laws continue to hold is strongly suggestive of shorter timelines.

If you understand the shape of the power law scaling curves, shouldn't this scaling hypothesis tell you that AGI is not close, at least via a path of simply scaling up GPT-4? For example, the GPT-4 paper reports a 67% pass-rate on the HumanEval benchmark. In Figure 2, they show a power-law improvement on a medium-difficulty subset as a function of total compute. How many powers of ten are we going to increase GPT-4 compute by just to be able to solve some relatively simple programming problems?

Re: OpenAI's plans according to sama

#162

Post by r3trohack3r »

sovietmudkipz wrote: post #7

I’m hoping GPT will remove the information cutoff date. I write plenty of terraform/AWS and it’s a bit of a pain that the latest API isn’t accessible by GPT yet. There’s been quite a bit happening in the programming space since sept 2021. I use GPT to keep things high level and then do my normal research methodology for implementation details.

Injecting the context yourself can help a lot. I frequently copy in a bunch of example code at the beginning of the conversation to help prime ChatGPT on APIs it knows nothing about.

For smaller projects that will fit, I've taken to: `xclip *` and then pasting the entire collection of files into ChatGPT before describing what I want to do.

Re: OpenAI's plans according to sama

#163

Post by sanxiyn »

Imnimo wrote: post #161

>The fact that scaling continues to work has significant implications for the timelines of AGI development. The scaling hypothesis is the idea that we may have most of the pieces in place needed to build AGI and that most of the remaining work will be taking existing methods and scaling them up to larger models and bigger datasets. If the era of scaling was over then we should probably expect AGI to be much further a…

Someone did that calculation and the result is here: https://www.reddit.com/r/slatestarcodex/comments/13u40yf/

100x GPT-4 to 85%.

Re: OpenAI's plans according to sama

#164

Post by bongobingo1 »

bigyikes wrote: post #159
tikwidd wrote: post #151

Earlier quoted context omitted.

After training my physics simulator on thousands of hours of video footage of trees moving in the wind, arborists tell me the trees are much more realistic (they are getting worried that I might put them out of business). But the physicists are still not satisfied. How many more videos do I need to generate the laws of motion?

Throw in the videos from the rest of the internet, and you might actually do it…

> The simulations incredible, but I have to ask, why do the trees all have breasts?

Re: OpenAI's plans according to sama

#165

Post by throwaway1777 »

mellosouls wrote: post #146
TapWaterBandit wrote: post #132

Earlier quoted context omitted.

When you stop listening to what Sam Altman says and just focus on what he does, you can see the guy is a bit of a snake. Greedy power-hungry man imho.

Sam Altman is responsible for leading the team that have revolutionised AI in its position within society. There is plenty to criticise OpenAI for but what he and they have achieved is extraordinary, and there is no need for that sort of toxic personal attack.

It’s fine to feel the ends justify the means but not everyone believes that.

Re: OpenAI's plans according to sama

#167

Post by freediver »

timeserious wrote: post #147
brucethemoose2 wrote: post #118

Earlier quoted context omitted.

Anthropic's Claude is said to be very good. Instruction tuned LLaMA 65B/Falcon 40B are good, especially with an embeddings database. ...But OpenAI has all the name recognition and ease of use now, so it might not even matter if others ambiguously surpass OpenAI models.

Can you ELI5 why an embeddings database helps here? Can pinecone/milvus be used to 'extend memory' of OSS and vendor LLMs without retraining?

Any database can be used to extend the memory of LLMs. What a database does is store stuff and lets you search/retrieve stuff. Embeddings are differet form of data that are in many (but not all) cases superior to searching through text.

You do not need a fancy cloud hosted service to use an embeddings database like you do not need one to use a regular databse (although you could).

Check https://github.com/kagisearch/vectordb for a simple implementation of a vector search database that uses local, on-premise open source tools and lets you use an embeddings database in 3 lines of code.

Re: OpenAI's plans according to sama

#168

Post by neximo64 »

TapWaterBandit wrote: post #132
sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

When you stop listening to what Sam Altman says and just focus on what he does, you can see the guy is a bit of a snake. Greedy power-hungry man imho.

In your world how would you consider Sam Altman having no equity in OpenAI? And everyone finding out after it had a viral hit

Re: OpenAI's plans according to sama

#169

Post by choppaface »

razcle wrote: post #120
sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

I think I worded this poorly. What he said was that a lot of people say they want open-source models but they underestimate how hard it is to serve them well. So he wondered how much real benefit would come from open-sourcing them. I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than naviga…

Here is how hard it is to serve and use LLMs: https://github.com/ggerganov/llama.cpp

Re: OpenAI's plans according to sama

#170

Post by [deleted] »

paxys wrote: post #67
sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

More like – it won't be useful to small-time developers (since they won't have the capability to host and run it themselves) and so all the benefits will be reaped by AWS and other large players.

[deleted]
Post reply on HN