OpenAI's plans according to sama • Hacker News Board

Re: OpenAI's plans according to sama

#141

Post by visarga » Wed, May 31, 2023, 11:16 PM UTC

Loquebantur wrote: post #128

visarga wrote: post #111

Earlier quoted context omitted.

Yes. Solving context length has been tried in hundreds of different approaches, and yet most LLMs are almost identical to the original one from 2017. Just to name a few families of approaches: Sparse Attention, Hierachical Attention, Global-Local Attention,Sliding Window Attention, Locality sensitive hashing Attention, State space model, EMA gated attention.

I assume, there is a common point of failure? Notably, human working memory isn't great either. Which begs the question (if the comparison is valid) as to whether that limitation might be fundamental.

The failure mode is that only long context tasks benefit, short ones work fast enough with full attention, and better. It's amazing that OpenAI never used them in any serious LLM even though training costs are huge.

Re: OpenAI's plans according to sama

#142

Post by sashank_1509 » Wed, May 31, 2023, 11:17 PM UTC

alchemist1e9 wrote: post #84

sashank_1509 wrote: post #48

Really great news to give at cheaper and faster GPT4. As a GPT+ subscriber, the most annoying thing is the 25 message limit every 3 hours, I really want that removed. A bit sad to hear that the multimodal model will only come next year, was hoping to get it this year 100k to 1 Million context length, sounds phenomenal especially if it comes to GPT4. I've used Claudes 100k context length and I found it so useful that…

Did you have any tips on how you got access to Claude? I do the request access but never get any email or any contact.

I'm a graduate student doing AI research in a US University. And I applied pretty early (Last year December I think) , those might be two factors that got me access to Claude.

I think getting access to Claude through slack is much easier and I recently got it by just downloading it as a Slack App

Re: OpenAI's plans according to sama

#143

Post by rafark » Wed, May 31, 2023, 11:18 PM UTC

TapWaterBandit wrote: post #132

sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

When you stop listening to what Sam Altman says and just focus on what he does, you can see the guy is a bit of a snake. Greedy power-hungry man imho.

I’ve tried very hard to like him because like it or not, ChatGPT has revolutionized the AI industry but he’s so hypocritical I just can’t stand him.

Re: OpenAI's plans according to sama

#144

Post by rafark » Wed, May 31, 2023, 11:20 PM UTC

purplecats wrote: post #32

> Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time. this certainly aligns with the massive (albeit subjective and anecdotal) degradation in quality i've experienced with ChatGPT GPT-4 over the past few weeks. hopefully a superior (higher quality)…

Exactly. It’s become ‘cheap’. This is why we need more good competition.

Re: OpenAI's plans according to sama

#145

Post by jeffybefffy519 » Wed, May 31, 2023, 11:25 PM UTC

Title needs an update as Sama is also the name of the company which helped classify training data for ChatGPT: https://time.com/6247678/openai-chatgpt-kenya-workers/

Re: OpenAI's plans according to sama

#146

Post by mellosouls » Wed, May 31, 2023, 11:29 PM UTC

TapWaterBandit wrote: post #132

sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

When you stop listening to what Sam Altman says and just focus on what he does, you can see the guy is a bit of a snake. Greedy power-hungry man imho.

Sam Altman is responsible for leading the team that have revolutionised AI in its position within society.

There is plenty to criticise OpenAI for but what he and they have achieved is extraordinary, and there is no need for that sort of toxic personal attack.

Re: OpenAI's plans according to sama

#147

Post by timeserious » Wed, May 31, 2023, 11:31 PM UTC

brucethemoose2 wrote: post #118

purplecats wrote: post #32

> Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time. this certainly aligns with the massive (albeit subjective and anecdotal) degradation in quality i've experienced with ChatGPT GPT-4 over the past few weeks. hopefully a superior (higher quality)…

Anthropic's Claude is said to be very good. Instruction tuned LLaMA 65B/Falcon 40B are good, especially with an embeddings database. ...But OpenAI has all the name recognition and ease of use now, so it might not even matter if others ambiguously surpass OpenAI models.

Can you ELI5 why an embeddings database helps here? Can pinecone/milvus be used to 'extend memory' of OSS and vendor LLMs without retraining?

Re: OpenAI's plans according to sama

#148

Post by rushingcreek » Wed, May 31, 2023, 11:36 PM UTC

razcle wrote: post #120

sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

I think I worded this poorly. What he said was that a lot of people say they want open-source models but they underestimate how hard it is to serve them well. So he wondered how much real benefit would come from open-sourcing them. I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than naviga…

If he says he's inclined to open-source GPT-3, I don't see any good arguments not in favor of giving startups the choice of how they can run inference.

Re: OpenAI's plans according to sama

#149

Post by cwkoss » Wed, May 31, 2023, 11:38 PM UTC

boringuser2 wrote: post #71

layer8 wrote: post #70

Earlier quoted context omitted.

Someone without family?

You know the next step, right?

"Cooperate or we will find you a spouse, make you fall in love, and have a family"

Re: OpenAI's plans according to sama

#150

Post by boringuser2 » Wed, May 31, 2023, 11:44 PM UTC

CSMastermind wrote: post #94

cwkoss wrote: post #19

Earlier quoted context omitted.

Last night I was musing how many different countries' intelligence agencies have moles working at OpenAI currently. Gotta be at least 6, maybe as high as two dozen?

US, France, Israel ... then who? Maybe another five eyes country like the UK? Possibly China? I'm pretty skeptical Russia would be able to get someone in there but maybe.

You said Isr*el twice.